Traditionally, psychiatric education has focused on "noncognitive" skills such as establishing rapport, communicating empathy, and developing an appreciation for the dynamic interactions between patients and doctors. However, with the burgeoning increase in scientific information relevant to the field of psychiatry and the growing emphasis on the importance of detailed and specific diagnoses and treatments, the importance of objective written examinations as a means for assessing competence among trainees has grown. A result of this changing emphasis has been the development and utilization of meticulously structured examinations testing psychiatric knowledge. At the graduate level, this is reflected in the usage of the Psychiatry Residents in-Training Examination (PRITE) (1) and an increasing number of examinations for board subspecialization (2). In medical student education, there has been an increasing usage of the National Board of Medical Examiners (NBME) Subject Test in psychiatry. The NBME Subject Tests are a valuable resource for clerkship directors in the major medical specialties. Items on these tests are current, well written, and validated. Scoring is based on national norms, and the subjects covered represent information students will be expected to master in order to pass their United States Medical Licensing Examination (USMLE) Step II tests. Clerkship directors who choose to use these tests as an alternative to departmentally developed multiple choice examinations are relieved from the burden of keeping up a bank of contemporary, secured items. This is particularly helpful in light of declining resources (e.g., faculty with time to write good items) available to academic medical centers. In addition, using these exams helps clerkship directors maintain credibility of their courses among other medical school departments, and may help the students feel better prepared for Step 2 of the USMLE (3). Despite the widespread usage of the NBME Subject Tests, the question frequently arises as to how the test should be utilized by clerkship directors. This study provides information that we believe will give practical guidance (or reassurance) to educators currently using or anticipating to use the NBME Subject Test for their clerkships.

While the NBME provides norms to aid in the interpretation of student performance, minimal guidance is provided regarding how to practically use the tests. The NBME recommends that the tests should be used with other measures of student performance in determining the overall clerkship grade, but it neither sets nor recommends a "passing" score. The rationale given is that clerkship objectives vary across schools, and therefore one cannot guarantee that test content matches clerkship objectives (4).

Based on requests for help by clerkship directors in determining scores for passing and honors, the NBME published a summary of the opinions of eight clerkship directors. The opinions of these clerkship directors were that the minimum subject score for passing should be between 51 and 61 with a mean of 57 and a minimum score for honors should be between 76 and 95 with a mean of 81 (5). While this information is useful, it is limited in scope and not necessarily representative of clerkship directors nationwide. Furthermore, the bases for these opinions and whether or not they represent real usage of the tests are unclear.

In the past decade substantial changes have occurred in the clinical curricula of medical schools. Such changes have included alterations in clinical teaching settings, increased managed care demands on clinical preceptors, combinations with other disciplines, and increased emphasis on "noncognitive" qualities such as professional behavior. Assessment has necessarily been altered to address changing trends in psychiatry clerkships. In order to determine the prevailing modes of assessment at this point in time, we developed a survey for psychiatry clerkship directors on the nature and usage of their assessments and their views on whether these assessments adequately measured their objectives. We included questions pertaining to the NBME Subject Test regarding extent of usage, how the test is incorporated into the overall grade, what "cutoff" scores are used for passing, and what is done when a student fails the test. We also surveyed users on their opinions regarding the utility of the NBME test in assessing the knowledge, skills, and professionalism of their students. It has been documented that clerkship length plays a role in Subject Test performance in both psychiatry (6) and surgery (7). We therefore hypothesized that clerkship length might correlate with the decision of whether or not to use the NBME Subject Test. A course director with a "shorter" clerkship might not have the time to establish and administer a departmental test. Alternatively, a director with a clerkship that is truncated or divided into sections over the course of the curriculum might avoid the NBME test because of the difficulty of preparing students for one test when their psychiatric training is interrupted over the course of the curriculum. The findings presented in this paper should inform current test users of how their practices conform to national norms and should provide prospective users with guidance regarding ways to incorporate the test into their existing clerkships.

U.S. and Canadian psychiatry clerkship directors (N=150) were sent an 18-item questionnaire surveying evaluation and remediation practices. Details regarding methods are outlined in a previous paper in this issue titled, The "Shrinking" Clerkship: Characteristics and Length of Clerkships in Psychiatry Undergraduate Education (8).

Those Who Use the NBME Subject Test

Of the 111 clerkship responders (74% of 150 questionnaires), 69% (N=76) reported using the NBME Subject Test. For course directors using the test the following purposes were identified:

How the Test Score Is Weighted and Converted to a Clerkship Grade

There was considerable disparity in how the test was assigned a weight and converted into a grade in U.S. and Canadian schools. Directors attributed the following weights to the test results:

Scores received from the NBME have been converted by course directors into a clerkship grade in the following ways:

How "Passing" Is Determined from the Test

Although 72 clerkship directors used the test for grading purposes, only 79% (N=60) of these directors used the test to determine "passing" of the clerkship. The scores set as "passing" for the different medical schools varied considerably. From the 60 clerkship directors selecting a score as "passing" the following was reported:

What Directors Think the Test Measures

Clerkship director opinions regarding the ability of the test to measure knowledge, clinical skills, and professionalism were gathered with the following results:

How Directors Manage "Failure" on the Test

When students did not "pass" the test on their first attempt at the required levels as noted above, the majority of programs had students retake an NBME Subject Test to remediate the "failure." Among those using the test for grading purposes the following remediation methods were reported:

Hypothesized Relationships With Length and Type of Clerkship

When the relationships between clerkship length (longer versus shorter), clerkship type (combined versus free standing), and test usage were evaluated using chi-square analysis, no significant associations were found.

The National Board of Medical Examiners Subject Test in psychiatry is a popular method of student evaluation, used by approximately two thirds of clerkship directors in the U.S. and Canada. While the majority of users (71%) endorsed that their primary reason for using the test is that it is a good way to compare students, a significant number of users (34%) administer the test because their medical school requires it. Most clerkship directors who use the test believe that it is a good measure of knowledge base. However, users are split regarding whether or not the test is a good measure of clinical skills, and the majority of users do not feel the test is a good measure of professionalism. When students fail the test on their first attempt, the majority of clerkship directors (71%) require that the student remediate the failure by retaking the examination.

Although the test is popular and well accepted, there is no standard method for using the exam to help determine the overall clerkship grade. There is considerable variability in the weight that the exam is given and in the method used to convert the NBME provided score into a clerkship grade. Of the 72 users who convert the exam into a "grade," almost as many clerkship directors use the raw subject score (N=29) as the percentile score (N=32). Yet among the 60 users who report that they use the test for "passing," nearly 3/4ths (N=43) of clerkship directors prefer the raw score. Unfortunately, our data do not illuminate the reasons for this inconsistency. Our finding that the mean passing subject score is 58 is consistent with the data published by the NBME in its subject test score interpretation guide. Of interesting note, a subject score of 58 generally corresponds to a 6th percentile rank. Therefore, those clerkship directors who use the percentile score tend to have a higher standard than those who use the subject score.

Contrary to our hypothesis that clerkship length might impact the usage of the subject exam, no statistically significant difference was found relative to usage and clerkship length. A more detailed discussion of the impact of clerkship length on evaluation is found in a companion article in this issue titled The "Shrinking" Clerkship: Characteristics and Length of Clerkships in Psychiatry Undergraduate Education (8). There seemed to be a trend towards greater usage in shorter clerkships, though the low numbers did not prove statistically significant. Our hypothesis that clerkship type (free standing vs. combined with other specialties) might affect test usage was also not supported.

Clerkship directors anticipating using the NBME subject test in psychiatry can be reassured that it is a widely used and generally well accepted measure of clerkship student knowledge base. The test is not, however, considered a good measure of clinical skills or professionalism. Given the significance of these other domains in gaining a more complete measure of medical student competency, the NBME test should be complemented by other assessment modalities.

In this same survey, and more fully described in a forth-coming paper, clerkship directors reported that clinical skills were measured well by clinical attending evaluations (94% in agreement), by objective-structured clinical examinations (90% in agreement), and by oral examinations (79% in agreement). Professional attitude development was thought to be measured well by clinical attending evaluations (94% in agreement) and oral examinations (73% in agreement). These and other assessment modalities should be considered along with the NBME Subject Test for a complete clerkship assessment process.

Those who look to their colleagues to help determine how the exam should be used to determine grading or passing will not find a "standard" procedure for weighing or converting the test. The information that we obtained with this survey is consistent with the opinions provided by the NBME in its score interpretation material. A "passing" raw score of 58 does indeed represent the national mean. Nevertheless, the wide variation in usage of the test suggests that the NBME may be correct in not recommending norms. An important piece of information that we did not obtain is the mean subject test score by school. From informal observations of our colleagues and ourselves, there appears to be wide variation in this number. This may account for the differences observed in score usage.

There are several limitations to this study. We did not inquire about a number of important questions, including prevailing mode of determining "honors," and number of years the test had been used. In retrospect, we wish we had asked the reasoning behind choosing the methods that were chosen to determine grading and passing scores. Moreover, our failure to ask about mean school scores prevented our ability to test the hypothesis that grading and passing scores were based on school—wide norms. Nevertheless, this study is a good first step in better understanding how and why clerkship directors use the NBME subject test. Educators who anticipate using the test in the future can be reassured that the test is a national standard for determining knowledge in psychiatry. Clerkship directors can also feel comfortable in the knowledge that there is a wide variation among their colleagues in the way that the test is used. Adapting grading procedures and passing scores to their own particular programs would be consistent with the behavior of their colleagues.

Approval for this study was granted by the Institutional Review Board of the University of Texas Medical Branch, Galveston, Texas.

Data contained in this manuscript were presented at the Annual Meeting of the Association of Directors of Medical Student Education in Psychiatry, Jackson Hole, Wyoming, on June 14, 2003.