The role of the National Board of Medical Examiners (NBME) and their testing services, specifically, the subject examinations and the United States Medical Licensing Examinations (USMLE), continues to be a prominent one. Most medical schools require passing scores on USMLE Steps 1 and 2 before graduation, and medical boards across the United States require passing scores on the USMLE for licensure. Also, USMLE scores are a significant factor in the residency application and match process (1, 2).
The NBME subject examinations, available for both basic science and clinical subjects, are most commonly used for end-of-course or clerkship evaluation of students, including a large majority of psychiatry clerkships in the United States (3). However, subject examinations are also often used in various other ways. For students, subject examinations aid in the preparation for USMLE Steps 1 and 2 (4, 5). For educators, item-analysis reports show how a tested group of students performed as compared with the national sample on each item, and they list the corresponding content area to assist educators in program-improvement efforts. Year-end reports similarly provide comparisons for an entire class, as compared with the national sample, and also describe relative performance in some general content areas. Also, Chairs, Deans, and curriculum committees may use student performance on NBME subject examinations as an outcome of educational effectiveness (6).
The importance of NBME subject examinations as evaluative tools and as possible predictors of performance on USMLE Step 1 and 2 exams (7) make strategies to improve subject exam performance of interest. Several studies have evaluated factors that might correlate with performance on psychiatry subject examinations. One study found a significant difference in student performance for those taking the clerkship later in the year (8), but others did not (9, 10). There are several reports that have found no significant differences based on various clerkship training sites (11–13). A survey of medical students from six different medical schools (14) reported that Team-Based Learning (TBL) (15), topic-based didactics, and the combination of case discussion/topic-based lectures were associated with improved performance, whereas, somewhat surprisingly, commercial materials geared toward step test preparation (“step-or-prep” books) were not. Concerning specific interventions to improve performance, we are aware of only two studies that report curricular changes associated with improved NBME scores. The use of TBL (16) and a longitudinal, integrated clerkship curriculum (17) were both associated with improved student performance. In this column, we describe a straightforward and portable strategy that has been utilized at one medical school that appears to be effective in improving student performance on the NBME psychiatry subject examination.
For the academic year 2000–2001, NBME psychiatry subject exam results for the author’s students were at the 28th percentile nationally, and lowest among the six major clerkships in the medical school. Psychiatry subject exam results had never been better than third among the six major clerkships, and only rarely had been above the 50th percentile. With pressure from leadership to improve results, we implemented strategies to help improve student performance. The following describes the strategies that were implemented over the course of several years to improve student performance.
1. Review of the NBME Psychiatry Subject Examination by Faculty Annually
The NBME allows faculty of medical schools and authorized staff of other appropriate medical education institutions to review once a year a current form of a web-based exam online. The Executive Chief Proctor, the individual at each medical school designated to oversee all NBME examinations, can arrange for such a review. The examination review can be scheduled in May or June so that materials can be updated in a timely fashion for the upcoming academic year. No note-taking or later reconstruction of test questions is allowed, but the review allows course directors and faculty to see content areas that are covered and emphasized as well as the construction of the questions and the answer choices. The answers are not provided—which can make for stimulating discussion among faculty. The review can help identify what curriculum deficiencies may exist and provide a road-map as to what resulting changes or additions may be needed to the course to ensure that NBME-relevant content is adequately covered.
2. Revise Current Curricular Materials on the Basis of Faculty Review of the Subject Examination
All of the core clerkship faculty are encouraged to attend the annual subject examination review. Faculty are instructed to update their instructional materials, including lectures, TBL sessions, and clinical teaching, based on what they have learned from the examination review. As an example, based on our annual reviews, child psychiatry and some relevant basic science topics were not being adequately taught. The child psychiatry didactics were modified to include coverage of several less-common disorders. Information was added on neurotransmitters and neuroanatomy to didactics on psychosis and anxiety disorders and various causes of dementia. Overall, the topic areas covered and time allotted for classroom hours remained the same, but content was updated to ensure that material that is tested by the exams was adequately covered.
3. Develop Self-Directed Learning Materials to Target Areas Not Sufficiently Covered
With limited lecture time and clinical assignments that can vary greatly, there are topics that are frequently found on the NBME subject examination that may receive little attention in the formal curriculum. Self-directed learning materials at any level, from simple handouts to multimedia online-learning modules, provide students with additional information without additional classroom time and with the flexibility to review that information when they feel it is appropriate. A set of such materials, with self-assessments, can be implemented via an online-learning management system and can allow for comparable educational experiences as required under LCME accreditation standard ED−8. In our case, the added materials started with a simple outline of some areas that had been previously neglected, and, over the years, have grown to an extensive set of topic reviews, 10 multimedia online modules, 180 online review questions, and electronic “flashcards.” An example from our experience noted that the psychiatry subject exam frequently included questions about neurologic, metabolic, and genetic disorders that may present with psychiatric symptoms. A short handout was initially used to cover the material, but later a detailed multimedia module with imbedded questions was developed and placed on our learning-management system for student self-directed use. Of note, the review questions and flashcards were almost entirely written by psychiatry residents interested in education who were recruited to assist in the project.
4. Replace Traditional Lectures With Team-Based Learning (TBL)
Team-based learning (TBL) is an educational method that increases classroom engagement and is associated with higher psychiatry subject NBME performance (16). TBL utilizes active learning and a small-group instructional strategy that makes students accountable for preparation before class as well as group work during class (15). After another clerkship in the medical school started using TBL, students requested that the method be adopted in place of traditional lectures for the psychiatry clerkship. A national expert in TBL in psychiatry with extensive experience in training faculty to use TBL was brought in for a whole-day workshop for faculty. The clerkship faculty then developed TBL materials, including required readings, individual and group “readiness-assurance tests,” and application exercises for the major psychopathology areas: child/adolescent psychiatry, psychotic disorders, mood disorders, anxiety disorders, personality disorders, and dementia/delirium. Use of this teaching methodology in the subsequent years has improved student feedback about classroom experiences and, possibly because of the active learning nature of TBL and the graded quizzes (the readiness-assurance tests), seems to have led to more consistent and evenly-spaced study time over the length of the clerkship.
Results on the psychiatry subject examination improved measurably over the next several years (Figure 1), both absolutely and relative to the other major clerkships, and have been sustained. Psychiatry subject exam scores have been above the 50th percentile and best among the six major clerkships for the last 7 academic years. There is also evidence for benefits on the USMLE Step 2 CK. For the last 5 years of available results, the two highest-performing subscales on USMLE Step 2 CK for our students were the two psychiatry-related subscales, “Psychiatry” and “Mental Disorders.” This had never been the case prior to these efforts, and results point to the possibility that curricular efforts to improve subject examination score may improve USMLE performance and pass rates.
FIGURE 1.Psychiatry NBME Subject Exam Results
Percentile score is nationally normed for all test-takers for the academic year.
Other benefits were noted. The initial pressure from leadership to improve student performance has been replaced by annual praise. The faculty incentive plan recently adopted educational benchmarks that include, among other items, student performance on NBME examinations. Although our efforts began many years before the new incentive plan was developed, the previous efforts are now providing tangible benefits to several faculty who have been heavily involved in these efforts over the years. More importantly, student feedback and outcomes have improved. In both our informal end-of-clerkship feedback session and the annual Dean’s Office course evaluations, students comment favorably about how prepared they felt for the psychiatry subject examination.
Although the anecdotal nature of the results make it impossible to prove causation, it seems likely that the combined effects of the strategies implemented helped improve student performance on the NBME psychiatry subject examination over time. Weighting the effects of the individual steps is also not possible, but the process outlined above began with the faculty review of the subject examination. Once this initial step is completed, the changes needed may be addressed in a myriad of ways, not limited to those described in this article.
An understandable concern of such overt attention to the NBME subject examination is an unintended consequence of reduced emphasis on other important competencies. The strategies implemented, however, have not led to additional classroom time or the removal of other key components of the clerkship. Other than the TBL pre-class preparation materials, all of the additional materials involve self-directed learning and are utilized outside of clinical training time. The weight given to the NBME subject examination on the final grade has remained at 25%, consistent with the other clerkships at the medical school. Although the changes did not lead to increased lecture time or grading emphasis for the NBME, the argument remains that the time and efforts used by faculty and residents in these efforts and by the students during self-directed NBME preparation might be better spent addressing other competencies.
With regard to faculty time, as significant changes in the curriculum in the middle of the year might inappropriately benefit students taking the clerkship later in the year, these steps typically only require additional effort from faculty in the weeks leading up to the new academic year. Developing self-directed learning materials can be a time-consuming task, but, once they are completed, they require only minimal updates over the years. With faculty time at a premium and the need to develop expertise in teaching among residents, residents are a potentially untapped resource in the development of such materials.
In summary, a targeted approach to improving student performance on the NBME psychiatry subject examination, starting with an annual faculty review of an actual examination and subsequent efforts to revise the existing curriculum and develop additional self-directed learning materials, appears to have been effective. Although such steps do require additional effort, the time required for faculty to update and develop materials is usually limited to the weeks leading up to the new academic year. Adoption of TBL in place of standard lectures has improved student feedback and appears to have led to more consistent studying during the clerkship. Also, residents interested in education can provide additional resources in the development of new curricular materials. Positive outcomes have included benefits for both faculty and students, including apparent improvement not only on the NBME psychiatry subject examination, but also on the USMLE Step 2.