For the vast majority of students in the United States and Canada (1), the NBME Psychiatry Subject Exam (PSE) accounts for a substantial percentage of the overall Psychiatry clerkship grade, and, often, a minimum score is required to pass (2). Although performing well on the NBME subject exams is important for clerkship grades, there is also evidence to suggest that studying for shelf-exams is beneficial for long-term consolidation of memory and performance on Step 2 CK (3).

Although the importance of preparing for NBME examinations has been established, there remains a high degree of variability in the methods by which students and educators alike accomplish this task. Briscoe et al. (4) found that most students on their psychiatry clerkship rely heavily on “step” or “prep” books, and that these review materials “did not demonstrate significance as a superior preparation resource for the psychiatry subject exam (4).” Effective curriculum change appears to be the most established and reproduced positive predictor of score improvement and therefore higher performance on NBME exams (5–7).

A formalized PSE review session and accompanying handout was offered to students on their psychiatry clerkship in an effort to provide an effective and efficient intervention. The purpose of this study is to assess the extent to which this intervention improved group performance on the NBME PSE.

This study was designed with the goal of providing psychiatry clerkship students with a summative review of the material covered during their rotation. It was approved by the Institutional Review Board (IRB) under the “Educational Exempt” category. The session was led by a senior adult psychiatry resident with previous teaching experience. It was allocated 1.5 hours and was given 3 days before the PSE. Attendance was optional.

Creation of the Review Handout

A four-page “fill-in-the-blank” document was prepared in congruence with the NBME PSE Content Guidelines (8). The majority of the handout compared and contrasted DSM-IV criteria for all psychiatric diagnoses. Also covered were medication uses and toxicities, substance intoxication and withdrawal, medical illnesses with comorbid psychiatric features, and neurological disorders. Answers to the review questions were gathered from the students’ assigned text (9). Students actively participated by answering questions in a clockwise fashion and were encouraged to discuss relevant cases.

Flow

Although the monthly review session was offered to a total of 62 students from May through August of 2010, only 38 students chose to come to the review session, whereas 24 did not. Those who did not were e-mailed the blank review handout with accompanying answers. All 62 students offered the review were considered part of the intervention group, regardless of attendance. Interestingly, the percent of students attending the review session each month steadily rose, from 31% to 56%, 67%, and finally 93% in May through August.

The control group was composed of 47 students who took the PSE 1 year before this experiment. The difference in number between the experimental and control groups was due to random, year-to-year variation in class size and number of students per clerkship.

Data Collection and Intervention Implementation

The students who attended the review signed a consent form and were assigned a subject identification number as they were given a survey. However, the results of this survey were not significant and therefore have not been included in this paper. The IRB did not require that non-attenders sign consents, given that they were not assigned a subject identification number and did not fill out a survey, and individual scores were not tracked. The study coordinator was blinded to subject responses.

Data Analysis

Means of PSE raw scores and percentiles for each group were calculated; however, only raw scores used as percentiles showed a skewed distribution to the right. High scores, low scores, range of scores, and standard deviation/error were also calculated for all groups.

ANOVA was performed for the intervention, with and without covariate analysis. Group means were compared for significance and effect size. Given that the intervention was implemented between the months of May and August, two academic classes were involved in both the experimental and control groups. Covariate analysis was necessitated by the presence of two academic classes.

Presence of Intervention Effect Without Accounting for Potential Confounds

The PSE mean raw score of the 62 students who were offered the review in 2010 was 84.53, versus 77.15 for the 47 students who were not, in 2009. This finding was statistically significant, with a p value of <0.0001. The effect size for the compared groups was 0.89. No differences in high scores, low scores, or range of scores was detected.

Presence of Intervention Effect With Additional Analysis Accounting for Potential Confounds

The mean intervention scores were then adjusted, using academic class as a covariate, and the result remained significant (85.60 versus 75.89, with a significant p value of 0.0002 and an effect size of 1.17).

This study may suggest that offering a comprehensive review session to third-year medical students 3 days before their NBME PSE significantly improves their scores. The covariate analysis of academic class adds to the strength of this study, as doing so accounts for potential class differences in test-taking ability, various changes in curriculum (including lectures, rotation sites, attending and resident supervision, etc.), changes in the exam itself, and any other potential class differences that could have falsely contributed to an intervention effect. The finding that no significant separation from control was found in high scores, low scores, and ranges of scores, implies that improvement was evenly distributed and did not significantly select for high-achieving or low-achieving students. Of note, a larger percent of students attended each month, as reported earlier. Given that there was no increase in exposure or encouragement from faculty members or residents, the likeliest explanation is that enthusiasm spread among the student body by word-of-mouth.

A comparison of the scores within the intervention group of students who attended the session with those who did not and were e-mailed the materials is the current focus of a future analysis. Initial data may suggest that both the lecture and the e-mail subgroups independently performed significantly better than controls (85.47 and 82.20 versus 78.35, for p values of <0.0001 and 0.012, respectively). It also appears at this time that the lecture subgroup significantly outperformed the e-mail subgroup (p=0.0388). However, a full comparison is not possible at this time.

There are many potential limitations of this study. First, it was only performed in only one setting, and, therefore, the results of this study may not necessarily be reliably reproducible in other academic settings. Second, the number of subjects could have been more robust, so as to decrease an unintended selection bias. Similarly, the control group was much smaller than the experimental group, and this may have caused the data to be skewed. Third, the study was not carried out throughout an entire academic year, which may again be contributing to a false selection bias and does not allow for time-of-year analysis. Fourth, as exit surveys were not collected and the control group was not accessible, we do not know if students changed their preparation habits or spent more time studying than did controls. Last, we do not know specifically why this intervention worked. It could have been due to a number of factors, including consolidation of memory, comfort in asking questions of a resident, the skill of the senior resident leading the session, prioritizing of information, the review of medical and neurological conditions, additional clinical pearls from cases, or any number of other reasons.

In conclusion, despite its limitations, this study supports the use of a review session and accompanying handout as a form of curriculum intervention aimed at improving PSE scores. These results will need to be replicated at other institutions for improved reliability, and a more sophisticated understanding of the interaction between shelf reviews and PSE scores is needed.