With the rapid pace of advancement in psychiatric knowledge, it can be challenging for clerkship directors to select the best topics and teaching methodologies. Expert consensus guidelines (1, 2), specialty workgroup reports (3), and organized medicine (4) provide valuable curricular content guidance. To complicate matters, anecdotally clerkship directors have noticed that students do not always follow their guidance regarding learning resources. Nonetheless, the question remains as to which teaching methods and learning vehicles provide the greatest knowledge mastery in psychiatry clerkship students. This is a very relevant question, and while there are myriad approaches, there is little published research to guide those interested in optimizing knowledge gain in their students.
A range of 69%–84% of psychiatry clerkship directors in the United States and Canada use the National Board of Medical Examiners (NBME) Psychiatry Subject Exam (5, 6). This is a 100-item, multiple choice, end-of-clerkship testing instrument whose advantages include that it is expert-reviewed, current, well-written, and validated. Scoring is based on national norms, and the covered subjects represent information students will be expected to master in order to pass their United States Medical Licensing Examination (USMLE) Step II Clinical Knowledge tests (6). Its use may also mitigate effects of clinical grade inflation (7).
The NBME Psychiatry Subject Exam (PSE) may have some limitations. Studies are mixed as to whether exam scores increase as the clerkship year progresses (8–12). In addition, the PSE is not recommended as the sole basis for a judgment of program effectiveness (13) and there are no other standardized testing instruments available from which to choose. Finally, because the PSE is generally not considered a good measure of skills or professionalism (6), it will never replace comprehensive clinical assessments.
Nonetheless, given its important advantages, the PSE is the most commonly used standardized testing instrument for assessing scholarly knowledge in U.S. psychiatry clerkships. The purpose of this study is to investigate the learning materials and teaching methods used by clinical clerks in psychiatry which result in the highest rate of knowledge retention, as reflected by the scores on the PSE, and overall learning experience.
We developed a list of clerkship students from six geographically disparate U.S. medical schools who had recently completed their clerkship in psychiatry from January through April 2007, which included assessment via the PSE. These students were sent an e-mail asking them to consider participation in a voluntary, anonymous, 16-item web-based survey to investigate study materials and methods used for PSE preparation (e.g., textbooks, lecture handouts, teaching techniques encountered, and other relevant factors which might have influenced their PSE scores). The survey included a question requesting the student’s PSE subject test score (not the same as the percentile or percent correct) and USMLE Step I score.
A total of 489 students were invited to participate in the study. Of this group, 471 were e-mailed a second invitation within 2 weeks of the first. The total population of third-year medical student classes at these six institutions was 1,640; therefore, approximately one-third (30%) of the third-year class at each institution was sampled. Of the 489 students invited, 151 completed this cluster sampling type survey, yielding an adequate response rate of 30.9%. Clerkship lengths varied from 4–8 weeks. Based on data from the Association of American Medical Colleges (AAMC) (14), we categorized the participating schools as small [total school enrollment in smallest third of all schools and below 1 standard deviation (SD)], medium (middle third and size within ±1 SD range), and large (largest third, >1 SD); our study included one small, four medium, and one large school participants.
A “step-or-prep” book was defined as a book residing in the category of “Test Preparation and Review” at Amazon.com. Typically these books explicitly state that they are intended to serve as a study aid for the PSE, are usually written in an outline format, contain “essentials” (at the expense of comprehensiveness), and are shorter than conventional texts.
Statistical analyses were performed using SPSS 15.0 for Windows (SPSS, Inc.) and Microsoft Excel 2007 (Microsoft, Inc.) and included the following statistical categories: descriptive, difference inferential, and associational inferential. Approval for research was granted by the institutional review boards of all respective participating institutions.
+
How Students Prepared for the PSE
Students were asked, “Which of the following did you read prior to taking the NBME Psychiatry Subject Examination (check all that apply)?” The most frequently reported reading materials were step-or-prep book(s), used 84.7% of the time. Handouts, assumedly from instructors, were also popular, with over 60% of students using them. Assigned texts were utilized less than 40% of the time. Journal articles were infrequently used (14%).
Students were asked which of 37 currently available psychiatric textbooks (by our count) they used to prepare for the PSE and given the instructions to “check all that apply.” The top six were step-or-prep books as opposed to a conventional textbook. Out of the top 10 titles only one was not classified as a step-or-prep book. Three percent of respondents reported not using any textbooks.
In their preparation for the PSE, 84.6% reported using practice questions, and in many cases these may have come from step-or-prep books. Only 8.8% did not report using this resource type, and 6.8% did not respond to this question.
The students were surveyed on a range of topics regarding their preparedness for the PSE. For example, they were asked, “On a scale of 1 to 5, how prepared were you for taking the subject examination? (1=not at all prepared; 5=highly prepared).” The average score was 3.8 (SD=0.81, skew=−0.37). Second, the students were asked, “Looking back, I would rank the following components in order of importance in preparation for the subject examination (1=most important, 7=least important).” They ranked books as most important (mean score 2.2, SD=1.6), and then practice questions (mean 2.6, SD=1.8), didactics (mean 3.4, SD=1.6), discussion with faculty outside of lectures (mean 3.5, SD=1.5), multimedia focused on the psychiatric material (mean 4.9, SD=1.7), recommended journal articles/additional reading (mean 5.0, SD=1.7), and other (mean 5.3, SD=2.2), respectively. Third, students were asked, “Which of the following would have helped you to improve your preparation for the subject examinations (pick the top 3).” Students ranked “more time to read/study” as the most important (32.7%, n=49), followed by “clinical-faculty-teaching discussion” (25.8%, n=39), “didactic coverage” (20%, n=30), and “team-based learning” (15.3%, n=23). Team-based learning is a technique that “allows a single instructor to teach through conducting multiple small groups simultaneously in the same classroom” (15). This technique was formally utilized in two of the six schools studied.
+
Most Successful Strategies
The students self-reported their PSE test scores, which were cross-referenced to their method of preparation. The students were told to “check all [methods] that apply,” because any one student may have used any combination of reading materials. Eighty-nine percent of students used more than one method to prepare for the PSE. No one method or combination of methods demonstrated significance as measured by analysis of variance (ANOVA) (p>0.05 in each case).
Seventy-six percent of students reported exposure to more than one didactic method (average of 2.1 methods). We compared students’ PSE scores to the didactic methods they encountered. Using ANOVA, the two didactic categories topic-based lectures and team-based learning, and the combination of case discussion/topic-based lectures, did demonstrate significance. Topic-based lectures increased the students’ average score from 75.42 to 81.72 (F=7.024, df=1, 86, p<0.05), team-based learning increased the average score from 78.82 to 81.83 (F=9.475, df=1, 86, p<0.01), and the combination of case discussion/topic-based lectures increased the average score from 72.33 to 80.30 (F=5.584, df=1, 86, p<0.05).
The top quartile of students (PSE test score ≥81, as per NMBE) reported using the same most frequently used step-or-prep book as the bottom quartile (PSE test score ≤70)—Case Files Psychiatry (16). Four of the five top books used by the top quartile were also listed in the favorites of the bottom quartile.
The average number of books used to prepare for the PSE was dissimilar between groups: 1.7 books for the bottom quartile (SD=0.9; skew=−0.3) and 2.7 books for the top quartile (SD=1.6; skew=0.7). We found that using book combinations was the norm for both quartiles, reported by 63.6% of the bottom quartile (SD=0.9) and 77.4% of the top quartile (SD=1.6); however, the bottom quartile group did not report using any of the top book combinations used by the top quartile.
We compared respondent PSE scores to USMLE Step I total scores, for those who reported these values (73.5%, n=111). There was a modest positive correlation of 0.343.
We suspect several attributes of step-or-prep books contribute to their popularity among students (84.7% of students reported their use). First, the books’ easy-to-read, “essentials-only” outline format allows rapid digestion of large amounts of information once or possibly twice during the rotation. This may make the task of the psychiatry overview seem less ominous and alleviate student anxiety. Second, despite lack of verity, students may believe these books lend themselves to mastering factual scholarly material and consequent success on standardized examinations such as the PSE. Anecdotal success stories involving these types of books may have spread, reached critical mass, and became a part of the students’ collective unconscious. Third, given that these books are typically shorter and often cheaper than other more comprehensive learning resources, it is possible that students are simply choosing the easiest path in PSE preparation without regard to long-term learning benefits. Last, students may see various combinations of these factors too seductive to forgo.
Our finding that 65% of students utilize “handouts” to prepare for the PSE is not surprising, as these materials represent concise, distilled expert guidance from their clerkship faculty. However, we wondered why students utilized the book assigned by the clerkship directors less than 40% of the time. It may be that clerkship directors prefer to prescribe more lengthy and detailed readings than what is found in the step-or-prep books, but that students often settle for the merits of the latter category instead. This raises interesting questions, such as what precisely might the students be missing in this compromise? Surely leading contemporary psychiatry textbooks have essential depth and breadth of content which is being sacrificed in this process.
It was both interesting and important to note that while in the preparation phase for the PSE students used practice questions at essentially the same high rate as step or prep books (84.8% and 84.5%, respectively), but after taking the PSE students ranked practice questions as less important than step-or-prep books. The recommended journal articles and additional readings were rated with low importance, perhaps because these resources were perceived as too advanced or superfluous in nature, especially given the already burdensome knowledge base for which students are responsible.
Given the data that students feel that books were the most valuable learning resource for the PSE, it is hardly surprising that the highest rated preparation step to improve student performance was “more time to read/study.” We find this commensurate with anecdotal comments from students. Students also noted that more “clinical-faculty-teaching discussion” and “didactic coverage” would be helpful in their PSE preparation, though somewhat less so than reading time. Consideration of these findings, while not yet replicated, may have relevance when developing clerkship structure.
+
Most Successful Strategies
While step-or-prep books were the most popular study material, their use did not result in significantly higher Psychiatry Subject Exam (PSE) scores for students in our study. In fact, no one preparation material type or combination of materials proved superior, as the ANOVA produced all nonsignificant finding (p>0.05 in each case). The fact that students who utilized journals had the second highest average PSE score may reflect extraneous factors, as discussed below.
On ANOVA the didactic types topic-based lectures and team-based learning, as well as the combination of case discussion/topic-based lectures, demonstrated significance (p≤0.05 in all cases), indicating that their use did result in significantly higher PSE scores. Given that 95% of respondents were exposed to either topic-based lectures or team-based learning, most students had equal opportunity to excel on the PSE, obviating the need for didactic structure change in the clerkships studied. It is speculated that conventional topic-based lectures are very common in psychiatric education, and student participants benefit similarly. Team-based learning may be less common, but participants nonetheless share these benefits. Moreover, the heterogeneous didactic approach (76% surveyed reported exposure to more than one didactic method, n=114) may confer educational advantages not studied herein.
Given that the top quartile of students used more books and different combinations of books on average than bottom quartile students, it may be that the highest scoring students have a special skill in choosing the right combination of books and/or mastering knowledge contained within a combination of books.
There are limitations of this study. First, certain factors were not studied, such as the degree to which quality of clerkship teaching and/or the innate attributes of students —native intelligence, work ethic, reading comprehension skills, test taking skills or any combination thereof—play in PSE scores success. The modest positive correlation of PSE scores with USMLE Step I total scores suggests that such attribute independent variables may influence PSE scores. Yet since they probably impact PSE success, they certainly deserve further study. Second, while the findings are generalizable to the institutions studied, they may not generalize to the body of all U.S. medical schools, as such a scaled comparison exceeded the scope of this study. Third, it is possible that the study window, January through April, included students whose study habits differed significantly from their peers, but prior studies are mixed as to the existence of differences in PSE performance due to the clerkship timing effect (8–12).
Potential weaknesses of the study include the fact that the students’ PSE scores were self-reported (i.e., unconfirmed). Second, student responses regarding the adequacy of their preparation, as well as the value and importance of individual learning resources, are subject to the inaccuracies found in self-assessment of adult learners (17–19). Last, retrospective study design carries its own potential liabilities (20).
In conclusion, the Psychiatry Subject Exam, taken at the end of the third-year psychiatric rotation, is an important and valuable standardized testing instrument for students and educators alike. Students exhibited several trends in their preparation for the PSE, and these trends have important implications for medical student education. Topic-based lectures and team-based learning, as well as the combination of case discussion/topic-based lectures, demonstrated significance in preparing students for the PSE. Students should be aware that while step-or-prep books have certain positive qualities, they did not demonstrate significance as a superior preparation resource for the PSE. Moreover, while not specifically explored in this study, it is possible that if students eschew the more comprehensive, rigorous, and sophisticated conventional learning resources, it is possible that their clinical competence in general psychiatry may suffer more subtle deleterious effects (e.g., the limitations and error susceptibility in those with a superficial understanding of a field). Educational guidance from seasoned psychiatric educators vis-à-vis balanced eclecticism may prove invaluable for contemporary clinical clerks in psychiatry.
Data contained in this article were presented at the Annual Meeting of the Association of Directors for Medical Student Education in Psychiatry, Park City, Utah. Approval for research was granted by the Institutional Review Boards of all respective participating institutions.
Dr. Pelic was previously on the Speaker’s Bureau for Pfizer and Eli Lilly. Drs. Briscoe, Levine, Carlson, and Al-Mateen disclosed no competing interests.