The Objective Structured Clinical Examination (OSCE) is now widely accepted as a valid and reliable method to assess medical students’ clinical skills (1–4). Although the use of OSCEs in psychiatry has been described as less rapid and widespread than in other medical fields (1, 4, 5), in recent years there has been increased interest in its use in psychiatry (1, 4). Previous analyses of the OSCE have focused on reliability (6), validity (6–9), fairness and acceptability (6), generalizability across content domains (3), use of psychiatric OSCEs in medical settings (5), the impact of the order of clerkships or so-called “timing effect” (8), and the benefits (1) and challenges (4, 5) inherent in its use. This study focuses on the impact on OSCE performance of having taken the psychiatry clerkship.
At Harvard Medical School, scheduling in 2005 required over half of fourth-year students to take their comprehensive OSCE, including the psychiatry station, before completing their core psychiatry clerkship. All students had completed a 12–14 week second-year introduction to psychopathology course, and most had completed a majority of their other core rotations. Given the multiple anticipated contributions to OSCE performance, we were interested to know whether OSCE performance on the psychiatry station would, as hypothesized, be superior for students who had completed their core psychiatry rotation before taking the OSCE compared with those who had not yet taken or completed this clerkship and, if so, in what areas performance would significantly discriminate between clerkship completers and noncompleters.
As part of their required comprehensive examination, 136 Harvard Medical School students took part in a nine-station OSCE with one psychiatry station in August 2005. Of these, 63 students (46%) had already completed their core psychiatry clerkship, and 70 (51%) had not. Three students who did not answer this question (2%) were excluded from the analysis. This project was reviewed by the Harvard Medical School Committee on Human Studies and identified as exempt from institutional review board review because the research presented no more than minimal risk to participants and involved no procedures for which written consent would normally be required outside of a research context. All students had previously taken a 3-month psychopathology course in their second year, consisting of lectures alternating with clinical site visits, patient interviews, and write ups.
Students were told that the setting of the case was the emergency department. Each of the standardized patients, who had received several hours of training on the case, reported that he was a 24-year-old man experiencing sudden chest tightness, tingling of hands and lips, with his heart “beating like a jackhammer.” The standardized patient expressed concern about dying of a heart attack, fear that he was “going crazy,” and a desire to be admitted to the hospital for medical evaluation despite having been cleared medically.
Within the 15 minutes for the case, students were to take a personal and family history, perform a mental status exam, and present the case to the observing faculty examiner, including a diagnosis (thought to be panic disorder), rule-outs, and recommendations for immediate triage/intervention, including additional laboratory studies for medical differential diagnosis and recommendation about the appropriateness of psychiatric hospitalization. All faculty members involved were experienced OSCE examiners trained on this specific case. Each rated the student with a “did/did not do” checklist (10) that included 34 items on history taking, mental status exam, differential diagnosis, and treatment recommendations. Additionally, the standardized patients rated the students’ communication skills on a 23-item instrument using a 5-point scale from poor to excellent. The 23 items were used to calculate 14 communication scores (two in each of seven core competencies), and these 14 scores were then added to the previous 34 items to create a maximum score of 48. Means for each of the 48 items were computed for those who had completed the clerkship and those who had not, and t tests were performed to compare means between the two groups.
The mean score on the station for all students taking the exam expressed as a percentage of 100 was 55.00 (n=133). The mean score for those who had a clerkship was 59.82 (n=63). The mean score for those who had not yet completed a clerkship was 50.77 (n=70). Thus, those who had completed a psychiatry clerkship scored significantly higher than those who had not yet completed the clerkship (p<0.001).
The performance evaluation consisted of 34 scored items for each student. On nine items, completers and noncompleters differed at or less than the 0.05 level of significance in the percentage of each group getting the correct answer (Table 1).
Six of these nine items evaluated knowledge of mental status and phenomenology: appearance and behavior, quality of thinking, homicidal ideation, depersonalization-derealization, hallucinations, and delusions. Three of these items concerned differential diagnosis: determining that the patient was not psychotic, stating why the patient was not psychotic, and ruling out generalized anxiety disorder.
There were no significant differences between the two groups regarding identification of suicidality, depression, panic symptoms, substance abuse, or other assessed aspects of patient history and mental status examination. There also were no significant differences between the groups regarding overall communication skills as rated by the standardized patients, nor on any of the six communications subskills (e.g., building a relationship, gathering information, understanding the patient’s perspective). Finally, there were no significant differences between groups regarding recommended diagnostic investigations and management (e.g., safety assessment, reassurance and support, toxicology screening).
In this study, performance on the OSCE exam early in the fourth year was compared between medical students who had completed their 4-week psychiatry clerkship and those who had not. Consistent with our hypothesis, clerkship completers demonstrated superior performance on the exam. The overall mean OSCE score based on a binary checklist was approximately 18% higher for completers than noncompleters. Items that discriminated between the two groups included aspects of the mental status exam and differential diagnosis, such as more explicit addressing of the presence or absence of psychotic and dissociative symptoms and homicidal ideation.
Nevertheless, although these differences between clerkship completers and noncompleters were significant, they are modest. The two groups did not differ significantly in their performance on most items assessed by faculty examiners, including correct identification of the most likely diagnosis; inquiry about suicidality, depression, substance abuse, family history, and past medical history; consideration of further testing; and general plans for management and disposition. In addition, the two groups were similar in ratings by the standardized patients on doctor-patient communication. In this respect, OSCE performance appeared only inconsistently related to whether students had taken the psychiatry clerkship prior to the exam and supports previous literature (1, 8) suggesting a marginal influence of the clerkship on OSCE performance.
The absence of greater differences on OSCE performance between students who had and who had not completed the psychiatry clerkship prior to the OSCE may be related to redundancy built into the medical school curriculum, which progressively reinforces and extends core knowledge and skills, including those relevant to general evaluation and management of the psychiatric patient. These would include interviewing skills, history taking, overall assessment of functioning, triage, and general doctor-patient communication skills. As at other medical schools, patient-doctor interviewing skills are introduced in the first year in the Patient Doctor I course and further refined in the Patient Doctor II and Psychopathology and Introduction to Clinical Psychiatry courses. The latter course, given in the second year, introduces the core features of psychiatric disorders and their treatment, as well as the mental status exam as related to patients with psychiatric disorders. The course also includes up to five patient interviews observed by faculty and up to six sessions involving patient interviews with required student write ups.
The core psychiatry rotation helps consolidate learning from the second year while providing an opportunity for more intensive exposure to patients with psychiatric disorders and enhancement of knowledge of psychiatric assessment, differential diagnosis, and treatment modalities. Our study suggests that several areas of psychiatric skills and knowledge can be acquired not only in the psychiatric clerkship but also in several preclinical courses and nonpsychiatry core clerkships. Such areas include past medical history and medical-psychiatric differential diagnosis, the latter being especially important in this case, as the standardized patient was instructed to portray himself as believing that he had a medical illness and in fact needed medical admission, despite having been medically cleared before being seen by psychiatry. However, more sophisticated types of assessment, such as the presence or absence of psychotic and dissociative symptoms and homicidal ideation, are more uniquely taught within the psychiatry clerkship.
Another reason for the relatively modest impact of the psychiatry clerkship on OSCE performance may be related to the use of checklist ratings (9, 10), which reflect a minimum standard of performance but do not distinguish among levels of performance. All or none credit for inquiring about alcohol use, for example, would not differentiate a student who simply asked “Do you drink?” from a student who pursued a more detailed inquiry. Although the OSCE identified significant deficiencies in knowledge and skills requiring remediation, as it was intended, the scoring system used in this or other typical OSCEs does not allow us to discriminate between adequate and advanced performance. To achieve greater separation between clerkship completers and noncompleters, it seems likely that the OSCE would require a more graduated assessment scale coupled with clinical vignettes requiring more sophisticated knowledge and skills, such as the evaluation of dangerousness or the differential diagnosis of unipolar versus bipolar mood disorders, psychosis versus dissociation, or dementia versus delirium, as recommended by Vaidya (1) and Park et al. (8, 9).
Our study is limited in several respects. We did not seek to demonstrate a validation of the OSCE through comparison with other standardized and qualitative assessments, such as shelf exam scores or clerkship grades, for psychiatry clerkship completers. Also, although faculty examiners were asked to inquire about clerkship completion only after completing their scoring of OSCE performance, they were not truly blinded as to whether a student had completed the psychiatry clerkship. In addition, the psychiatry clerkship at our medical school is a 4-week rotation. It is possible that a greater difference in OSCE scores between completers and noncompleters would have emerged following a longer clerkship experience, even though United States Medical Licensing Examination data do not show a consistent relationship between psychiatry clerkship length and shelf exam performance.
In our analysis, we did not seek to separate out the influence of having completed other clerkships on OSCE performance. It seems likely that knowledge and skills acquired in the context of other rotations, such as medicine, pediatrics, and primary care, contributed to performance on the psychiatry OSCE, particularly for patient-doctor communication items, as well as for differential diagnosis and some other aspects of patient assessment. This knowledge and skill transfer across content domains is referred to as “generalizability” elsewhere in the literature (3, 11). Nevertheless, the great majority of students in our sample had completed most or all of their other clerkships, whether or not they had completed psychiatry, so the impact of these other clerkships should be similar between the two groups. We also did not look at the timing of the psychiatry rotation with respect to the date of the OSCE; possibly some students benefited simply from recent exposure to psychiatry prior to the exam.
Finally, the group of students who deferred their psychiatry rotation until after the OSCE exam may include those students least likely to be considering psychiatry as a possible future career choice. We cannot exclude the possibility that their somewhat poorer performance on the OSCE may be indicative of lower motivation for the subject area rather than lack of exposure to the psychiatry clerkship.
In conclusion, our experience with the psychiatry OSCE demonstrated a significant but relatively modest positive effect of psychiatry clerkship completion on examination performance. This result reflects a combination of factors, such as the impact of preclinical didactic courses, the nature of OSCE data as confirming a minimum level of proficiency rather than discriminating between levels of achievement (10), and the effect of clinical work on rotations other than psychiatry upon OSCE performance.
The authors would like to thank the following for their valuable contributions to this OSCE station: the faculty examiners from the Harvard Medical School Department of Psychiatry; Dr. Cynthia Kettyle, Director Emerita of Medical Student Education in Psychiatry at Harvard Medical School; and the Harvard Medical School OSCE Steering Committee.
Dr. Goisman was a consultant for Solvay Pharmaceuticals, Inc., in December 2007. At the time of submission, Drs. Levin, Krupat, and Pelletier reported no competing interests. Dr. Alpert has provided full disclosure from several public and private sources that are available upon request.