One of the challenges facing a psychiatry clerkship director is selecting and developing clinical settings to provide medical students with learning experiences that will be most relevant to their future practice of medicine. Increasing focus on reducing inpatient care, maximizing outpatient management, and training the generalist physician have resulted in a move away from the inpatient psychiatric ward as the primary clerkship site and into a variety of outpatient psychiatric settings.
Currently, almost one-quarter of United States medical schools include some form of outpatient experience as part of the psychiatric clerkship (1). Several investigators have begun looking at the educational impact of these experiences. Werkman et al. (2) reported that students on a 6-week, multisite clerkship rated their outpatient experience as "more useful" than their emergency, consultation—liaison, or inpatient experiences. Eagle and Marcos (3) report on a combined in- and outpatient 5-week clerkship, in which 64% of the 80 clerks reported the outpatient portion as "more valuable to them as prospective physicians" than the inpatient portion. More recently, Rodenhauser (4) has described sending 185 clerks to one of several outpatient sites for up to four half-day sessions during a 4-week inpatient clerkship. Although the inpatient and outpatient components were not compared, it appears that most students rated the outpatient experience positively on a dimension that included "interest [and] enjoyment." On the basis of our literature review, it appears that many of the outpatient experiences are part-time and that reported learner satisfaction or other outcome data have been primarily unidimensional or were not compared with other training sites.
Another trend is the growth of the incarcerated population in the United States and the recognition of the substantial mental health needs of this population. Between 1990 and 1998, the jail population has increased by almost 5% annually, and the number of jail inmates per 100,000 U.S. residents has risen from 163 to 219 (5). Although study techniques vary, it is likely that more than 15% of the jail population suffers from diagnosable mental illness, not including personality disorders (6—9). (Note: Jails are "locally-operated correctional facilities that confine persons before or after adjudication. Inmates sentenced to jail usually have a sentence of [1] year or less, but jails also incarcerate persons in a wide variety of other categories" [5]).
There are reports of small numbers of medical students having brief clinical training experiences in general psychiatry in correctional settings. Kaufman et al. (10) describe an elective weekly medical student rotation in the New Mexico prison system chosen by 27 medical students over approximately 5 years. Anecdotal qualitative outcomes were described as positive. Bender et al. (11) describe a program in which, over a 2-year period, 26 medical-student clerks spent 2—3 hours per week at a New York prison during an 8-week clerkship, also with anecdotal positive qualitative learning outcomes.
Nevertheless, the perception is widespread that mental health care in correctional settings is, by definition, substandard; the clinical population is homogeneous; and the work environment is unpleasant (9,12). In the authors' experience, these perceptions contribute to the belief that correctional settings would be poor choices for providing the broad-based clinical exposure desirable in psychiatry clerkships.
This article describes a retrospective analysis of 2 years (AY 1997 and 1998) of comparative student satisfaction and learning-outcomes data from the University of California, Davis School of Medicine psychiatric clerkship, in which medical students spent 4 weeks of the 8-week clerkship fulltime in an outpatient psychiatric clinic in the county jail system. The current authors believe that this is the first presentation in the recent literature of a comparative analysis using multiple outcomes of a fulltime outpatient psychiatric clerkship in the jail setting.
The Jail Psychiatry Service (JPS) is a program of the University of California, Davis (UC Davis) Department of Psychiatry that has provided psychiatric services to inmates in the Sacramento County Jail System by contract with the Sacramento County Department of Mental Health since 1978. JPS provides services to an inmate population of approximately 4,000 men and women through outpatient crisis-intervention services, outpatient clinic services, and an 18-bed inpatient psychiatric hospital, all within the walls of the County's two jails. All eight psychiatrists are members of the UC Davis School of Medicine Department of Psychiatry. UC Davis medical students, psychiatry residents, and forensic psychiatry fellows have rotated through various combinations of the JPS services beginning in the mid-1980s. JPS is accredited by the independent Institute for Medical Quality (13).
There are approximately 15,000 visits per year to the two outpatient clinic sites. Inmates living in the general-population cellblocks come to the clinics by appointment and then return to their cells and usual jail activities. The clinics are staffed by psychiatrists, psychiatric social workers, and psychiatric nurses. The social workers and nurses perform initial assessments and crisis intervention and triage patients for medication management by the psychiatrists.
Medical-student clerks assigned to the clinics spend time with the non-physician clinicians, the PGY-I/II general psychiatry residents who rotate through JPS, the PGY-V forensic psychiatry residents, and the faculty psychiatrists. The students perform initial patient assessments, assist with treatment planning, perform medication evaluations, and see patients for follow-up appointments.
All medical students are required to complete an 8-week psychiatric clerkship, which is divided into two 4-week blocks. The students work at different sites for each block. Students may request site assignments, although some do not, and some students are assigned to sites that were not their top choices. During the study period, July 1997 through June 1999, a total of 199 medical students each rotated through two of the following sites: University hospital consultation—liaison service, county adult inpatient unit, state hospital adult inpatient unit, private children's inpatient unit, combined Veterans Administration/ U.S. Air Force inpatient unit, and outpatient Jail Psychiatric Service clinic (JPS). The number of students assigned to any one site varied from 1 to 15 during the study period, depending on site capacity. Six students did not complete the clerkship during the study period and so were dropped from the analyses, for a final total of 193 students.
Each student completed a clerkship evaluation at the end of the 8-week clerkship. The instrument was developed in the late 1980s by the UC Davis Office of Medical Education, based on a survey of clerkship evaluation instruments used by other U.S. medical schools at the time. The evaluation contained 18 items scored on a 1 (Low) to 3 (High) scale. Seven of the items ask about site-specific educational experiences; the other 11 ask only about the clerkship as a whole. The seven site-specific items were 1) quality of faculty teaching and supervision; 2) quality of resident teaching and supervision; 3) appropriateness of student responsibilities; 4) variety of patients seen; 5) preceptor observation of student clinical skills; 6) feedback and evaluation provided to the student; and 7) usefulness of site orientation. Evaluations were administered, compiled, and reported by the Office of Medical Education. To ensure the confidentiality of the evaluations, site-specific ratings were available only as mean group ratings on each evaluation variable across students at each site for each clerkship block. For this study, we combined ratings of faculty teaching with ratings of resident teaching, since the two separate ratings were highly correlated. The 11 general items were not analyzed.
At the end of the clerkship, all students took the National Board of Medical Examiners' (NBME) Subject Exam in Psychiatry (14) and underwent an individual 1-hour semi-structured case-discussion oral exam conducted by a faculty member. Site preceptors also completed evaluations of the students, which were summarized as a single rating of each student's clinical skills. The oral exam is scored 0 (Fail) to 5 (Honors), and the preceptors' evaluations are scored 0 (Fail) to 50 (Honors).
Our hypothesis was that there would be no differences between jail and non-jail sites. The first set of analyses examined the student site evaluation data. The second set examined the student academic performance data. t1 shows the means and standard deviations for all variables included in the two analyses. All analyses were performed with SPSS for Windows, Version 9.0.
+
Students' Evaluations of Clerkship Sites
Sixteen groups of students completed evaluations on 4-week clerkship blocks at the JPS sites, and 48 groups of students completed evaluations on 4-week clerkship blocks at non-jail sites. Three evaluations from the JPS site and 17 evaluations from the non-jail sites were missing, so the analyses included 41 evaluations of the jail site and 325 evaluations of the non-jail sites. Before analysis, in order to accurately adjust for the different number of students in each group, we weighted the data by the number of students in each group.
A one-way, between-subjects multivariate analysis of variance (MANOVA) was performed on the six dependent variables of the students' evaluation of clerkship sites listed in t1. The independent variable was type of clerkship site. The unique sums-of-squares method was used to address the discrepancy in cell sizes for jail vs. non-jail groups. Pillai's criterion was used for the inferential test because of its robustness regarding violation of the assumption of homogeneity of variance—covariance matrices, which often occurs when cell sizes are unequal. t2 presents the MANOVA results. The combined evaluation variables did not significantly vary between jail and non-jail clerkship sites. The strength of association between the type of site and the combined evaluation variables was small, with η2=0.03.
We then further investigated the nature of the relationship between the type of site and the student site evaluation variables. Because correlations for the evaluation variables were generally in excess of 0.30, step-down analysis was indicated to adjust for the risk of Type I error implicit in multiple analyses. t2 shows the results of the Roy-Bargmann step-down analysis, listing the variables in the order in which they were entered. The univariate F-test results, also in t2, assist in correctly interpreting the pattern of the step-down results. None of the univariate analyses reached significance. In summary, only the last two of the step-down F tests, "evaluation/feedback" and "usefulness of orientation," showed any significant differences in students' evaluation of jail vs. non-jail sites at the 0.05 level. The jail sites received higher ratings for "evaluation/feedback" and "usefulness of orientation" than non-jail sites, as shown in t1. Both the multivariate and univariate F tests showed no significant differences. The "type of site" variable explained only 3% of the variance in the combined student evaluation variables.
+
Clerkship Site and Student Grades
The individual clerkship grades for students who rotated through the JPS clinic (n=44) were then compared with the grades of those who only clerked at non-jail sites (n=149).
A one-way, between-subjects MANOVA was performed on the three dependent variables of student grades: 1) NBME scores; 2) clinical skills grade; and 3) oral exam scores. The independent variable was type of clerkship site. Although there were no univariate outliers in the data, NBME scores were significantly negatively skewed for the non-jail group (z=3.49; P<0.01). Because calculation of Mahalanobis distance detected multivariate outliers involving the NBME variable (P<0.001), we performed a reflected square-root transformation on NBME, bringing its skewness within an acceptable range. After transformation, no multivariate outliers remained for any of the groups. As with the first analysis, the MANOVA used the unique sums-of-squares method and Pillai's criterion. t3 presents the MANOVA results. The combined student grade dependent variables were not significantly affected by type of clerkship site. The strength of association between the type of site and the combined performance variables was small, with η2=0.00.
Again, although the results of the MANOVA were not significant, univariate analyses were performed to explore the effects of clerkship site on the individual performance variables. Similar to the student evaluation data, the student performance data demonstrated moderate multicollinearity in both jail and non-jail groups, so step-down analysis was performed. t3 shows both the results of the Roy-Bargmann step-down analysis, listing the variables in the order in which they were entered, and the results of the univariate analyses. In summary, none of the univariate or multivariate F-tests showed any significant differences (at the 0.05 level) in individual students' grades between students who had clerked at jail outpatient clinic sites vs. students who clerked only at non-jail sites. Type of site accounted for essentially none of the variance in the combined performance variables.
This study shows that a medical student clerkship in outpatient psychiatry can successfully be conducted in a correctional setting over time, with the same amount and quality of learning as in other, more traditional, non-correctional settings. In comparison with earlier studies of medical student education in correctional settings, these findings hold for two different multidimensional measures of learning: student evaluation of teaching quality and student performance.
Student evaluation of teaching is widely regarded as an important and valid measure of teaching effectiveness and is highly correlated with student learning (15,16), although much remains to be learned about the educational process and outcomes in ambulatory clinical settings (17). The measure used in the current study, although not independently validated, contains elements similar to those validated in other studies of clinical teaching in ambulatory and inpatient settings (18,19). For example, our variable, "teaching quality," is very similar to each of the two global ratings of teaching that Marriott and Litzelman (20) found correlated highly with the well-validated 26-item Stanford Faculty Development Program for Clinical Teaching.
In the step-down analysis, two variables did show significant differences between the jail and non-jail sites: 1) feedback and evaluation provided to the student, and 2) usefulness of site orientation. These differences were nonsignificant in the univariate analyses. This significance discrepancy suggests that the findings of significance may be a statistical artifact of proceeding with step-down analysis of a nonsignificant MANOVA, although other explanations may be possible. At the jail sites, students work with and receive feedback from a variety of non-physician mental health professionals in addition to psychiatrists, more so than at other sites. This difference could possibly contribute to a valid difference in feedback ratings between sites. Relative to the other clinical sites, the students may experience the jail setting as more foreign, unfamiliar, or intimidating, and thus find a formal, guided orientation more valuable. This difference could possibly contribute to a valid difference in orientation ratings between sites.
The measures of student performance used in the study are multidimensional, reflecting student knowledge, clinical skills, and conceptual integration. The performance evaluations were conducted at the end of the two-block, 8-week clerkship, and so measure knowledge, skills, and attitudes acquired over experiences at two clerkship sites. Although the possibility of a compensatory, remedial learning effect of the non-jail experience on students clerking at the jail site cannot be excluded, the results show that the jail experience does not have an overriding detrimental effect on student performance. The current study did not aim to show any superiority of learning at the jail site.
Also, all students who clerked at the jail site could compare that experience with their experience of clerking at their other, non-jail, site. This feature of the study enhances the face validity of the student teaching evaluation ratings and suggests that if there were undetected student performance differences between the sites, one would expect these to be reflected in the analysis of the teaching evaluations. This cross-over feature also enhances the finding of similarity in variety of patients seen in jail and non-jail settings, belying the misperception of the homogeneity of this correctional psychiatric patient population.
Students were not completely randomly assigned to clerkship sites, so the possibility of individual student characteristics affecting the results cannot be excluded. The inherent cross-over design of the two-block clerkship should help minimize any such effects, however. Because we were limited to mean group data for the analysis of student evaluations, it was not possible to directly correlate individual student teaching evaluations with individual student performance. The use of group means data for the analysis of the student site evaluations may have obscured more detailed findings; future studies might benefit from using individual student evaluations while maintaining the confidentiality of the students' reports. Nonetheless, given the overall robustness of the findings, the impact of the use of mean data is not likely to be large. Finally, the current study may be limited in its generalizability to other institutions. The UC Davis Jail Psychiatric Service is well established and well integrated into the teaching mission of the Department of Psychiatry, and other institutions with less academically-oriented correctional settings may achieve different results.
In conclusion, this study shows that medical student clerks who spent half of an 8-week psychiatry clerkship on a county jail outpatient psychiatric service experienced the same quality of teaching, the same level of clinical responsibility, the same patient variety, and the same level of clinical observation as they and their colleagues experienced in other more traditional clerkship settings. Similarly, there were no differences in student performance on standardized multiple-choice exams, semi-structured oral exams, or on preceptor clinical ratings. Further studies may show whether there are indeed specific advantages to clerkship training in this correctional setting.
The work in this article was presented at the Association of Directors of Medical Student Education in Psychiatry, in Santa Fe, NM, June 15—18, 2000.