Advanced Search

BRIEFREPORT | July 01, 2006

Inter-Site Consistency at a Multi-Site Psychiatry Clerkship

Margo Shultes von Schlageter; EunMi Park; Phebe Tucker

Academic Psychiatry 2006;30:356-359.

View Article Information

Received August 27, 2005; revised December 5, 2005; accepted February 2, 2006. Dr. Shultes von Schlageter is Volunteer Clinical Assistant Professor, Department of Psychiatry, University of Oklahoma, and Staff Psychiatrist, Oklahoma County Crisis Intervention Center, Oklahoma City, Oklahoma. Dr. Park is Assistant Professor of Medicine and Associate Director of Education, Geriatrics and Gerontology, John's Hopkins School of Medicine, Baltimore, Maryland. Dr. Tucker is Professor and Vice Chair of Education, Department of Psychiatry, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma. Address correspondence to Dr. Tucker, University of Oklahoma College of Medicine, Dept. of Psychiatry, PO Box 26901 WP 3440, Oklahoma City, OK, 73190; phebe-tucker@ouhsc.edu (E-mail). Copyright © 2006 Academic Psychiatry.

Article

Tables

References

text A A A

Abstract

Objective: This study examines the effects of clinical site assignment within a multiple-site psychiatry clerkship program on the convergent outcome of the National Board of Medical Examiners (NBME) subject examination. Method: NBME scores, controlled for baseline pre-clerkship knowledge base as measured by second year human behavior scores, were compared for inpatient and outpatient clinical clerkship sites. Results: No effect of clinical site assignment on NBME scores was demonstrated, suggesting that inter-site consistency occurred for the performance outcomes measured. Conclusions: Similar assessments can help other clerkship directors monitor clinical sites for quality of their educational programs, which is increasingly important as medical schools expand their clinical sites to community-based programs.

Figures in this Article

Method

Abstract | Method | Results | Discussion | References

Literature over the past decade has reported the challenges to teaching and learning in medical education related to variability across different clinical settings (¹—³). Medical educators have struggled to achieve equivalence in educational program quality across sites, not only by providing balanced educational service, but by striving for the best performance outcomes. According to a recent study conducted in an obstetrics and gynecology clerkship, different sequences of rotations affected national subject examination outcomes (³). Researchers recommended that medical educators consider assessing variability factors in clinical clerkship programs by developing, implementing, or revising clerkship programs prior to assigning sites to students (⁴). This would help ensure inter-site consistency as one recommended measurement standard for multi-site clerkship program evaluations (⁵).

Standardized examinations have been a core component of assessing medical students’ competence after they have completed clinical clerkships (⁶). Professional oversight bodies have challenged medical schools to undertake curricular reforms, including developing additional assessment procedures, that would contribute to measuring improved performance in medical school and later in graduates’ future clinical practice (⁷). Little is known about the effects of student training in different clerkship sites on educational outcome measures for psychiatry clerkships. Differences in experience between clerkship sites have been difficult to measure. A previous study introduced "inter-site consistency" as one measure of programmatic evaluation for multi-site internal medicine clerkships (⁵). Our study used the National Board of Medical Examiners (NBME) subject examination as an assessment tool for testing students’ knowledge acquired during a clinical psychiatry clerkship, based on studies of its reliability as an objective measure of student performance (⁸—¹⁰). Studies have shown conflicting results for the effect of site on the final exam score within various primary care and ob/gyn clerkships (¹¹—²⁰).

Our study addressed the following research question: Does the rotation site of our third-year psychiatry clerkship affect medical student examination scores on the National Board of Medical Examiners subject exam score for psychiatry? We examined the relationship between exam performance and both inpatient and outpatient sites for a third-year psychiatry clerkship program at a southwestern university medical school.

We assessed the performances of three consecutive classes of third-year medical students from 2001 to 2004 on the NBME subject examination, administered on the final day of each 6-week psychiatry clerkship. All third-year medical students at the University of Oklahoma Health Sciences Center who completed the required psychiatry rotation between July 2001 and June 2004 were eligible for this study. The study was reviewed and approved by the Institutional Review Board of the University of Oklahoma Health Sciences Center, consistent with policies for publishing educational studies. Data included in the study were:

The final grade from the Human Behavior II course, which focused on psychiatric conditions and their treatments, was used to estimate students’ baseline knowledge. Students’ post-clerkship knowledge was measured using their scores on the NBME subject examination in psychiatry, which each student took on the last day of the 6-week rotation. The reliability and validity of this exam as an estimate of content knowledge have been established (⁸—¹⁰).

We reviewed the records of 439 students, first excluding 153 students for whom we did not have a reported NBME score. These were students who completed their third year of training in the clinical track at the Tulsa campus, and whose scores were not available to us. The remaining 286 medical students were assigned 3 weeks at a psychiatric inpatient site and 3 weeks at a psychiatric outpatient site in Oklahoma City. Some medical students alternated between multiple part-time sites during the 3 weeks, spending 1 or 2 days a week at each site. For the purposes of this study, we limited ourselves to comparisons of full-time sites only. Drawn from this group (N=245), our study sample was selected by random assignments into seven equally sized (N=35) sample groups for the sites. For this exploratory study, the sample size of 35 subjects per group was computed to detect medium effect (d=0.05) in the difference of NBME passing raw scores across clerkship sites, two-tailed, and power 0.80, with a significance level of 0.05.

Each site group for all seven sites represented homogeneous distributions of Human Behavior II scores. This was achieved through systematic randomization procedures using proportional numbers obtained from the defined population’s Human Behavior score distribution in the five ranges: less than 20th percentile, above 20th to 40th percentile, above 40th to 60th percentile, above 60th to 80th percentile, and above 80th percentile. After the random assignment into each group, each site had the subject size distribution of 5, 7, 11, 7, and 5 in the five ranges.

To determine whether the independent variables affected the dependent variable differently, we identified independent variables as inpatient site and outpatient site. The dependent variable for the study was NBME subject examination percentile score. After controlling for students’ prior knowledge in psychiatry at each site, randomly assigned site groups were compared on NBME subject examination scores in psychiatry.

Site comparison was conducted for four inpatient sites and for three outpatient sites by analysis of variance (ANOVA). Directional associations from the site factor (nominal data) to the NBME subject examination percentile scores (interval data) were analyzed by Eta. Effect size of variance analyses were calculated by Eta squares (η2).

Results

Abstract | Method | Results | Discussion | References

Table 1 depicts effects of inpatient clerkship sites on NBME subject examination score. There were no statistically significant differences among the four inpatient clerkship sites (which included one community-based site) on their NBME subject examination scores in psychiatry. The site factor by inpatient clerkship sites did not affect medical students’ NBME performance score outcome.

Table 2 reports the effects of the outpatient clerkship site on NBME examination scores. There was no statistically significant difference among the three outpatient clerkship sites (which included one community-based site) on NBME subject examination score in psychiatry. The site factor by outpatient clerkship sites also did not affect medical students’ NBME performance score outcome.

Discussion

Abstract | Method | Results | Discussion | References

In this study, the location of clerkship site did not affect scores on the NBME psychiatry subject examination. There were no significant differences between sites on NBME scores when evaluated, using Human Behavior II final grades as a control. Based upon these data, we found that the psychiatry clerkship site was a factor that did not affect student achievement outcome as measured by NBME subject examination in psychiatry. We concluded that our reviewed multiple-site clerkship program satisfied an inter-site consistency standard to produce the convergent outcome of the National Board of Medical Examiners (NBME) subject examination score for psychiatry, considering both inpatient and outpatient full-time sites.

Our results addressed recommendations of previous studies that medical educators should consider assessing variability factors in clinical clerkship programs while developing, implementing, or revising clerkship programs (⁴) in order to ensure inter-site consistency as a recommended measurement standard for multi-site clerkship program evaluation (⁵). We believe that this study’s design provides a replicable and feasible approach for other directors of academic programs to determine whether their current clerkship provides equivalent opportunities for learners to achieve on a national standardized test of psychiatric knowledge, regardless of multiple site variances.

The present study selected the NBME subject examination percentile score as the standardized student outcome measure because of its reliability and widely accepted use. Although NBME’s strengths include its standardized assessments of diverse subject areas, it does not assess actual "hands-on" patient care skills. We did not attempt to compare any of these technical and interpersonal skills that are highly variable. We recommend that future studies explore additional measures of student outcome in areas beyond NBME scores for inter-site consistency.

Additional limitations of our study were that NBME scores for students on the smaller Tulsa campus were not included in the analysis, nor were those for the students who were assigned to more than one outpatient site, resulting in a smaller sample size. This may have affected the power of the study. The standard deviations of NBME scores were large, so that small differences in performance would be missed (²¹). Moreover, we did not examine other factors that could affect student performance, such as timing of clerkships within the academic year or specific site supervisors. Finally, we cannot rule out that the required weekly teaching conference and standardized weekly afternoon lecture series covering psychopathology, psychotherapy, and biological treatments may have accounted for similar performances on NBME across clinical sites.

Multiple-site clinical education has expanded to community-based sites beyond teaching hospitals, making assessment of inter-site consistency an even more valuable clinical educational standard. This standard is a helpful tool for medical educators in diverse disciplines to review their current curriculum and to provide valuable feedback to clerkship site directors and site supervisors relevant to faculty development.

It is an important educational goal to ensure that a uniformly high quality of medical student psychiatric education is achieved in both the academic and community environments. Our study describes a method of continuously assessing inter-site consistency using NBME subject examination scores, using an approach that can be easily adapted to other clerkship programs.

View Large | Download Slide(.ppt)
Add to My POL

TABLE 1. Effect of Inpatient Clerkship Site on the National Board of Medical Examiners Subject Examination Score

View Large | Download Slide(.ppt)
Add to My POL

TABLE 2. Effect of Outpatient Clerkship Site on the National Board of Medical Examiners Subject Examination Score

References

Abstract | Method | Results | Discussion | References

Irby D: Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med 1995; 70:898—931

Markham FW, Rattner S, Hojat M, et al: Evaluations of medical students’ clinical experiences in a family medicine clerkship: differences in patient encounters by disease severity in different clerkship sites. Fam Med 2002; 34:451—454

Manetta A, Manetta E, Emma D, et al: Effects of rotation discipline on medical student grades in obstetrics and gynecology throughout the academic year. Am J Obstet Gynecol 1993; 169(5):1215—1217

Carney PA, Eliassen MS, Catherine FP, et al: Ambulatory care education: how do academic medical centers, affiliated residency teaching sites, and community-based practices compare? Acad Med 2004; 79:69—77

Durning SJ, Pangaro LN, Denton GD, et al: Inter-site consistency as a measurement of programmatic evaluation in a medicine clerkship with multiple, geographically separated sites. Acad Med 2003; 78:36S—38S

Hemmer PA, Szauter K, Allbritton TA, et al: Internal medicine clerkship directors’ use of and opinions about clerkship examinations. Teach Learn Med 2002; 14(4):229—235

Edelstein RA, Reid HM, Usatine R, et al: A comparative study of measures to evaluate medical students’ performances. Acad Med 2000; 75:825—833

Hoffman KI: The USMLE, the NBME subject examinations, and assessment of individual academic achievement. Acad Med 1993; 68:740—747

Williams RG: Use of NBME and USMLE examinations to evaluate medical education programs. Acad Med 1993; 68:748—752

Minion DJ, Donnelly MB, Quick RC, et al: Are multiple objective measures of student performance necessary? Am J Surg 2002; 183:663—665

Whalen JP, Moses VK: The effect on grades of the timing and site of third-year internal medicine clerkships. Acad Med 1990; 65:708—709

Baciewicz FA, Arent L, Weaver M, et al: Influence of clerkship structure and timing on individual student performance. Am J Surg 1990; 159(2):265—268

Prislin MD, Fitzpatrick CF, Radecki S: A comparison of family medicine clerkship student performance across multiple teaching sites. Fam Med 1998; 30(4):279—282

Jacobsen MJ, Sherman L, Perlman I, et al: Clerkship site and duration: do they influence student performance? Surgery 1986; 100(2):306—311

Irby DM, Vontver LA, Stenchever MA: Evaluation of student performance in a multi-site clinical clerkship. Am J Obstet Gynecol 1980; 136(8):1020—1023

Talbert LM, Stritter FT, Riddle MH: A clinical clerkship in a multicenter setting: an appraisal. J Reprod Med 1981; 26:421—424

Myles TD: Obstetrics and gynecology final examination scores at university and community hospitals: a comparison. J Reprod Med 2001; 46:371—375

Lind D, Marum T, Ledbetter D, et al: The effect of the duration and structure of a surgery clerkship on student performance. J Surg Res 1999; 84(1):106—111

Strand DA, Vontver LA, Stenchever MA, et al: Longitudinal analysis of a multi-site clerkship. Obstet Gynecol 1985; 66(5):723—726

Puryear DA, Miller DA: The effect of clerkship site on medical student examination grades. Med Educ 1990; 24:239—242

Hinkle DE, Oliver JD, Hinkle CA: How large should the sample be? Educ Psychol Meas 1985; 45:271—280