Acad Psychiatry 30:110-115, March-April
doi: 10.1176/appi.ap.30.2.110
© 2006 Academic Psychiatry
An Approach to Address Grade Inflation in a Psychiatry Clerkship
Brenda J.B. Roman, M.D. and
Justin Trevino, M.D.
Received March 30, 2005; revised August 30, 2005; accepted September 23, 2005. Dr. Brenda Roman is Associate Professor of Psychiatry and Director of Medical Student Education in Psychiatry in the Boonshoft School of Medicine at Wright State University, Dayton, Ohio. Dr. Trevino is Assistant Professor and Associate Director of Medical Student Education at Wright State University Boonshoft School of Medicine. Address correspondence to Dr. Roman, Department of Psychiatry, P.O. Box 927, Wright State University, Dayton, OH 45401-0927; brenda.roman{at}wright.edu (E-mail). Copyright © 2006 Academic Psychiatry.
|
ABSTRACT
|
OBJECTIVE: Universally, clerkship grading is diverse and not standardized. The authors’ faculty was troubled by the inability to provide meaningful evaluations, as more than 60% of students received the highest grade. Although a psychiatry clerkship mandate of a faculty-observed student clinical interview existed for several years, the majority of students reported not completing the interview under direct observation by a faculty member, and no meaningful feedback or evaluation for this activity existed. In order to create diversity in grading criteria and to examine clinical skills more thoroughly than previously, written and oral examinations were developed and supervised interviews of patients and written comprehensive psychiatric evaluations were added. A core group of department faculty was instructed in the use of materials and instruments designed to standardize the experiences and the student evaluations. RESULTS: Adding a wider diversity of experiences and evaluations to the clerkship, particularly assessment of interviewing skills, oral exams, and evaluation of comprehensive histories, has resulted in a more divergent spread of grades. CONCLUSION: Clerkship grades can be effectively computed using various methods to examine knowledge and clinical skills. The addition of new methods of evaluation has added specificity to the performance feedback provided to the students completing the psychiatric clerkship. These changes have been viewed positively by department faculty and medical students. While requiring further refinement, they may eventually provide data to identify students requiring special attention in specific cognitive, relational, and clinical skill areas.
|
INTRODUCTION
|
Clerkship grades tend to cluster in the above average range. One factor is likely grade inflation, as noted by Speer et al., who found that significant grade inflation exists in internal medicine clerkships (1). The parameters on which students are evaluated in clerkships may include National Board of Medical Examiners (NBME) subject exams, in house exams and/or quizzes, objective standardized clinical examinations (OSCEs), oral examinations and subjective preceptor grading (2). The results of the NBME subject exams vary greatly among students. However, this exam is not a good measure of clinical skills or professionalism (3). Historically, clinical grades are generated by attending physicians with little specificity and interrater reliability since numerous subjective factors are involved. These include fear of litigation for poor evaluations, fear of poor preceptor evaluations if students are not graded highly, not wanting to hurt students’ feelings, low priority of clinical assessment, and the general grade inflation seen in higher education. In general, clinical grading tends to be inflated, and accurate objective evaluation of clinical skills remains challenging. Additionally, faculty members simply have less time for academic teaching and hence effective evaluation, due to the demands of managed care (4). Financial constraints, time pressures and the perceived lack of recognition for teaching also hinder the ability to motivate and recruit faculty, especially generalists, to teach interviewing and physical exam skills (5).
Our institutional experience showed that more than 60% of the students received the equivalent of an A grade prior to changes in the clerkship evaluation methods begun in 1998. Direct observation of clinical skills by faculty members, specifically completion of histories and physicals by students, was occurring infrequently during medical school and was more often completed by residents rather than designated teaching faculty (6). Several studies (6–8) have shown that faculty tend to rate student and resident clinical performances as satisfactory or superior, despite marginal performance with errors. Most concerning is that a study examining faculty evaluation of interviewing skills of medical students found the positive predictive value for "adequate" interviewing skills was only 12% (9).
|
Clerkship Evaluation Description
|
In the Wright State University Department of Psychiatry, the clerkship director has developed grading criteria, based on both objective and subjective measures, which have resulted in a wider grade distribution in the psychiatry clerkship. This article will describe the various methods utilized in determining the clerkship grade. Although this particular schema still has room for improvement, faculty feel that clinical skills in particular are being evaluated far more objectively than before and that students with poor or marginal skills are being identified more reliably. Table 1 shows the current grading criteria for the Psychiatry Clerkship.
The most notable change was the reduction of the attending evaluation component, decreased in percentage worth over the last 8 years from 60% to 22%. Although the faculty attending and residents may work directly with the student, their clinical evaluations were inflated, with an average grade of over 95% given the vast majority of the time. This continued to occur despite working with inpatient attendings to be more objective in evaluations. Attendings tended to persist with the view that "if the student works hard, they deserve a good grade." Unfortunately, this perspective did not allow the opportunity to distinguish the student with knowledge and skill deficits from the high-performing student. Objective examinations such as the NBME subject exam provide standardized data and grading points, but only measure knowledge, not skills. While these are important evaluation tools, the clerkship director did not want such knowledge-based examinations to compose more than the majority of the clerkship grade. In order to diversify and better structure the learning and evaluation experiences of the clerkship students, a broader array of exercises including OSCEs, supervised interviews, and written comprehensive evaluations have progressively been added as required clerkship activities. Most recently, team based learning (10) has been implemented into the psychiatric clerkship, with 5 sessions replacing traditional lectures. During each session, students complete the individual readiness assessment test (IRAT) and group readiness assessment test (GRAT). Both tests are the same 15 multiple choice question examinations, which are graded, with the latter completed as a group exercise. After systematically reviewing each question, a group application exercise is completed. These team based learning exercises promote active student learning and collaborative work. For the purpose of this article, the OSCEs, supervised interviews, and written comprehensive evaluations will be described in detail.
|
OSCEs
|
To better evaluate clinical skills and clinical application of knowledge, the clerkship director first developed a series of "OSCE videos," one of which is written and the other oral. While not true objective standardized clinical exams, the use of videotapes allows for standardization across clerkship groups and for the evaluating faculty to discuss the videos and agree on elements indicative of an acceptable level of student mastery. Figure 1 shows the distribution of clerkship grades pre-OSCE and post-OSCE. These examinations consist of clinical videotapes of 25-minute duration of a psychiatric interview. In the first exam, after watching the videotape, the students have 45 minutes to complete a written assessment, including mental status exam, differential diagnosis, DSM-IV-TR criteria, major psychosocial considerations, Axis I-V diagnoses, comprehensive treatment plan, biological theories and prognosis. Use of textbook or other reference materials is not allowed. Written exams are graded by the clerkship director. The second videotape shows a patient with a psychiatric diagnosis differing from that demonstrated in the first videotape. Immediately following viewing of the video, students present orally to faculty examiners, describing the patient’s history of present illness, psychiatric and other contributing history, mental status exam, diagnosis and treatment plan over 30 minutes. To minimize examiner variability, the tapes are viewed by the examiners together, just prior to the oral exams, with discussion of important points. A grading form with specified criteria is then utilized. For the past 6 years, each clinical exam has counted for 10% of their final grade, with the requirement that the overall grade in these clinical exams must be passing in order to receive a passing grade in the clerkship.
View larger version (54K):
[in this window]
[in a new window]
|
FIGURE 1. Distribution of Clerkship Grades Pre- and Post-OSCE
OSCE=objective standardized clinical examinations
|
|
Observed Clinical Interview
|
Overall the video OSCEs have been successful as an assessment tool in examining presentation skills, diagnostic skills and clinical application of knowledge. What they lacked however was objective evaluation of clinical skills with patients. To address this deficit, a clerkship requirement of an observed patient interview with evaluation by an attending or senior resident was initiated. Despite development of specific scales to be utilized in this grading, the average grade for this unstructured clinical interview was 97% for the 5 years prior to 2003–2004 academic year. Students reported that they were rarely observed doing the interview for any longer than a few minutes. With renewed emphasis on clinical skills, and the implementation of the United States Medical Licensing Examination (USMLE) clinical skills exam, the clerkship director and core faculty developed a process to make clinical interviewing, with meaningful evaluation, a priority item. An evaluation instrument, with 10 content areas (including conduct of interview, rapport, elicitation of data and organization) and specific grading criteria (behaviorally based, utilizing a scale of 1–5 to rate student performance), was developed for faculty use with this activity. Students are assigned a time during the mid to latter portion of the psychiatry clerkship to arrange to interview a patient, not previously encountered, for 45 minutes. The interviews are observed by an off-site faculty member, one specifically trained in the use of the evaluation instrument. Please see distribution of grades in Figure 2 for the supervised interviews, comparing the grades before and after the institution of objective faculty grading guidelines.
|
Written Psychiatric Evaluations
|
To target another area of marginal student performance and one not specifically graded before, students are required to turn in two comprehensive psychiatric histories and physicals. The purpose of these written psychiatric evaluations is multifold:
- To ensure that students obtain comprehensive developmental and psychosocial histories for at least two patients during their clerkship experience;
- To challenge them to develop a biopsychosocial formulation;
- To put forward a long term comprehensive plan of care for the selected patients.
With experiences on inpatient units and outpatient clinics being pharmacologically and crisis intervention oriented, it is the only time that students are asked to consider defense mechanisms, psychodynamic issues, object relations, transference and countertransference.
|
Results
|
Prior to 1998, the average psychiatry clerkship grade was 89%, with more than 60% of the students receiving As (90%–100%). In the most recent academic year, the average was 85% with only 15% receiving 90% or above. Figure 3 shows the distribution of psychiatry clerkship grades for the 2003–2004 academic year, compared with the average distribution of grades prior to clerkship evaluation changes. More specifically, the OSCEs and the observed clinical interview evaluations have proven invaluable in identifying students who struggle with organization and presentation skills or who have difficulty in the application of knowledge to the clinical setting or in establishing therapeutic rapport and effectively eliciting a history from a patient. In regard to the observed clinical interview, faculty have noted the following major problem areas: poorly elicited history of present illness; lack of depth in exploration of social histories; and students appearing to be "checklist robots," interviewing only toward clarification of specific DSM diagnoses while making minimal efforts to empathically connect to the patient. With the observed clinical interview counting as 10% of their overall grade, students take this requirement seriously and appreciate the immediate feedback provided. Additionally, students who have difficulties with skills specific to patient interaction and verbal information presentation can now be identified, and plans to target deficiencies can be implemented. The written comprehensive psychiatric evaluations provide opportunities to determine the depth of information that students obtain from patients while challenging them to develop biopsychosocial formulations and comprehensive treatment plans. The major problem areas tend to be inadequate gathering of developmental histories, omission of sexual histories, brief mental status descriptions, no attempt at developing a formulation, and focusing on an acute pharmacologically based treatment plan.
|
Discussion
|
The importance of a comprehensive evaluation of medical student skills during a clerkship experience cannot be overemphasized. In particular, the clinical skills related to interviewing and development of a positive therapeutic alliance with patients can benefit from application of a structured, standardized approach. Medical students should be informed of the literature citing that physicians with malpractice claims were less likely to utilize patient-centered interviewing skills than were physicians who had not been sued (11). Likewise, they need to know that even in this highly technical world of medicine, inadequate data-collection skills are still a significant factor in diagnostic errors (12). Literature continues to support the fact that the combination of history and physical examination produces accurate diagnoses twice as often as diagnostic imaging (13), and yet so little time in medical education is devoted to solid assessment of basic clinical skills. For all these reasons, the emphasis on meaningful evaluation of interviewing skills is important and should be made a priority in all clerkship disciplines. The major limitation of the current clerkship grading criteria is the time demand for faculty. Currently, the Director of Medical Student Education (DMSE) in Psychiatry (also clerkship director) grades all the written OSCEs and all written histories and physicals. A core group of faculty (the DMSE, the Associate Director of Medical Student Education in Psychiatry, and the Associate Dean for Academic Affairs, who is a psychiatrist, along with 4–6 other faculty) conduct nearly all the observed interviews and another core group (including some of the same faculty just enumerated) the oral OSCE evaluations. Without the commitment to medical student education from the chair, residency training director and other faculty, this kind of comprehensive evaluation would not be possible. Despite time demands, the faculty find this kind of evaluation rewarding and realize that this kind of in-depth evaluation provides for thorough assessment of student skills in a way not previously done. Students who have difficulties in an area are now identified and offered opportunities for help in the psychiatry clerkship, as well as in other clerkships. Students report that the supervised interview and oral examinations are a source of anxiety; however, they appreciate the feedback and experience in these evaluation methods, particularly in anticipation of the USMLE Step 2 clinical skills exam. Further work will be done on correlation of specific clerkship scores with the general fourth-year OSCE scores administered by the medical school prior to graduation. Additionally, as the School of Medicine learns of any failures in the USMLE Step 2 clinical skills exam, the supervised interview and oral exam grades will be examined to see if any correlation exists.
|
Conclusion
|
In summary, evaluation techniques developed over the past few years for the psychiatry clerkship are now among the most diversified that are utilized in the medical school, particularly with the emphasis on objective clinical skills assessment. Knowledge, primarily factual recall and application, is evaluated through the NBME shelf exam and quizzes. Presentation and organizational skills, in addition to knowledge, are evaluated through the OSCE videos. Communication and interview skills are evaluated through the supervised clinical interviews.
Organizational skills, written communication skills, and application of knowledge are evaluated through in-depth written psychiatric evaluations. The results of greater diversity in grades, particularly with fewer in the highest range, may be due to increased faculty attention to evaluation of students versus the actual reliability and validity of rating instruments utilized. In fact, studies (14, 15) have shown that reliability and validity of oral examinations are low. Nevertheless, the authors argue that such evaluation methods are superior to the overreliance on generic attending evaluations, which offer no interrater reliability or validity, for a large percentage of clerkship grades. With these various evaluations targeting different skills, the psychiatry clerkship can now provide more meaningful student evaluations. This has allowed for improved identification of both the most talented students whose efforts and skills can be meaningfully acknowledged and encouraged, as well as the marginal students who need additional support and assistance in their clerkship efforts. Further research in the area of grade inflation in psychiatry clerkships should be undertaken, beginning with a survey of clerkship directors, like that done in internal medicine (1) With even greater pressure on medical schools to only graduate students who are competent in knowledge and skills, as well as professionalism, gathering data to assess the validity and reliability of various evaluation methods will be crucial. Priority should be given to the development of standardized experiences and rating instruments to facilitate this goal.
|
ACKNOWLEDGMENTS
|
The authors thank Dean Parmelee, M.D., the Associate Dean for Academic Affairs in the Boonshoft School of Medicine at Wright State University, and Jerald Kay, M.D., the Chair of the Department of Psychiatry at Wright State University Boonshoft School of Medicine, for support and assistance in the medical student educational program, and to Brenda Ullery, administrative assistant, for all her work in coordinating the OSCE videos and the supervised interviews.
|
REFERENCES
|
- Speer AJ, Solomon DJ, Fincher RME: Grade inflation in internal medicine clerkships. Teach Learn Med 2000; 12(3):112–116[CrossRef][Medline]
- Rosenthal RH, Levine RE, Carlson DL, et al: The "Shrinking" Clerkship: Characteristics and Length of Clerkships in Psychiatry Undergraduate Education. Acad Psychiatry 2005; 29:47–51[Abstract/Free Full Text]
- Levine RE, Carlson Dl, Rosenthal RH, et al: Usage of the National Board of Medical Examiners Subject Test in Psychiatry by U.S. and Canadian Clerkships. Acad Psychiatry 2005; 29:52–57[Abstract/Free Full Text]
- Simon SR, Pan RJD, Sullivan AM, et al: Views of managed care. A survey of students, residents, faculty, and deans at medical schools in the United States. NEJM 1999; 340(12):928–936[Abstract/Free Full Text]
- Zakowski LJ, Seibert C, Van Eyck S, et al: Can specialists and generalists teach clinical skills to second-year medical students with equal effectiveness? Acad Med 2002; 77(10):1030–1033[Medline]
- Holmboe ES: Faculty and the Observation of Trainees’ Clinical Skills: Problems and Opportunities. Academic Medicine 2004; 79:16–22[CrossRef][Medline]
- Schwartz RW, Donnelly MB, Sloan DA, et al: The relationship between faculty ward evaluations, OSCE and ABSITE as measures of surgical intern performance. J Surg Res 1994; 57:613–618[CrossRef][Medline]
- Noel GL, Herbers JE, Caplow MP, et al: How well do internal medicine faculty members evaluate the clinical skills of residents? Ann Intern Med 1992; 117:757–765[Abstract/Free Full Text]
- Kalet A, Earp JA, Kowlowitz V: How well do faculty evaluate the interviewing skills of medical students? J Gen Intern Med. 1992; 97:179-184
- Levine R, O’Boyle M, Haidet P, et al: Transforming a Clinical Clerkship with Team Learning. Teaching and Learning in Medicine 2004; 16(3):270–275[CrossRef][Medline]
- Levinson W, Roter DL, Mullooly JP, et al: Physician-patient communication: The relationship with malpractice claims among primary care physicians and surgeons. JAMA 1997; 277:553–559[Abstract/Free Full Text]
- Bondage G: Why did I miss the diagnosis? Some cognitive explanations and educational implications. Acad Med 1999; 74 (10 suppl):S138-S143
- Kirch W, Schafit C: Misdiagnosis at a university hospital in four medical areas. Report on 400 cases. Medicine 1996; 75:29–40[CrossRef][Medline]
- Sierles FS, Daghestani A, Weiner CL, et al: Psychometric properties of ABPN-style oral examinations administered jointly by two psychiatry residency programs. Acad Psychiatry 2001; 25(4):214–222[Abstract/Free Full Text]
- Leichner P, Sisler GC, Harper D: The clinical oral examination in psychiatry: association between subscoring and global marks’. Can J Psychiatry 1986; 31:750–751[Medline]
This article has been cited by other articles:
|
|
|
|
 
G. W. Briscoe, L. Fore-Arcand, R. E. Levine, D. L. Carlson, J. J. Spollen, C. Pelic, and C. S. Al-Mateen
Psychiatry Clerkship Students' Preparation, Reflection, and Results on the NBME Psychiatry Subject Exam
Acad Psychiatry,
March 1, 2009;
33(2):
120 - 124.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
B. R. Rush, R. G. Elmore, and M. W. Sanderson
Grade Inflation at a North American College of Veterinary Medicine: 1985-2006
J Vet Med Educ,
March 1, 2009;
36(1):
107 - 113.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
D. A. Reed, C. P. West, P. S. Mueller, R. D. Ficalora, G. J. Engstler, and T. J. Beckman
Behaviors of Highly Professional Resident Physicians
JAMA,
September 17, 2008;
300(11):
1326 - 1333.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
J. D. Schmahmann, M. Neal, and J. MacMore
Evaluation of the assessment and grading of medical students on a neurology clerkship
Neurology,
February 26, 2008;
70(9):
706 - 712.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
I. D. Glick, C. Salzman, B. M. Cohen, D. F. Klein, C. Moutier, H. A. Nasrallah, D. Ongur, P. Wang, and S. Zisook
Improving the Pedagogy Associated With the Teaching of Psychopharmacology
Acad Psychiatry,
June 1, 2007;
31(3):
211 - 217.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
L. W. Roberts, J. Coverdale, and A. Louie
Committing to Medical Student Education
Acad Psychiatry,
April 1, 2006;
30(2):
93 - 94.
[Full Text]
[PDF]
|
|
|
|
|
|
|
J. L. Cutler
Psychiatric Education for Medical Students: Challenges and Solutions
Acad Psychiatry,
April 1, 2006;
30(2):
95 - 97.
[Full Text]
[PDF]
|
|
|
|
|
|
|
H. Pardes
A Look at Psychiatric Education
Acad Psychiatry,
April 1, 2006;
30(2):
98 - 100.
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2006
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
