Acad Psychiatry 30:319-324, August 2006
doi: 10.1176/appi.ap.30.4.319
© 2006 Academic Psychiatry
Utilization of a Formative Evaluation Card in a Psychiatry Clerkship
Aurora J. Bennett, M.D.,
Linda M. Goldenhar, Ph.D. and
Kevin Stanford, B.S.
Received October 5, 2005; revised December 19, 2005; accepted December 21, 2005. Dr. Bennett and Mr. Stanford are affiliated with the Department of Psychiatry, University of Cincinnati Medical Center, Cincinnati, Ohio. Dr. Goldenhar is affiliated with the Office of the Dean, University of Cincinnati College of Medicine, Cincinnati, Ohio. Address correspondence to Dr. Bennett, Medical Arts Building, Suite 8500, Univ. of Cincinnati, 222 Piedmont Ave., Cincinnati, OH 45219; aurora.bennett{at}uc.edu (E-mail). Copyright © 2006 Academic Psychiatry
|
ABSTRACT
|
OBJECTIVE: This article discusses how formative feedback to medical students during their clinical rotations facilitates their successfully meeting the rotation’s educational objectives. To help students initiate mid-rotation feedback and to help preceptors structure that feedback, the authors designed the Instant Feedback Card (IFc). The goal of this study was to examine the degree to which the IFc facilitated the formative feedback process from the perspectives of both students and faculty. METHOD: The IFc contains a checklist of the 13 clinical competencies that are also used to provide summative evaluations to students. One hundred and sixty-five medical students completing a 6-week psychiatry clerkship between January and December 2004 were given an IFc to present to their inpatient preceptor at one of their two inpatient sites. At the end of each 3-week inpatient rotation, all students were e-mailed a brief questionnaire about the formative feedback received during the preceding rotation. At the end of the 12-month study period, faculty were e-mailed a brief questionnaire about how useful the IFc was in helping to facilitate the feedback process. RESULTS: One hundred thirty-eight students (80%) participated in the study and returned 267 questionnaires (1.9/student, 97% response rate). Two hundred five students indicated on 77% of the questionnaires returned that they received mid-rotation feedback and of those, 84% revealed that feedback was helpful. Eighty-five percent of the students received mid-rotation feedback when they used the IFc and only 69% received mid-rotation feedback without the IFc. A majority of the 14 faculty serving as inpatient preceptors during the study period found the cards useful for stimulating feedback discussions, for reducing the stress of providing feedback, and for getting students to request feedback. CONCLUSIONS: The results of this pilot study are not definitive, but they do indicate that the Instant Feedback card (IFc) did facilitate the process and structure of providing and receiving formative feedback as indicated by a significantly greater number of students receiving formative feedback when they used the IFc.
|
INTRODUCTION
|
The importance of providing nonjudgmental formative feedback that focuses on directly observable skills and behaviors has long been recognized as an integral component of medical education (1–3). It affords students as well as preceptors a vehicle for understanding educational objectives, an opportunity to process the student’s clinical encounters and problem solving skills, as well as an objective appraisal of the student’s self-assessment of his or her emerging professional development (4). Faculty in medical institutions provide formative feedback to their students using a variety of methods. However, it continues to be provided inconsistently during the clinical years (5, 6). Deterrents to providing students with this valuable component of their education include multiple preceptors viewing different components of a student’s clinical skills, preceptors’ time constraints, students’ ambivalence about requesting feedback, and preceptors’ discomfort with discussing areas of deficiency with students (7). To address these issues, we developed the Instant Feedback card (IFc) as a tool to facilitate the formative feedback process during students’ clinical rotations at the University of Cincinnati College of Medicine.
The IFc is not the first card of this sort to be used to help facilitate formative feedback in clinical settings. Previous studies have reported on feedback cards or forms used with residents and students in a variety of clinical disciplines (8–11). The forms in these studies were similar to the IFc in that they contained a list of competencies relevant to the rotation or residency, and several were similar in design to the pocket-size 4x6 IFc that is carried easily by students. The IFc differs from the other feedback forms in that others include scales for evaluating the student (e.g., 5-point Likert scale, three levels of performance—below, at, or above expectations); the IFc provides the competencies only as a checklist to remind faculty of the clinical objectives for which they will be asked to provide numerical scores in the summative evaluations. The time frame for completing the forms in other studies ranged from doing so immediately after a single clinical encounter to waiting until the midpoint of the rotation. To our knowledge, there is only one other study in psychiatry that has looked at mid-rotation feedback given during the inpatient rotation. In this Australian study, Chur-Hansen and Koopowitz (12) devised a mid-rotation feedback form that evaluated students on a three-point scale that was used to structure verbal feedback provided at Week 4 of a 6-week clerkship. The same form was used for the summative evaluation. The students worked at one of two hospitals during their clerkship, and only one hospital was willing to participate in using the feedback form. The majority of the students rated the feedback as useful on an anonymous questionnaire, although several remarked about the lack of constructive criticism provided. The students at the hospital that did not use the form were not asked to complete the questionnaire regarding formative feedback and the faculty were not asked about their experiences with providing formative feedback with or without the form.
Using and expanding on the designs and results from these earlier studies, the psychiatry clerkship at the University of Cincinnati asked third-year medical students to use the IFc during one of their 3-week inpatient rotations, but not at their other 3-week site. The purpose was to study the effect of the IFc on the frequency and perceived quality of formative feedback received by medical students from their preceptors, compared with when they did not use the cards, as well as the degree to which their preceptors found the IFc useful for providing that feedback.
|
Method
|
Subjects
The subjects for this study included all third-year medical students enrolled in the psychiatry clerkship between January and December 2004 (N=165) at the University of Cincinnati College of Medicine, as well as all faculty who served as students’ inpatient preceptors during this same time period (N=14). Participation in the study was voluntary. The University of Cincinnati Institutional Review Board (IRB) granted approval for exempt status for this study.
Data Collection
The 4 x 6 Instant Feedback card (IFc) is double-sided. One side is for pointing out areas in which the student is doing well; the other is to indicate areas needing improvement. On both sides there is an identical list of 13 clinical competencies with additional space for comments (Figure 1). The design of the IFc was modeled from the summative evaluation form completed for each student at the end of each clerkship rotation. The only difference between the two instruments is that each of the 13 clinical competencies on the summative evaluation is scored on a five-point scale whereas the same competencies on the IFc are not scored, but are intended only to provide structure to the feedback process. Furthermore, the list of clinical competencies is derived from the goals and objectives of the clerkship in an attempt to clarify them and to remind both students and faculty of the clerkship goals and objectives.
Students were introduced to the IFc during their psychiatry clerkship orientation and were given an explanation for how they should be used. The decision to use the IFc was voluntary. There is a total of eight 6-week psychiatry rotations over the course of one academic year, each having an inpatient component consisting of two consecutive 3-week inpatient rotations at two separate sites. Students enrolled in the first, third, fifth, and seventh psychiatry rotations of the year were instructed to use the IFc for their first 3-week inpatient rotation, but not to use it during their second 3 weeks. Students enrolled in the second, fourth, sixth, and eighth clerkship rotations were told to use the IFc during their second 3-week rotation, but not in their first. Varying the timing for using the IFc controlled for the possibility of students feeling less confident in their clinical skills in the first half of the clerkship, which may affect their willingness to ask for feedback from their preceptor. At the end of each 3-week rotation, participating students were e-mailed and asked to complete a brief questionnaire about the amount and quality of formative feedback received during the preceding card or no card 3-week rotation (Appendix 1). That is, students were asked to complete two questionnaires over the 6-week rotation, in essence serving as their own controls. Faculty had been notified about the implementation of the IFc prior to the beginning of the study period. At the end of the 12-month study period, faculty were sent a brief questionnaire containing questions regarding how useful the IFc was at helping to facilitate the feedback process (Appendix 2).
Analysis
Data were analyzed using SAS v. 8.2 (SAS Institute Inc., Cary, NC, 2005). All differences between and among categorical outcomes were assessed using Fisher’s exact test using a two-tailed test, with statistical significance set at the 0.05 level. Relevant descriptive statistics were calculated as needed. No inferential statistics were calculated for faculty response data due to the small population (N=14).
|
Results
|
Students
One hundred thirty-eight out of 165 students (84%) participated in the study and returned a total of 267 questionnaires (1.9/student, 97% response rate). On 77% of the questionnaires returned (N=205), students indicated that they had received mid-rotation feedback. Whether they had the IFc made no difference to the students’ perceptions of how helpful the feedback was that they received—86% and 87%, respectively, said that the feedback received was fairly or extremely helpful. Eighty-five percent of the students (N=111 of 131) completing questionnaires after the 3-week inpatient rotation during which they were asked to use the IFc said they received mid-rotation feedback compared with only 69% (N=94 of 136) receiving mid-rotation feedback after the 3-week rotation without the IFc (Fisher’s exact test, p=0.004). Also, significantly more students said that they initiated feedback during the 3 weeks when they used the IFc (63%, N=41 of 94) than during the 3 weeks they did not use the IFc (44%, N=70 of 111) (p=0.007).
Faculty
All 14 faculty completed the questionnaire they were sent. Data show that an equal number of faculty preferred combining verbal feedback with the IFc or providing verbal feedback only. Only two preferred using the IFc alone. When asked in what ways the IFc was useful, a majority of the faculty found it useful for stimulating feedback discussions, for reducing the stress of providing feedback, and for helping students to request feedback (Table 1).
|
Conclusions
|
The findings of this pilot study indicate that 85% of the respondents received mid-rotation feedback when they utilized the IFc, whereas only 69% received similar feedback without the IFc. Students were more likely to request feedback with the use of the card than without it. Several students commented that the card symbolized clerkship support for the importance of obtaining feedback, therefore lessening their concerns about being perceived as "pushy" or "seeking praise" when approaching a preceptor for feedback. Approximately 86% of the students described the feedback as useful regardless of whether the IFc was formally used in providing the feedback. A few of the faculty reported they had structured their informal feedback to the student based on the competencies on the IFc, but did not specifically use the card during their meetings with the students to comment on each competency in a verbal or written format (personal communications).
The IFc was designed to provide a continuum from the clerkship goals and objectives to the mid-rotation feedback and finally to the scored summative evaluation because all three reiterated the same 13 core clinical competencies to be learned by the end of the clerkship. The intent in providing such continuity was to enable the students to take a more active role in seeking clinical experiences to enhance their education in those areas defined as necessary to achieve clinical competence in the clerkship. Several students reported anecdotally that the formative feedback prompted them to ask residents and faculty to reevaluate areas of deficiency that they had worked on improving such as chart documentation, obtaining a detailed mental status examination, and responding to patients’ nonverbal communications with greater empathy. Furthermore, a few of the faculty acknowledged observing the students’ efforts in a more detailed manner due to the level of interest shown by students to improve their clinical skills. The format of the IFc also provides students with guidance in developing their skills of self-assessment as they compare their own evaluations of their performances with the feedback received from preceptors (13). Although not formally queried in this study, students commented during a feedback meeting with the clerkship director that not having a grade attached to the IFc made them more comfortable discussing their relative deficiencies with their preceptors. They verbalized a desire for more detailed feedback about their deficiencies, continuing to perceive faculty as minimizing these and giving generally positive remarks. Faculty providing vague and inadequately detailed feedback has been a long-standing complaint of students during their clinical years.
There are several limitations to our study that should be delineated. First, the students and faculty were aware that the questionnaires were being sent as part of a study, which may have increased their utilization of the feedback cards. Five of the 14 faculty who responded said that more students had requested mid-rotation feedback since the card system was put in place, but this increase was not specifically quantified in this study. Continuing to follow the frequency of the IFc usage beyond the study period will help determine its actual usage. Second, although 86% of students rated the feedback as useful, only 21% of these rated it as extremely helpful, whereas the remaining 65% rated it as fairly helpful. Their narrative comments provide confirmation that the feedback provided by faculty lacks sufficient detail to reinforce positive behaviors and to identify and correct deficiencies. Third, the formative feedback cards were not collected from the students to look at whether deficiencies identified at mid-rotation showed improvement on the end of clerkship summative evaluations. Fourth, several students commented on their questionnaires that they had not received mid-rotation feedback "because the attending had provided almost daily feedback on their performance," indicating that we needed to better define formative feedback to include all feedback occurring throughout the rotation. Similarly, we did not clarify with faculty that card usage included using it as a template for providing feedback despite not physically having the card in the feedback meeting with the student.
This study expands on the existing literature regarding formative feedback by using the identical clinical competencies on the formative feedback card that are used for the summative evaluation, by utilizing students as their own controls, and by asking both students and faculty to assess the usefulness of this format of mid-rotation feedback. Currently, students are being advised to use the IFc at all of their inpatient and outpatient sites in our psychiatry clerkship to expand the numbers of preceptors providing them with feedback from a variety of settings. Students are clear in their communication that a more detailed and descriptive formative assessment of their emerging clinical skills that not only reinforces their strengths but also identifies areas for improvement with recommendations on how to do so would be a valuable learning tool. Future studies are needed to look at whether such feedback improves students’ summative evaluations, to look at faculty’s insufficient constructive remarks, and to assess faculty’s perceptions of the efficiency of using a card to provide feedback.
|
ACKNOWLEDGMENTS
|
This study was supported by a medical education grant from the Dean’s Office, University of Cincinnati College of Medicine.
|
REFERENCES
|
- Ende J: Feedback in clinical medical education. J Am Med Assoc 1983; 250:777–781[Abstract]
- Maudsley RF: Effective in-training evaluation. Med Teach 1989; 11(3-4):285-290
- Katz PO: Providing feedback. Gastrointest Endosc Clin N Am 1995; 5:347–355[Medline]
- Prystowsky JB, DaRosa DA: A learning prescription permits feedback on feedback. Am J Surg 2003; 185:264–267[CrossRef][Medline]
- Heckelman FP, Vanek E, Kelly K, et al: Characteristics of family physicians’ clinical teaching behaviors in the ambulatory setting: a descriptive study. Teach Learn Med 1993; 5:18–23[Medline]
- Salerno SM, O’Malley PG, Pangaro LN, et al: Faculty development seminars based on the one-minute preceptor improve feedback in the ambulatory setting. J Gen Intern Med 2002; 17:779–787[CrossRef][Medline]
- Gil DH, Heins M, Jones PB: Perceptions of medical school faculty members and students on clinical clerkship feedback. J Med Educ 1984; 59:856–864[Medline]
- Brennan BG, Norma GR: Use of encounter cards for evaluation of residents in obstetrics. Acad Med 1997; 72(10 Suppl 1): S43-44
- Lye P, Bragg D, Simpson D: Improving feedback with a clinical encounter form. Acad Med 1997; 72:444–445[CrossRef][Medline]
- Greenberg LW: Medical students’ perceptions of feedback in a busy ambulatory setting: a descriptive study using a clinical encounter card. South Med J 2004; 97(12):1174–1178[CrossRef][Medline]
- Ogburn T, Espey E: The R-I-M-E method for evaluation of medical students on an obstetrics and gynecology clerkship. Am J Obstet Gynecol 2003; 189(3):666–669[CrossRef][Medline]
- Chur-Hansen A, Koopowitz LF: Formative feedback in teaching undergraduate psychiatry. Acad Psychiatry 2005; 29:66–68[Abstract/Free Full Text]
- Rolfe I, McPherson J: Formative assessment: how am I doing? Lancet 1995; 345:837–839[CrossRef][Medline]
Get information about faster international access.
Privacy Policy
Copyright © 2006
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
