In the 1980s, as a mandate from the United States Department of Education, the accreditation of professional educational institutions began focusing on measurable goals and objectives to assess the quality of individuals and institutions. Specifically, they wanted to develop standardized means of determining how medical students and residents are trained and how they are held accountable for their professional behavior (1). In 1999 the Accreditation Council for Graduate Medical Education (ACGME) designated Six General Competencies for All Physicians: Patient care, Medical knowledge, Practice-based learning and improvement, Interpersonal and communication skills, Professionalism, Systems-based practice (1). By following the ACGME criteria, the Psychiatry Residency Review Committee determined requirements for training programs to document residents’ satisfactory completion of the mandatory clinical rotations (e.g., 9 months of inpatient rotation, 12 months outpatient rotation).

Psychiatry residency promotions committees (RPC) are dependent on attending psychiatrists’ written evaluations of residents when considering promotions or remedial training of competencies, since hands-on supervision is the cornerstone of psychiatric training. Supervising attending psychiatrists are the only individuals who can evaluate the residents’ practical skills (1). Without written documentation of these skill levels, the RPC have limited evidence upon which to determine a resident’s clinical competency (2). Given the nature of the expert-trainee relationship in clinical practice, typically there is only one attending physician per psychiatric rotation, and these two individuals attend rounds together, discuss their patient case-load, and plan ongoing treatment. As such, each evaluation is very important in determining whether the resident has the expected level of competency for their training level. Also, although ongoing feedback is very important in improving the quality of residents’ skills (3), written evaluations provide an invaluable opportunity for residents to receive more formal routine feedback (4). Documentation of clinical competency is necessary for promotion, and such documentation should always include feedback from the attending psychiatrist who is most knowledgeable about the resident’s current level of skills.

Although evaluations are important, our baseline data indicated that fewer than half of attending psychiatrists at a university-based psychiatry residency program submitted resident evaluation forms. These findings have also been documented by others (5). In the literature, several reasons have been posited for the poor evaluation completion rate, including a) reluctance to document marginal performance due to the potential adverse impact on the relationship between the attending psychiatrist and the resident; b) the perception that no administrative action will be taken; c) limited faculty ability and motivation to judge resident performance; and d) the length and the complexity of the evaluation form. In an attempt to improve the rate of performance evaluations at their institution, Littlefield and Terrell (6) created a task force to revise the appraisal form, pilot test it, and encourage their faculty to respond to questions. The authors reported that their interventions resulted in an increase in response rates.

The purpose of this study was to expand upon Littlefield and Terrell’s work at our own institution in order to further elucidate the factors that deter our attending psychiatrists from completing resident evaluation forms and to use this information to create an intervention to increase compliance. Specifically, the objectives were to gather feedback from the attending psychiatrists regarding their own frequency of completing the forms, the reasons why they may not complete the forms, and processes to help them increase their rates of completion in order to devise and implement an intervention to increase completion rates. It is assumed that focusing on the process by which attending psychiatrists complete the forms would result in increases in completed forms.

Questionnaires

Initially the authors mailed a two-item questionnaire to attending psychiatrists (N=21) at an inpatient psychiatric hospital. The purpose of the questionnaire was to assess the physicians’ compliance history for completing resident rotation evaluation forms. The attending psychiatrists were asked to indicate whether they completed the resident rotation evaluation forms on an always, usually, sometimes, rarely, or never basis. In addition, a comment section was provided for participants to list the barriers to completing the evaluations. Only three of the 21 psychiatrists responded to the questionnaire, thereby providing a response rate of 14%.

Based on the initial questionnaire as well as the discussion, a short, seven-item questionnaire was created to assess physicians’ self-reported completion of the resident evaluation form as well as the helpfulness of five proposed interventions to increase completion rates. The questionnaire was administered via the web, using a web survey delivery system.

Focused Discussion

An open discussion with 17 of the 21 attending psychiatrists was also held during a regularly scheduled hospital staff meeting to further delineate barriers to completing resident evaluation forms and to identify interventions to help increase compliance. The topic was an item on the agenda and was introduced by the PI. Those present made comments about difficulties they experienced in completing the forms on time and they brainstormed and came up with potential helpful reminders and support to complete the forms. This information was used in developing the second questionnaire.

Participants

The seven-item questionnaire was sent to all of the attending psychiatrists (N=21) at a large urban psychiatric hospital affiliated with a medical school. Overall, 16 psychiatrists completed the survey, for a response rate of 76%. Of our 21 psychiatrists, 15 are female and 6 are male; 10 are Caucasian, 3 African American, 5 Asian, and 3 Hispanic. One of the psychiatrists is a full professor, 5 are associate professors, and 15 are assistant professors.

Data Analysis

Descriptive statistics were used to analyze responses to the questionnaire. Specifically, response frequencies and percentages were calculated. To determine the impact of our interventions (i.e., e-mailing each attending physician a reminder and an electronic copy of the resident evaluation form), data regarding the number of resident evaluations completed and returned by attending psychiatrists were collected 4 months before and after the intervention. Chi-square analysis was utilized to determine if the numbers were statistically significantly different before and after the intervention.

Barriers to Completion

Based on the results from the initial questionnaire, completed by only 3 of the 21 attending psychiatrists, as well as the open discussion with the attending psychiatrists during a medical staff meeting, several reasons were posited as barriers to completing resident evaluation forms, including a) length of the form; b) availability of second copies of the form; and c) lack of reminders regarding completing the form.

A seven-item questionnaire was developed following the above input to further elicit information regarding resident evaluation completion history and rating of potential interventions to increase completion rates. Sixteen of the 21 attending psychiatrists (76%) responded to this second questionnaire. Of these, only approximately 50% indicated that they always completed the resident evaluation forms. In addition, only 19% reported that they always completed evaluation forms on time (See Table 1). When asked to identify potential interventions that could help improve the completion rates, most (88%) of the attending psychiatrists indicated that an e-mail with an attached electronic copy of the evaluation form would be “very helpful” (Table 1).

Intervention and Outcomes

Prior to any changes being made, the residency coordinator from the residency training office mailed a paper copy of the evaluation to each of the psychiatrists within 2 weeks following the start of the rotations. No reminders or additional evaluation forms were sent. Based on the results of our questionnaire, the authors elected to continue having the residency coordinator mail a paper copy of the evaluation within 2 weeks following the start of the rotation. However, the residency coordinator elected to e-mail a reminder to all attending psychiatrists within a week prior to the end of the rotation and attached an electronic copy of the evaluation form as an intervention to increase completion rates. The authors did not decrease the length of the form because it was believed that the information collected was necessary.

Importantly, our data revealed a statistically significant increase in completion rates after implementing the e-mail reminders with attached resident evaluation forms. During the 4 months prior to the intervention, 21 of the 44 resident evaluations were completed by the attending psychiatrists (48%). During the 4 months after implementing the reminder and electronic copy of the evaluation form, 35 of the 44 resident evaluations were completed, for a completion rate of 80%. Chi-Square analysis indicated that this increase was statistically significant (χ²=9.63, p=0.002).

Prior to our intervention, the return rate of evaluations was 48%. Following the intervention, the return rate increased to 80%. Several factors may have been related to the attendings’ initial poor completion rate of resident evaluation forms. First, based on the poor initial response to the survey and the ensuing discussion, it seems that the type of items and the means of delivering them to the psychiatrist had an impact on response rates. Specifically, the items were considered too broad and the physicians did not believe that they would provide any useful information. Also, since the surveys were placed in the physicians’ mail boxes, they did not perceive the completion as being important. Following the focus given to completion of evaluations during a medical staff meeting, the response to the next survey increased dramatically. Importantly, after instituting the use of an e-mail reminder and electronic copy of the evaluation form, the completion rate of evaluations increased significantly. Apart from the emphasis given to the importance of completing evaluations by making it an agenda item at the medical staff meeting, it appears that reminders may be helpful to busy physicians, who may delay completion of the forms when prompted initially and then may forget to return to the task unless reminded. A third intervention that may have contributed to the increase in response rates is attaching a blank form to the reminder e-mail, thereby making it more convenient for the attending physicians to complete the form.

Limitations

There are several limitations to the generalizability of this study. First, the sample size is small, consisting only of attending psychiatrists in one inpatient psychiatric facility. Second, it is not possible to determine whether one or all of the interventions had an impact on the completion rates (e.g., reminders, convenience of attached forms, or heightened focus on the completion of the forms). Finally, this study only covers the 4 months prior to the intervention and the 4 months following the intervention. It is not possible to say whether the increased response rate would continue over a longer period.

Future Directions

Gathering information about the reasons why attending physicians did not complete evaluation forms, and then responding to their suggestions, was valuable because the process was effective in increasing completion rates. Our intervention included three parts: focusing on the importance of the forms, sending reminders to the attendings 5 days prior to the due date, and attaching blank resident evaluation forms to the e-mail reminder. Therefore, as stated in the limitations above, it is not clear whether one or all of these were responsible for the increase in completion rates. It will be important to continue to monitor completion rates for a year, when only reminder e-mails and attached blank forms are being sent without any specific focus on the importance of completing the forms. Following the one-year implementation, it is recommended that the completion rates for the year be compared with those of the previous year, before the new system was implemented, in order to determine whether the improved completion rates are sustained. In addition, there still is room for improvement because 100% of evaluations should be completed, given their importance, and only 80% were completed following implementation of the intervention. As such, it is recommended that additional discussions with the attending physicians occur in order to determine other factors that may be deterring some physicians from completing forms. For instance, it would be quite serious if physicians were not completing negative evaluations because they did not want to face any negative repercussions, such as poor relationships with the residents. However, such residents are the very reason why the authors need evaluation forms in order to ensure that residents are competent before they are promoted to the next level. Perhaps if incompetencies were documented by all physicians, there would be less reticence about providing residents with negative feedback. Another possible future direction would be to implement a system whereby those attending physicians who did not complete evaluation forms would be required to provide a verbal report through a venue such as the promotions committee meetings. Although such negative reinforcement is unlikely to be popular, it might be effective for those physicians who never complete the evaluation forms.