Unprofessional behavior of medical doctors can lead to poor patient outcome, and growth of consumer awareness has led to a greater call for professional accountability. Studies have shown that documented unprofessional behavior in medical trainees leads to an increased likelihood of disciplinary action among practicing physicians (1). The Royal College of Physicians and Surgeons of Canada (RCPSC) has identified the professional role as one of seven CanMEDs roles (2). Similar competencies have been endorsed by the Accreditation Council for Graduate Medical Education (ACGME) in the United States. Postgraduate medical training programs are mandated to evaluate trainees in this role (2); however, methods to evaluate professionalism remains a key challenge to medical education (3).

Although a variety of methods for evaluating this competency have been explored in the literature (4), the RCPSC guidelines for evaluating residents on the professional role recommend direct observations and in-training evaluation reports by the supervisor and multi-source feedback as the preferred assessment methods and recommended tools (5). Multi-source feedback, or 360-degree global rating evaluation, consists of measurement tools completed by multiple categories of observers who have had the opportunity to interact with evaluees and observe them performing a task. It is possible that residents interact differently with their supervisor than they do with patients, resident colleagues, nursing staff, and other members of the allied health team. An evaluation that is completed by multiple members of the healthcare team would offer insight into residents' professional behavior outside of the scope of the supervisor's observation and in-training evaluation reports.

Despite routine use of direct observation and the in-training evaluation reports at our center, uniform use of multi-source feedback is less often applied, and, when it is, it is in an abbreviated fashion. Previous studies have demonstrated that multi-source feedback evaluations can lead to more comprehensive feedback and improvements in resident performance, particularly when baseline adherence to recommended practice is low and intensity of feedback is high (6). In this study, a needs-assessment survey was conducted to help understand the perspectives of psychiatry residents at the University of Toronto with respect to the current mode of evaluation of the professional role and their opinions on incorporating MSF into resident evaluation.

A structured survey was developed to assess the perceptions of residents in the University of Toronto psychiatry program on the evaluation of the CanMED professional role during training. The survey consisted of 12 subjective questions, 10 using Likert-scale answers and 2 using free answers. A priori research ethics approval was obtained from the University of Toronto Research Ethics Board.

The survey was distributed directly to 128 residents in postgraduate years (PGY) 1 through 5 enrolled in the psychiatry program between February 2, 2009 and April 22, 2009. Surveys were collected in an anonymous fashion, and participation was voluntary. Residents who had not been in attendance at core teaching seminars when surveys were distributed or were away on vacation or elective were e-mailed an online version of the survey. Survey results were analyzed with descriptive statistics.

Of the 128 Psychiatry residents given the survey, 110 responded, yielding a response rate of 86%.

Residents believed that the professional role was one of the top three most important CanMED competencies with respect to perceived importance to patient care. Despite this, fewer than half of the residents (49%) felt that the professional role was being adequately evaluated during psychiatry residency training. Most residents (70%) felt that patient care could be enhanced by improving the evaluation of the professional role.

All residents (100%) felt that their primary supervisor should be involved in their evaluation of the professional role using multi-source feedback. This was followed by the allied health staff (90.9%), patients/family members (72.7%), self-evaluation (60.0%), and co-residents (57.3%), whereas about half of residents believed that administrative staff (54.5%) should be involved in their multi-source feedback evaluations. Most believed their primary supervisor should have the largest proportionate weight of their evaluation (50%), followed by allied heath staff (19%), patients (16%), co-residents (12%), self (11%), and administrative staff (9%). Overall, 83% of the residents were in favor of incorporating multi-source feedback for the evaluation of the professional role, whereas 11% were undecided, and 5% were not in favor.

Approximately two-thirds of the residents responded to the qualitative questions. Their comments highlighted three main themes related to the implementation of multi-source feedback evaluation; they were 1) concern about the resource requirement; 2) effect on the culture of learning; and 3) validity of feedback. For example, one resident stated “I believe that multi-source feedback will be very useful for evaluating this competency, but I worry that it will be too challenging to implement from a resources standpoint.” Others raised concerns that multi-source feedback may be used to penalize residents or that they may feel “overly scrutinized,” as one stated. Some residents expressed concerns about the relationship between evaluations and clinical outcomes beyond their control. For instance, a resident stated, “I would be concerned if the patient was not happy with treatment outcomes. This may be reflected in the evaluation, despite its not being the failure of the resident.” Also, concerns regarding the suitability of non-MD staff in their role as evaluators were expressed.

To our knowledge, this is the first study looking at psychiatry residents' perception of multi-source feedback. Residents agree that the professional role is among the top three most important CanMEDs roles for patient care. As demonstrated in previous studies and the RCPSC (7), multi-source feedback is a preferred tool for the evaluation of residents on this role. Over two-thirds of the residents believed that an enhanced method of evaluation would lead to improved patient care. Over 80% of the residents were in favor of incorporating multi-source feedback in the evaluation of the professional role. Residents indicated that their primary supervisor should still have the greatest weighted input on their evaluation. There was some discrepancy regarding which members of the evaluees' clinical sphere should be involved in their evaluation process. In the event of a poor evaluation, the residents believed that their primary supervisor should provide feedback. A potential limitation of this study is that it was conducted at a single center, which may limit generalizability of the results.

Resident's responses to the qualitative questions offered insight in to the areas of perceived challenge with respect to implementation of a multi-source feedback evaluation tool. Residents voiced a concern related to the resource cost to time; both for the collection of data and providing adequate feedback. Previous studies indicate that evaluators have expressed concern with the increased time and effort required for the evaluation process (8). Although multi-source feedback surveys actually take little time to complete, a study of radiology staff found that scheduling the small amount of time to complete the multi-source feedback was challenging and resulted in 60% of the staff physicians' finding this evaluation tool burdensome (9). Ultimately, the majority of the burden would fall upon the primary supervisor, who would likely be responsible for assimilating the data and providing informative feedback. Strategies to increase efficiency of this evaluation method, such as electronic data collection and assimilation and increased administrative support would function to streamline the evaluation process and reduce time burden, thereby increasing compliance.

Although the vast majority of residents agree with the integration of multi-source feedback, some expressed concerns regarding a lack of trust in the process. As long as the feedback is offered in an informative fashion, this concern should be mitigated. The multi-source feedback should not be tied to overall performance unless there are issues of gross deficiencies or exceptional strengths. The purpose of this tool is informative rather than summative feedback. Last, in the interest of transparency, it is critical that before implementation of this evaluation tool, residents are given clear guidelines outlining the assessment tool domains and the evaluation process.

An interesting feature of multi-source feedback that is particularly salient to psychiatry is the capacity of patients providing feedback. Effects of transference may play a role in biasing responses. However, if an adequate number of evaluations have been completed, the majority will likely dilute the outliers. Previous studies have shown that a wide range of raters is required to achieve validity; as such, individual questionnaires would need to be studied to identify the appropriate number of raters to achieve validity (10). Interestingly, in a study by Wood et al. (9), radiology residents felt that the patients' feedback was important and may empower patients. Second, residents raised concern about the validity of the evaluations from the allied health staff. To reduce this gap, the role of the residents and their professional expectations must be clearly understood by all evaluating parties and the evaluee. Nonetheless, other studies have shown that a vast majority of residents found the process favorable and appreciated feedback from sources that would normally not assess them (9).

The data presented indicate that the resident body generally supports the integration of multi-source feedback for their evaluation of the professional role. However, before implementation, acceptance would need to be obtained from other members of the healthcare team and auxiliary staff involved. Understanding the potential long-term benefits of multi-source feedback, the staff would have to also accept the increased time commitment for all members involved in the evaluation process.