In response to team-based care and recognition of psychiatry trainees’ exposure to a complex healthcare system, psychiatry residents are being introduced to the term “physician-manager” (PM) or “physician-leader” in an effort to recognize this growing role (1, 2). Both the Accreditation Council for Graduate Medical Education (ACGME) and Royal College of Physicians and Surgeons of Canada (RCPSC) have acknowledged the need for greater administrative training and have included these specific domains within their general training competencies for residents (2–5). Moreover, the Medical Leadership Competency Framework in the United Kingdom (6) and a 10-day leadership course for postgraduate trainees in Denmark (7) further highlight the global importance of physicians’ being better managers.

In Canada, the CanMEDs framework states, “As Managers, physicians are integral participants in healthcare organizations, organizing sustainable practices, making decisions about allocating resources, and contributing to the effectiveness of the healthcare system” (2). The Manager role overlaps with the ACGME Systems-Based Practice (SBP) competency and encompasses skills in managing practice finances, participation in quality improvement (QI), and leading change. Despite the development of specific learning objectives for this role, a 2001 RCPSC survey of faculty reported that the Physician-Manager role was one of the most difficult competencies to teach and evaluate (4).

Several needs-assessment studies have identified the need for PM training in response to the CanMEDs framework and the need for developing future physician leaders (3, 8–10), and, as a result, PM training initiatives, ranging from introductory seminars to longitudinal curricula involving the Faculty of Management, have been introduced into several postgraduate training programs (11, 12); these include a PM curriculum for psychiatry residents (13).

Although training opportunities for medical residents now exist in most programs, assessment of postgraduate trainees in the PM and leadership domains are predominantly limited to in-training global rotation rating scales (14), a trend that is problematic because of concerns about the validity and reliability of this assessment method (5, 15). Studies have confirmed poor reproducibility of in-training global rating scales because of the “halo effect” (when all ratings are influenced by one characteristic), leniency of raters, skewing of ratings to higher scores, supervisor concerns with failing trainees, and poor rater training (16–19). Moreover, data from psychiatry trainees showed that current PM assessment methods were inadequate and failed to provide meaningful assessment (13). It was suggested that clearer and more comprehensive assessment tools could improve the learning of PM competencies.

Both the ACGME (United States) (5) and RCPSC, CanMEDs assessment tool handbook (Canada) (14), recommend the use of portfolios as one assessment method for evaluating the SBP competency and the PM role, respectively. However, these competencies are relatively new, and the usefulness of portfolios in assessing resident competency has not yet been determined.

In medical education, portfolios have evolved into a multipurpose instrument utilized for assessment of how learners fulfill tasks and for documenting progression of competence (20). Assessment measures incorporated into portfolios include self-reflective papers, procedural and case logs, QI projects, chart notes, ethical dilemmas, patient surveys, and other types of learning material.

There are several advantages of portfolios over traditional methods of assessment, such as their ability to provide both formative and summative feedback, ongoing assessment of learning performance, opportunities for self-directed learning, and longitudinal assessment over multiple settings (21–23). Challenges of ensuring interrater reliability and validity and difficulties in assembling content are potential obstacles in employing this assessment tool (23, 24). However, the use of multiple raters (internal and external assessors), five or more entries, a pass/fail grading system, detailed descriptors, and web-based implementation can address these limitations and may enhance the use of portfolios in medical evaluation, and, potentially, the PM role (23–27).

Given the paucity of literature on PM assessment methods, it is unclear whether portfolios will be beneficial as a PM assessment tool. Norcini et al. have delineated the following criteria for good assessment: validity of coherence, reproducibility, equivalence, feasibility, educational effect, catalytic effect, and acceptability (28). The acceptability of portfolios in the assessment of the PM or SBP competency has been limited to one study, involving 16 Postgraduate Year (PGY) 1 to PGY4 psychiatry residents, who reported portfolios as being “somewhat effective” in assessing SBP competency (29). As a result, we conducted this larger study to determine 1) residents’ and educators’ perceived acceptability of currently-used PM assessment tools, and 2) the feasibility and acceptability of portfolios as a formative PM assessment tool.

Setting and Sampling

Psychiatry residents in their PGY2 to PGY5 training years at the University of Toronto and University of Alberta were surveyed in the 2009–2010 academic year and represent a large and a midsized residency program, respectively. PGY1 psychiatry residents were excluded because of their limited exposure to psychiatry training. A total of 124 and 37 PGY2–5 psychiatry residents were enrolled at the University of Toronto and University of Alberta, respectively. The proportion of male trainees at each institution was 34% and 29%.

At the time of this study, the University of Toronto provided a formal physician-manager curriculum delivered over 3 years of psychiatry training (details previously published) (13). Residents at the University of Alberta received PM seminars on conflict, professional communication, transition to practice, and QI. A physician health workshop was also offered to all residents in the program. Residents at both universities have opportunities to participate in committees and resident leadership roles.

In the second phase of the study, educators, who were postgraduate directors or faculty with experience in PM training, from 14 of the 16 total psychiatry residency programs in Canada (the two directors who were study investigators were excluded) were asked to participate in telephone interviews. Eligible educators were identified by psychiatry program directors, and interviewees provided informed consent. The study was approved by the University of Toronto and University of Alberta Research Ethics Boards.

Psychiatry Resident Questionnaire

A 9-item resident questionnaire was administered through resident meetings at the two study sites. Residents completed either an anonymous online or paper questionnaire, depending on the hospital site. Paper surveys were returned to an independent administrator. Online surveys were administered to those residents who could not attend the resident meetings.

The questionnaire asked residents to rate five statements about their perceptions of PM assessment in their respective residency programs. These statements ascertained the trainee perceptions of PM evaluation and expanded on preexisting needs-assessments by evaluating perceived acceptance of current PM assessment methods, acceptance of portfolios, and receptiveness to changing assessment methods. Statement responses were recorded on a 5-point Likert scale, with scores ≥4 coded as Agree and scores ≤2 coded as Disagree. The survey also included questions about respondents’ sex and level of training.

Additional survey questions asked respondents to indicate the two most useful assessment tools to be included in a PM portfolio tool for each PM domain. Each PM domain was based on a previously published PM needs-assessment survey (9), and the list and definition of portfolios and portfolio content were based on the CanMEDs assessment tools handbook, review of the literature on portfolios, and expert feedback (14). Although not included in the CanMEDS handbook, PM workshop attendance logs were included as possible content for the portfolio, given that PM workshops offered in the surveyed residency programs provided interactive components facilitating learner acquisition of PM knowledge and skills. Logs tracking resident participation in interactive PM workshops would be similar to clinical exposure logs recommended by the CanMEDS assessment toolkit (14) and workshop attendance forms being used anecdotally to assess residents in Canadian psychiatry residency programs.

The questionnaire was reviewed by psychiatry educators and piloted with a psychiatry resident focus group, and their feedback was incorporated into the final questionnaire.

Qualitative Educator Interviews

Fourteen psychiatry program directors and educators with experience in PM training or supervision were invited to participate in the study. A research assistant conducted telephone interviews, using an interview template consisting of standardized questions, and questions evolved on the basis of emerging themes (template available from the authors on request). For questions related to portfolios, a definition was provided to interviewees upon request. All interviews were tape-recorded and transcribed for analysis.

Statistical Analysis

We used SPSS 18.0 software to analyze residents’ responses to the questionnaire and performed a Mann-Whitney-Wilcoxon test to compare responses on the five Likert scale questions about residents’ perceptions of current PM assessment methods in their respective programs.

Program-director and PM education expert interviews were analyzed by use of a grounded theory approach, and recruitment continued until saturation was achieved. Transcripts were analyzed with HyperRESEARCH 3.0, a qualitative software, and by manual techniques. The first two transcripts were coded independently by two separate raters, an author (SS) and a research assistant, using constant comparative method (30, 31). Raters identified higher-order categories based on unifying themes in order to minimize formation of categories based on researchers’ biases. After the initial independent coding by the raters, categories and coding were reviewed to reach a consensus, and the research assistant proceeded with coding of remaining transcripts. One study investigator reviewed the research assistant’s transcript coding, and saturation was determined if two consecutive transcripts did not yield new themes.

Psychiatry Resident Questionnaire

The overall response rate was 54.7% (88/161), and the response rates for each university were 46% and 84%, respectively. Respondents’ mean age was 30.3 years (standard deviation [SD]: 3.2 years); 31.8% of respondents were men (N-28). The PGY level of respondents were the following: PGY2: 30% (N=26); PGY3: 28% (N=25); PGY4: 23% (N=20); and PGY5: 19% (N=17).

Only 43% of resident respondents reported having clear PM objectives for their psychiatry rotations during their training. Nearly half of respondents (49%) agreed that the PM role should involve several methods of assessment. Approximately 47% of respondents indicated that they did not want to change the way they were being assessed in the PM role. Fewer than half of residents (42%) indicated that it would be difficult to incorporate portfolios into their residency training assessment. Most residents were satisfied with their program’s current assessment of the PM role (77%). There were no significant differences in responses between the two universities.

Table 1 summarizes residents’ responses regarding which portfolio assessment methods would best evaluate trainees in each PM domain. Only QI assessment methods differed between the two training programs, with residents at the University of Toronto and the University of Alberta recommending chart recall and workshops attendance, respectively, as the most effective assessment tool (p=0.02). Results did not differ on the basis of PGY level.

Qualitative Educator Interviews

We stopped recruiting interviewees after analyzing a total of 14 interviews, having achieved saturation of themes at this sample size. Interviewees were recruited from five different Canadian psychiatry residency programs, and three interviewees were psychiatry residency program directors. Analysis of interview transcripts revealed a total of six interview themes:

1. Current PM Assessment Practices

Many respondents identified in-training global rating scales as the primary method currently being used to evaluate the PM role in their respective programs. Less frequently, psychiatry residents’ PM skills were assessed using Observed Structured Clinical Exams (OSCEs) that included the PM role as a component of the evaluation. The use of resident quality improvement (QI) projects to evaluate residents in this PM domain was used in only one residency program.

2. Supervisor Barriers to PM Evaluation

Psychiatry program directors and educators discussed the challenge of evaluating residents in the PM role because of the PM’s role being poorly defined in psychiatry residency programs. Moreover, educators stated that ambiguous assessment forms without clear descriptors for PM competencies were a limitation of current in-training global rating scales. One educator stated:

Moreover, educators mentioned the lack of faculty development during the introduction of the PM role and the need for faculty training on how to evaluate trainees in the in PM role. As a result, educators felt ill-equipped to accurately assess residents in the PM role in the context of busy clinical practices. Moreover, educators indicated that a formal orientation to assessment methods for the PM role would be beneficial.

Time constraint was identified as a predominant supervisor barrier during the interviews. Educators indicated that supervisors are often managing clinical pressures that limit the ability to provide detailed feedback and to use an array of assessment methods related to the PM role.

3. Resident Barriers to PM Evaluation

Residents are currently faced with increasing training requirements in Canadian psychiatry residency programs, and educators identified clinical workload issues facing residents, because of a finite number of supervisors and residents, as barriers to introducing PM assessment tools.

Limited exposure to PM training opportunities was identified as an additional resident barrier. Interviewees stated that residents might lack empowerment to take on leadership roles in which they could be appropriately evaluated in the PM role. A need for more formal PM training opportunities was suggested.

4. Alternative PM Assessment Approaches

Educators identified a need for expansion of PM assessment tools, and the inclusion of reflective tools emerged as an important theme during the interviews.

Additional PM assessment tools identified by educators included OSCEs, 360-degree evaluations, and multisource feedback (MSF), which were all being “trialed” in residency programs outside of psychiatry in some universities. Allied healthcare professional contribution was felt to be essential to the use of these tools in order to enhance resident feedback. One educator stated:

5. Merits of Portfolios for Reflection and Formative Evaluation

Educators agreed that reflective opportunities were key components of a portfolio assessment tool for the PM role. It was felt that reflective components should be supplemented with a list or log of all PM educational experiences that a resident has had during training. The benefit of a log would be to enhance tracking of training experiences and trainee reflection over the course of residency.

6. The Use of Portfolios: Challenges With Feasibility

Most educators cited the time commitment needed to administer, coordinate, track, and complete portfolios as a potential challenge to using portfolios for assessing the PM role. Based upon the challenges with time commitment and coordination, educators noted the need for a solid infrastructure to support the implementation of portfolios during residency training.

The results of this study demonstrate that current use of PM assessment methods in Canadian Psychiatry Postgraduate Programs is in its infancy. Approximately 50% of residents indicated that clear, rotation-specific PM objectives were currently used in their training program. This complicates accurate assessment of the PM role and raises questions about the validity of widely-used in-training global rating scales as the dominant means to assess this competency in residency programs. Among the assessment tools, only responses about QI projects differed between residency programs; likely because of the development of a formal QI curriculum at the University of Toronto (32). Despite residents’ comfort with current assessment methods, both educators and trainees in our study were receptive to changing the assessment methods for the PM role, provided that the workload demands of these new methods of assessment were taken into account.

Using the assessment framework proposed by Norcini et al. (28), our study addressed two components of acceptability and feasibility of PM assessment tools. Most residents (77%) reported feeling satisfied with the status quo for PM assessment, likely representing reluctance by residents to accept change of PM assessment because of concerns about workload burden and lack of awareness of alternate PM assessment tools. There was a strong preference by residents for using workshop attendance records as a PM assessment tool, and this is likely a product of resident “comfort” with a “familiar” assessment method. The lack of evidence for workshop attendance records in combination with educators’ lack of support for this tool should raise questions about their ongoing use in the assessment of PM competence.

Feasibility is also a “criteria of good assessment,” and both residents and educators identified workload burden as key challenge to PM assessment. Portfolios can be resource- and time-intensive, and sufficient infrastructure, resources, and the use of electronic online platforms was felt to be important by trainees and supervisors in the success of using portfolios for PM assessment.

Validity and coherence are also components of good assessment; needs-assessment data (3, 9), combined with the recognition of PM domains within the CanMEDs framework, further support the validity of PM assessment. Furthermore, educators identified faculty development as a necessary component to successful PM assessment, and training of supervisors will be needed to support the use of more complex portfolios and enhance reproducibility and consistency. Research is needed to confirm the equivalency of implementing portfolios for PM assessment across Canadian psychiatry training sites. The catalytic (i.e., that the assessment tool results will drive future learning) and educational (that the assessment tool will motivate learners to prepare in a fashion that has educational benefit) effects of portfolios in PM assessment have yet to be determined and require further investigation to determine the effectiveness of portfolios in assessing PM competency.

Educators also identified reflective tools as an integral component of portfolios (33–35). The use of written reflections, defined as one’s understanding of an experience, as part of portfolios, may facilitate self-assessment of the trainee that can be later validated by accumulated evidence from portfolios (36). These suggestions for content of portfolios are congruent with the only other needs-assessment (29). In addition to portfolios, educators also favored the use of OSCEs focusing on PM domains, although this was not supported by the resident data.

Our results should be interpreted in light of some methodological limitations. Although the 55% response rate for the resident questionnaire is comparable to other PM surveys in Canadian settings (8, 9), the opinions of educators volunteering for this study and residents from the two residency programs may not be representative of perceptions of all psychiatry educators and residents in Canada. Our questionnaire results were also limited by residents’ biases toward familiar assessment tools and a general resistance to changing PM assessment methods. Despite these limitations, we feel that we were able to capture a helpful dataset that will enable educators to further improve the assessment of PM competencies.

In summary, our study provides preliminary data on the acceptability of a PM assessment tool largely based upon a portfolio comprising specific assessment components, including reflective exercises. Given the limited literature on assessment methods for the PM role and recent emphasis on competency-based training and office-based assessment, research evaluating PM assessment tools is timely. Further studies will be needed to evaluate the ability of these PM assessment methods to meet the criteria for “good assessment.”

We thank Anbreen Khizar for her work in coordinating the study. This study was funded by the University of Toronto Dean’s Fund Grant.

The authors have no conflicts of interest to disclose.