Self-evaluation has gained importance in medical education because it helps trainees in self-reflection and in developing insight into their strengths and weaknesses, two necessary skills for professional life (1). But do trainees really understand their weaknesses or deficiencies? Interestingly, research in self-assessment skills has shown that deficits in the ability to assess one’s own competence are common among subjects who perform poorly (2, 3). This lack of insight about one’s deficits has been ascribed to relatively poor metacognitive skills, leading to lack of understanding about weaknesses and poor training in self-assessment. This could be overcome by providing both earlier knowledge and subsequent feedback regarding the desired competency (3, 4).

Over- or under-estimation of performance was observed among residents in family medicine evaluating their patient-interviewing skills (4) and pharmacy students’ self-evaluating seminar performance (5). There is limited research in this area, particularly in psychiatric residency training. Resident-run seminars or lectures are a useful and commonly-used academic exercise in several settings where the trainee has to make a presentation on a predetermined topic that is followed by a discussion involving the audience, consisting of faculty and other residents. It involves skills of compilation of literature, critical evaluation, and communication with peers and teachers. Despite the wide use of such seminars and the growing importance of self-evaluation in training, only one study, conducted among trainees in a pharmacy course, has examined this aspect (5). In this study, we compared faculty evaluation of seminar presentations with trainee self-evaluation of their own presentations. This method was chosen to examine trainees’ ability to self-rate, as self-assessment maps onto important competencies for any physician. The resident-run seminar was chosen as a medium for examining self-evaluation, because it is a pragmatic and commonly-used method of training in most psychiatry training centers in India.

The study was conducted at the National Institute of Mental Health and Neurosciences at Bangalore, in India, a center for postgraduate psychiatry training. Trainee performance was assessed in 50 resident-led seminar presentations, using a 12-item, semi-structured instrument developed for the purpose. The tool was developed by consensus among the faculty members in charge of training, after discussions and a literature search regarding presentation skills and assessment. This reduced the risk of individual bias and enhanced content validity. Internal consistency (a necessary condition for construct validity) of this tool was acceptable. A Cronbach’s alpha of 0.947 demonstrates that the 12 items in the assessment rubric consistently measured students’ presentation skills. A high degree of internal consistency demonstrated by the assessment rubric also indicates good convergent validity.

The instrument consists of 12 items, assessing completeness (breadth of coverage of topic), relevance of content to topic, organization of slides, time-management, presentation style, creativity in presentation (use of flow-charts, tables, etc.), handling questions from the audience, summarizing and concluding, critical evaluation of the topic, and quality of handouts (content, layout, and referencing). Each item is rated on a scale of 0 to 10. All seminars (50) during the study period were rated by at least two faculty members immediately after the seminar, with the raters being blind to each other’s rating. The mean of the two raters’ scores was taken. This measure eliminated the chance of rater bias. The trainees rated themselves on the same tool within 24 hours of the presentation. Informed consent from the respondents and approval of the department chair was obtained before the study. Analysis was done with SPSS Version 10, and paired t-tests were used to compare the ratings.

Trainee self-assessments were higher on all 12 items than were faculty assessments (Table 1). Significant differences were shown on the following seven items: relevance of content (p=0.015), organization of slides (p=0.002), time-management (p=0.009), creativity in presentation (p=0.008), handout content (p=0.004), quality of handouts (p=0.003), and referencing (p=0.017).

This study attempted to compare trainee self-evaluation and faculty evaluation of resident-run seminars by use of a standardized rubric. The results revealed that trainees tended to overestimate their performance in all areas of seminar presentation. This finding is consistent with those of other studies comparing student and peer assessment in academic exercises (5–7). Several factors might contribute to this discrepancy. Students may not be clear on what is expected of them. It may also indicate inadequate self-assessment skills, resulting in an overestimation of one’s performance. Hodges et al. (4) have emphasized the need for training in self-assessment at all stages of medical education to overcome this problem. Several methods can be used to enhance self-assessment. Better and more detailed training in various aspects of seminar preparation and self-assessment might help. Benchmark presentations that help in role-modeling and practice sessions that include self-assessment are other methods. The value of immediate and analytical feedback in enhancing trainee self-assessment has also been emphasized (2, 4).

Strengths of our study include an adequate sample size and use of a rating instrument with good psychometric properties. Rater bias was eliminated by independent rating and using the mean of two ratings. However, as the study was conducted only at one training site and involved only one method of training, the results may not be generalizable. Further studies need to be done using the same tool at different training sites. More studies need to be conducted on methods of enhancing self-assessment among psychiatry trainees. Research is also needed to examine whether enhancing self-assessment and providing effective feedback actually improve performance.