Traditional concepts support the idea that evaluation of the results of learning can promote students' motivation to learn. On the other hand, many educators emphasize that the curriculum, not evaluation, is the most important component in any clinical course (1). In reality, students may feel overloaded by work and respond by studying only for the parts of the course that are assessed. Normative evaluations can be educational and can promote learning. Therefore, students can learn from tests and receive feedback on where to build their knowledge and skills. Pragmatically, assessment is one of the important engines on which to harness the curriculum.

The pyramid of clinical ability introduced by Miller (2) is a concise conceptual model that outlines the dimensions and essential facets of clinical ability that medical educators need to evaluate when they design test content. In Miller's theory, knowledge is the fundamental part of clinical ability. The multiple-choice examination is the most widely used format for testing knowledge (3, 4). These tests have high reliability because of the large number of items that can easily be tested and scored. However, there is criticism of multiple-choice questions, such as the cueing effect, the "single best answer," extended matching, and long and short menus of options (5, 6). Many researchers have also argued that scores do not necessarily correlate with clinical ability (7).

Standardized patient (SP)-based evaluation has gradually become an important method for assessing clinical ability (8, 9). The use of an SP evaluation during a psychiatric clerkship provides a potentially valuable experience that may improve the students' interviewing skills (10, 11). An SP is usually used in assessment of "shows-how" in Miller's theory. However, idealized, "textbook" scenarios, which may not mimic real-life situations, and the high cost are the disadvantages of SP-based examinations (12). It is also less appropriate in assessment of clinical reasoning or judgment (13). The mini-clinical evaluation exercise (mini-CEX) is another assessment that has been used as a feedback tool in clerkships (14) and is feasible for formative assessment (15). The mini-CEX is appropriate for evaluation of "does" in Miller's pyramid (13). Its strengths are real-world use, low cost, and feasibility. However, although psychiatric practice has shifted to ambulatory settings, clerkships in psychiatry remain largely inpatient-based. This may be one of the problems of the mini-CEX because, given their inpatient-based clerkships, students have few outpatient encounters. Another disadvantage is that real patients selected for any clinical examination are bound to vary considerably in the degree of difficulty their cases present.

Each assessment tool has its strengths and weaknesses, and the evidence demonstrates that no single assessment method can successfully evaluate the clinical ability of medical students in psychiatric clerkships (16). Also, the assessment tools that are chosen may or may not be highly associated. Many studies have explored the use of a single assessment in medical students, but there is little research on multiple assessments, especially in psychiatry. The use of multiple, highly-associated assessments may be too time-consuming for students. On the other hand, assessments that are not highly associated can evaluate different aspects of clinical ability, and thus promote learning by giving feedback to the students. The aim of this study was to examine the relationship among the results from different types of assessments of medical students' clinical ability in psychiatric clerkships and whether it is reasonable to use these multiple assessments.

A total of 196 medical students (141 men, 55 women) in psychiatric clerkships between September 2008 and May 2009 were enrolled in this study. All students had completed the classroom portion of their psychiatric training and had finished clerkships in internal medicine, surgery, pediatrics, and obstetrics and gynecology before the psychiatric clerkships. At the beginning of the clerkship, students were informed that they would be assessed by three types of tests to improve their clinical abilities. All students agreed that the results of their evaluations could be analyzed. The psychiatric clerkship is a 4-week program with 2 weeks in the department of psychiatry in a university hospital and 2 weeks in two psychiatric teaching hospitals in Kaohsiung, Taiwan. Most training was done with inpatients. During the course in the psychiatric teaching hospital, students were evaluated by the mini-CEX. Students took the multiple-choice examination and the SP-based test at the end of the clerkship.

The multiple-choice examination was a 90-minute test consisting of 50 questions. Students were asked to choose a best answer to each question; 2 points were given for each correct answer, and total scores ranged from 0 to 100. The educator gave feedback when students finished the test.

For each mini-CEX encounter, the attending physician for the inpatient ward team completed the evaluation. This physician observed how the student conducted a focused interview with an inpatient or outpatient to gather clinical data and make a psychiatric diagnosis. Each student had one encounter in each psychiatric teaching hospital. The sections for the mini-CEX were the medical interview, physical examination, clinical judgment, humanistic qualities, counseling, organization, and overall clinical competence. The examiners used a 9-point scale (a score of 1–3 was "unsatisfactory;" 4–6: "satisfactory;" and 7–9: "superior") for each domain evaluating the student's ability. The interrater reliability was 0.68 (Pearson's correlation r). After completing the rating form, the examiners provided feedback to the students.

At the end of the training program, all students had an opportunity to individually conduct a 15-minute interview with a "standardized patient" (SP) presenting with a clinical case of suicidal ideation. All SPs were volunteers who had finished at least senior high school. The scenario was written by a member of the senior visiting staff. SPs were trained to portray a patient with suicidal ideation by lectures, videos, and discussions with experts. Every student interviewed an SP in an individual examination room, and the examiner observed the interview in the same room. After the interview, the student left the examination room, and the examiner spent 10 minutes completing a global rating of the student's clinical ability in communication skills and gathering data, and then gave feedback to the student. The scores ranged from 0 to 100. Then the examiner and the student met for a 5-minute discussion about the interview.

Analyses were conducted with JMP software (SAS Institute, Inc., Cary, NC, U.S.). The scores of the three assessments were calculated and compared between female and male students by use of the t-test. The correlations among the scores on the multiple-choice examination, SP-based test, and mini-CEX were analyzed with the Spearman ρ test. We set inferences at the 0.05 level of significance for all inferential statistical procedures.

The scores on the three assessments are shown in Table 1. The mean scores on the multiple-choice examination and SP-based test were 86.90 (standard deviation [SD]: 2.66) and 80.85 (SD: 4.42), respectively, and the mean scores of mini-CEX domains ranged from 3.85 to 4.81. The correlations among the scores on the three assessments are shown in the Table 1. The correlation coefficient between the multiple-choice examination and SP-based test was statistically significant (r=0.33; p<0.0001). There was a weak but statistically significant positive relationship between the multiple-choice examination and score on the overall clinical competence domain on the mini-CEX (r=0.20; p=0.0057). Meanwhile, the scores for the clinical judgment and organization domains on the mini-CEX also positively correlated with the scores on the multiple-choice examination. However, the scores for the medical interview, physical examination, humanistic qualities, and counseling domains did not statistically associate with the scores on the multiple-choice examination. All domains of the mini-CEX except the physical examination and counseling domains positively correlated with the SP-based test.

According to Miller's pyramid, all three assessments can theoretically be used to evaluate different levels of clinical ability. Previous studies of undergraduates and postgraduates have found acceptable reliability and validity in SP-based tests (1, 13, 17, 18). This study found a positive significant association between the scores on the multiple-choice examination and the SP-based test in psychiatry, which is in line with the study of Walters and colleagues (19). However, the correlation coefficient between the scores on the multiple-choice examination and SP-based test was 0.33, which is a weak level of correlation. This may be because multiple-choice examinations evaluate students' knowledge, whereas the SP-based test evaluates clinical skills and attitudes toward patients with psychiatric disorders, in addition to knowledge. Smee (20) suggested that the SP-based test is effective in screening for sufficient clinical skills, which cannot be assessed by written examination. Holmboe and colleagues (13) concluded that SP exercises measure unique components of clinical competence and may be considered complementary to other evaluation tools. According to the results of our study and previous studies, it would be reasonable to use multiple-choice examinations and the SP-based test to improve clinical abilities because these two assessments evaluate different aspects of clinical ability.

The use of the mini-CEX in medicine clerkships has been examined in a previous study (15); however, there is little research on its use in psychiatric clerkships. The significant correlation between the scores on the mini-CEX and multiple-choice examination in this study is in line with previous studies of medicine clerkships and residency training (15, 21). However, the weak correlation between the scores on the mini-CEX and the multiple-choice examination indicates that these two tests may measure different components of competence. Similar results were reported in a study in internal medicine (13). Therefore, it may be reasonable that students be assessed by both the mini-CEX and the multiple-choice examination.

The benefits of using "standardized patients" versus real patients for assessing students' learning remains an area of interest. The advantages of SP-based tests are controlled environments, safety, and the ability to tailor scenarios to the level of skill to be assessed. Our results showed a fairly low correlation between the scores on the SP-based test and the mini-CEX. This trend is also supported by previous studies (15, 19). Rao and colleagues said that although the standard patient is a simulation of a real-life situation, the idealized "textbook" scenario may not be the same as in real-life conditions (12). On the other hand, patients in the mini-CEX were real patients with psychiatric diagnoses. One possible reason for the weak correlation between these two assessments is that the mini-CEX can evaluate the student's ability to focus and prioritize a diagnosis, which cannot be done in simulation settings (22). Another possible reason may be the difference in context between the two examinations. In this study, the SP-based test was designed to evaluate assessment of suicidal ideation, whereas the mini-CEX focused on obtaining a psychiatric history. The results of previous studies support the notion that the SP-based test and mini-CEX are context-specific (15, 19, 23).

Some situations can be simulated well, but some cannot. For example, an SP can only portray a limited set of physical symptoms (24). Because of this reality, some situations are suitable for SP-based tests and some for mini-CEX. The strategy of combining an SP-based test and mini-CEX may be appropriate for learners and educators. Because of the weak correlations among the three assessments, each test may evaluate different aspects of clinical ability. To promote more efficient learning, it is reasonable to use all three tests.

Some limitations of this study should be kept in mind. First, the medical students in this study were from a single medical school. Further study is needed to examine the results in medical students from other medical schools. Second, insufficient encounters in the mini-CEX may result in unstable test–retest reliability. Third, the three tests in this study were conducted at different time-points during the 4-week psychiatric clerkship. Students may improve their clinical ability during the 4 weeks. If these three tests are conducted at the same time-point, the association among them may be different from those in this study.

On the basis of results of this study, we suggest that it is necessary to use multiple methods to assess medical students' competency in psychiatric clerkships. There is no gold-standard tool for evaluation of clinical ability (2, 13). A multiple-choice examination, mini-CEX, and SP-based test may evaluate different dimensions of competency described in Miller's theory. The implementation of a variety of methods to assess clinical ability can address multiple dimensions of competency in psychiatric clerkships. Given that there were some limitations, this should be considered a preliminary study. Further study is needed.

This study was supported by Grant 9M38, awarded by Kaohsiung Medical University Hospital, Kaohsiung, Taiwan, ROC. At the time of submission, the authors reported no competing interests. Dr. Cheng-Chung Cheng, Dr. Frank Huang-Chih Chou (who is also Adjunct Associate Professor, Meiho University), Dr. Hin-Yeung Tsang, Dr. Mei-Feng Huang, and Dr. Cheng-Fang Yen are affiliated with the Dept. of Psychiatry, Kai-Suan Psychiatric Hospital. Dr. Cheng-Fang Yen is also affiliated with the Graduate Institute of Medicine, College of Medicine, Kaohsiumg Medical University, Kaohsiung, Taiwan.