Examinations have played a major role in the medical education of psychiatrists. The process begins with the Medical College Admission Test, continuing with course examinations and the United States Medical Licensing Examination during medical school, then the clinical evaluations and examinations during residency, and ends with the written and oral examinations required for board certification. All examinations offer an assessment in some area that is ideally in the area that the examination was designed to test (+1).

The Psychiatry Resident In-Training Examination (PRITE) has been produced and given annually since 1979 as an educational experience to measure cognitive information for psychiatry residents and training programs. The development of a national examination followed the trend in medical specialties, which began in 1963 with orthopedics, to develop national in-training resident examinations (+2).

Smeltzer and Jones evaluated the reliability and validity of the PRITE from 1980 to 1987 (+3). The reliability of a set of test scores provides statistical information about how much an obtained score is likely to vary from a true, error-free test of the individual's ability. Because the purpose of the testing is inferential, the items should be a representative random sample of all possible questions. Reliability was calculated by Kuder-Richardson formula 20, which is used for estimating the reliability of instruments in which all items are scored dichotomously (+4, +5). The global score on each of the PRITEs evaluated had reliability comparable to standard certification examinations used in medical education. The 11 psychiatry subscores are derived from a smaller number of items than the global psychiatry score or the global scores in neurology, and their reliability coefficients were accordingly lower, and generally not acceptable for group comparisons (+3).

Smeltzer and Jones also examined the validity of the PRITE (3). They noted that content validity, the degree to which the items on the PRITE are accurate and reasonably representative of the knowledge that is needed to be mastered during a psychiatry residency, was appropriately evaluated. The construct validity of the examination is the qualities that the test measures and the degree to which constructs account for the performance on the examination (+3, +6). The researchers demonstrated this with the finding that there were significant increases in the mean examination score with each additional year of training.

Criterion validity, the degree to which the test results correlate with criterion obtained independently from the examination, has not been well studied (+3). Webb et al. reported that the national correlation between PRITE scores and subsequent performance on the American Board of Psychiatry and Neurology Part 1 written examination is 0.67, a fair correlation (+7). This finding reflects how well PRITE scores predict future performance (predictive criterion validity), rather than how well they evaluate the professional knowledge and competency of the psychiatry resident at the time of the examination. This study looks at concurrent criterion validity or the degree to which there is a correlation between faculty evaluations and performance on the PRITE in the same academic year for individual residents.

Residents in the general psychiatry and the medicine-psychiatry program at the University of Iowa from 1990—1996 were included in this study (postgraduate year [PGY]-1—PGY-5). The individual resident's total score on the psychiatry portion of the PRITE was used for the comparisons done. The individual PRITE scores were included for each resident for each year of the study. Medicine-psychiatry residents were included, and were assigned to their appropriate PGY, except that the fifth-year residents were included in PGY-4 analyses.

The departmental evaluation form used to assess the resident's performance on rotations was used as the faculty's evaluation of the resident's capabilities and knowledge (Appendix 1). The faculty evaluate the resident on a monthly basis in the PGY-1 and PGY-2. In PGY-3, the residents are assigned to the outpatient clinic for a 12-month period and are evaluated by each faculty member (from 8 to 14) in the outpatient psychiatry clinics twice (at 6-month intervals). There are at least four assessments of the resident's performance in PGY-4 and -5. The individual items on the evaluation were averaged for each form to obtain a composite score, and then the mean scores of faculty evaluations from the academic year were averaged. Pearson correlation coefficient was calculated to establish the degree to which the relationship between the two variables was linear.

There were an average of 27.7 residents in the general psychiatry and medicine—psychiatry program during the 7-year period included in this study. There were 194 PRITE scores from 86 individual residents during this period of time. The distribution of residents by year is shown in +Figure 1. Males represented 57.4% of the sample.

The faculty assessment includes 13 individual aspects of resident performance, ranging from professional knowledge to intellectual honesty and emotional maturity. There was acceptable interrater reliability for individual items across the year for PGY-1 through PGY-4 residents (kappa=0.72) and between raters at the same evaluation time for PGY-3 residents (kappa=0.78).

The correlations between the PRITE scores and faculty assessment of performance were calculated by postgraduate year. +Figure 2 demonstrates the linear correlation between faculty assessment and PRITE scores. In the PGY-1 group, the Pearson correlation equals 0.89. For the PGY-2 group, correlation equals 0.87. In the PGY-3 group, the correlation equals 0.82, and in the PGY-4 and -5 group, the correlation equals 0.85. Factor analysis revealed that the two items on the resident evaluation that were most correlated with PRITE scores were Item 1 (professional knowledge) and Item 13 (awareness of the literature) and the two least correlated were Item 5 (resident recognition of need for consultation) and Item 9 (effectiveness with patients). All were significantly correlated with PRITE scores.

This study evaluated the relationship between faculty evaluations of residents and their performance on the PRITE. The PRITE is given annually and is intended to be a significant educational experience for psychiatry residents, faculty, and training programs. It is used as an external criterion for the training program to evaluate the content and educational effectiveness of the curriculum. It is not intended to be used as a sole criterion for evaluating a resident, although the results can be used to evaluate the performance of individual residents. This study provides support for concurrent criterion validity of the PRITE.

This study demonstrates a correlation between the performance of the resident on the PRITE and the evaluation of the faculty of their professional knowledge and clinical ability. The faculty assessment of the resident can be viewed as the gold standard for the study. Faculty have direct contact with residents and observe their skills first-hand. They observe the resident's interviewing skills; fund of knowledge; and their ability to integrate the data that they gather to formulate a diagnosis, then delineate and carry out a treatment plan. They have personal interactions with both the resident and the resident's patient population. Therefore, they are in an ideal position to assess the skills of each resident physician. The linear correlation between faculty assessment and the PRITE results indicates that the PRITE is reasonably accurate at assessing the professional knowledge and ability of resident physicians.

The reverse may also be true. The PRITE has been shown to have a reasonable correlation with the resident's subsequent performance on the American Board of Psychiatry and Neurology written examination, and may be argued to be a standard by which to judge professional knowledge and ability. As the quality and quantity of medical information expands, faculty are increasingly responsible for assessing the adequacy of a resident's preparation and training to practice medicine outside the medical educational framework. This study demonstrates that faculty members, as a group, assess the professional skills and knowledge of residents consistent with their performance on standardized examinations.

The results of this study argue against a significant disservice being done to residents who test poorly on the PRITE. There may be a small percentage who have significantly better professional skills and knowledge than is demonstrated on the examination. However, in our resident population over a 7-year period, this occurrence was a rare event (1.1% of residents scored below 33% for their educational rank on the PRITE but were judged to be consistently above average or better by faculty assessment). This finding is partly explained by the process of matriculating and subsequently graduating medical school. Poor test takers do not often enter the resident population to begin with, and therefore poor test-taking skills are not as significant a factor in this population than they would be in a less selected group. Matthews and Ticknor reported that residents who took the examination were overall very satisfied with the PRITE (+6). The researchers felt that it played a valuable role in their education and was good preparation for future board examinations. The residents who were least satisfied with the PRITE were the residents who did poorly on the examination. The results of this study would suggest that the professional knowledge and skills of those who performed poorly on the PRITE would benefit from additional professional educational assistance and that the use of the PRITE as a part of the evaluation of the resident is appropriate.

This study of 86 individual residents over a 7-year time period demonstrates that the faculty assessment of residents is well correlated with their annual performance on the PRITE. While the PRITE has well-established construct validity, criterion validity is less firmly established. This study provides evidence that the PRITE can accurately evaluate the professional skills and knowledge of resident physicians.