This study aims to analyze the relationship between performance on The American College of Psychiatrists’ Psychiatry Resident-In-Training Examination (PRITE) and the American Board of Psychiatry and Neurology’s (ABPN) Part 1 examination.
A previous study of the relationship between the 1992 PRITE and the 1994 Part 1 examination yielded Pearson correlations of 0.67 (p<0.01) for psychiatry and 0.43 (p<0.01) for neurology (1). A study analyzing the relationship between performance on the in-training examination for neurology residents (RITE) and the Part 1 neurology examination yielded similar results. The correlations between the scores on the RITE and the neurology component of Part 1 and the psychiatry component were 0.75 (p<0.01) and 0.61 (p<0.01), respectively (2). Studies in family medicine (3), internal medicine (4), obstetrics and gynecology (5), physical medicine and rehabilitation (6), radiology (7), and urology (8) have also consistently yielded positive associations between performance on in-training and board-certification examinations.
The PRITE was developed by The American College of Psychiatrists to provide feedback to residents and training directors about the status of trainees’ knowledge compared with others at the same level of training. Almost all training programs use the PRITE to meet the Residency Review Committee’s requirement that a test of cognitive knowledge be administered annually. Hence, most psychiatric residents will take the PRITE three or four times during residency training.
The 2002 PRITE consisted of 246 multiple-choice questions covering growth and development (8.5%), adult psychopathology (16%), emergency psychiatry (6%), behavioral science and social psychiatry (8.5%), psychosocial therapies (8.5%), somatic treatment methods (16%), patient evaluation and treatment selection (8.5%), consultation-liaison psychiatry (6%), child psychiatry (8.5%), alcoholism and substance abuse (8.5%), and miscellaneous topics (5%). There was also a 54-item section covering neurology and the neurosciences, and both a global psychiatry and a global neurology score were reported. In October 2002, 5,316 residents, of whom 905 were advanced residents (30+ months of training), sat for the examination. The internal consistency reliabilities were 0.90 for global psychiatry and 0.70 for global neurology.
To qualify for the Part 1 examination, an individual must have satisfactorily completed residency training in psychiatry and possess an unrestricted license to practice medicine in a state, commonwealth, territory, or possession of the United States or province of Canada. Examinees who pass Part 1 are then eligible to take the Part 2 (oral) examination.
The 2003 Part 1 examination had two components. The psychiatry component consisted of 210 multiple-choice items covering development through the life cycle (6%), neurosciences (10%), behavioral and social sciences (5%), epidemiology and public policy (5%), diagnostic procedures (9%), psychiatric disorders (30%), treatment of psychiatric disorders (30%), and special topics (e.g., suicide, emergency psychiatry, ethics; 5%). The neurology component consisted of 130 multiple-choice items covering basic science aspects of neurologic disorders (20%), incidence/risk of neurologic disorders (5%), and diagnostic procedures related to, clinical evaluation of, and management and treatment of neurologic disorders (75%).
The examination was administered in November 2003 to 1,815 candidates, of which 1,110 were new candidates and 705 were repeaters. The internal consistency reliability coefficients were 0.92 for the psychiatry component and 0.91 for the neurology component.
Both the PRITE and the Part 1 examination were developed and reviewed by committees of subject-matter experts, including neurologists and test-development specialists. The physicians were board-certified, except for the resident members of the PRITE editorial board. The two tests had similar content outlines but were not identical. Both tests included items based on clinical scenarios that required examinees to interpret data and select diagnoses or management options. Both tests were given under secure conditions following standardized test administration procedures. For the PRITE, both raw and standard scores based on norm group performance (mean=500, SD=100) were reported. The norm group consisted of all general psychiatric residents (beginning through advanced) plus first-and second-year child and adolescent psychiatric residents. The Part 1 results were reported as percent correct scores.
Given its educational purpose, there were no pass/fail standards for the PRITE. Part 1 candidates had to pass both the psychiatry and neurology components to pass. The ABPN used a criterion-referenced approach to standard setting; hence, there were no preset pass/fail standards or rates for any groups. Rather, the standards were based on the test committee members’ judgments about what scores examinees needed to achieve to be considered minimally competent.
The College and the ABPN agreed to collaborate in this study, and PRITE examinees signed a release statement if they were willing to participate when they took the test. Their scores were forwarded to the ABPN, and the names of these individuals were matched with the names of the examinees who took the 2003 Part 1 examination. As part of the ABPN application process, the applicants signed a statement agreeing to let the ABPN release information about examination results and scores, provided that such data were reported in the aggregate.
Of those individuals who took the 2002 PRITE and signed the release statement, 297 subsequently took the 2003 ABPN Part 1 examination, representing 33% of the advanced residents (n=905) who took the 2002 PRITE and could potentially have taken the Part 1 examination in November 2003 and 27% of the new Part 1 candidates (n=1,110).
For the PRITE, the mean standard scores for the 297 study participants were 580 (SD=68) for global psychiatry and 536 (SD=88) for global neurology. The mean standard scores for the 905 advanced residents were 564 (SD=79) for global psychiatry and 523 (SD=95) for global neurology. For the study participants, the mean raw scores were 184 (SD=14) for global psychiatry and 34 (SD=5) for global neurology. The mean raw scores for the advanced residents were 181 (SD=16) for global psychiatry and 33 (SD=6) for global neurology.
On the Part 1 examination, the mean scores for the study participants were 84% (SD=6) for the psychiatry component and 84% (SD=8) for the neurology component. The mean scores for the total group of 1,110 new examinees were 82% (SD=7) for psychiatry and 83% (SD=9) for neurology. Of the study participants, 91% (n=270) passed the Part 1 examination, compared with 86% (n=952) of all new examinees. Hence, the study participants performed slightly better on both examinations than the total groups of examinees.
The correlation between the two PRITE scores was 0.52 (p<0.01), and the correlation between the two Part 1 scores was 0.74 (p<0.01). The correlation between the PRITE global psychiatry and the Part 1 psychiatry scores was 0.59 (p<0.01), and the correlation between the PRITE global neurology and the Part 1 neurology scores was 0.39 (p<0.01). Performance on the PRITE accounted for 35% of the variance in the Part 1 psychiatry scores and for 15% of the variance in the Part 1 neurology scores. Correlations of 0.40 to 0.60 are commonly characterized as moderate, and those in the 0.60 to 0.80 range as strong (9).
To further explore the relationship between performances on the two examinations, the score distributions were reviewed to identify scores on the PRITE that were associated with low passing rates on the Part 1 examination. For psychiatry, examinees who scored more than half an SD below the mean (<451) on the PRITE global psychiatry had a pass rate of 50% (7/14) for Part 1 psychiatry and 36% (5/14) for Part 1 overall. All other examinees had a pass rate of 97% (275/283) for Part 1 psychiatry and 95% for Part 1 overall.
For neurology, examinees who scored more than one SD below the mean (<401) on the PRITE global neurology had a pass rate of 65% (11/17) for Part 1 neurology and 53% (9/17) for Part 1 overall. All other examinees had a pass rate of 95% (265/280) for Part 1 neurology and 93% (261/280) for Part 1 overall.
Although the PRITE and the ABPN Part 1 examination have different purposes and independent development processes, there were significant correlations between scores on the two tests, and these correlations were of the same magnitude as those found in a similar study performed a decade earlier, as well as those found in other specialties. These outcomes support the use of PRITE results to guide preparation for the Part 1 examination.
Based on this sample, training directors would be well advised to counsel their trainees who score <451 on the PRITE global psychiatry that they are at risk of failing Part 1 psychiatry. If they score <401 on the PRITE global neurology, they are at risk of failing Part 1 neurology.
Limitations of the study include the relatively low rate of participation in the study and that the residents who did participate had scored somewhat higher on both examinations than the total groups. However, the resulting restriction in the range of scores may have resulted in somewhat lower correlation coefficients.
At the time of submission, Drs. Juul, Schneidman, Fernandez, Ebert, Winstead, and Faulkner declared no competing interests. Disclosures of Academic Psychiatry editors are published in each January issue.