Child and adolescent psychiatry began to emerge as a distinct medical subspecialty in the decade following World War II. Many of the pioneers were pediatricians who discovered that emotional and behavioral disorders were a significant but underrecognized source of morbidity for children, and during the late 1950s, sentiment grew for a distinct specialty of child psychiatry. In 1959 the American Board of Pediatrics, Inc. (ABP) supported the formation of a Committee on Certification in Child Psychiatry (CCCP) under the auspices of the American Board of Psychiatry and Neurology, Inc. (ABPN), and the ABP was granted in perpetuity one pediatrician member on the CCCP (
+1).
In 1980, the Graduate Medical Education National Advisory Committee report to Congress identified child and adolescent psychiatry as the medical specialty most in need of enhanced recruitment to meet the nation's health care needs (
+2). At the same time, members of the CCCP observed that fewer pediatricians were entering psychiatry and child and adolescent psychiatry. Informal surveys suggested that medical students who were interested in combining pediatrics and child and adolescent psychiatry were dissuaded by the 7 or 8 years of serial residencies coupled with rising medical student debt. It was a daunting task to create a combined residency agenda that would save time and money but would not result in an unacceptable dilution of quality in the training provided.
A proposal was fashioned for a 5-year curriculum that would include 24 months of pediatrics and 18 months each of general psychiatry and of child and adolescent psychiatry. The directors of the ABP and the ABPN agreed to approve and oversee what came to be referred to as the "Triple Board" if unanimous written endorsement could be gained from the major professional organizations in the three fields, a seemingly Herculean task. Formal approval was obtained from the American Academy of Child and Adolescent Psychiatry (AACAP), the American Academy of Pediatrics (AAP), the American Association of Chairmen of Departments of Psychiatry, the American Association of Directors of Psychiatric Residency Training, the American Psychiatric Association (APA), the Society of Professors of Child and Adolescent Psychiatry, and the Accreditation Council for Graduate Medical Education (ACGME) Residency Review Committees in Pediatrics and in Psychiatry.
The Boards were cautious in their approval. Only six\6 pilot programs were to be chosen nationally, and all of the programs and the residents were to be carefully monitored over a 10-year period. To our knowledge, this is the only residency track that began with the inclusion of a formal, prospective evaluation component.
An oversight committee, the Pediatric—Psychiatry Joint Training Committee (PPJTC), was formed in 1984 and included representatives of the ABP, ABPN, CCCP, AAP, APA, and AACAP. The PPJTC had the authority to make administrative decisions about the project. Because the Boards served both accrediting and certifying roles, all decisions about standards remained with the directors of the ABP and of the ABPN.
Thirty-two sites requested application forms to join the project, and eight submitted proposals that were judged worthy of site visits. These were conducted by a pediatrician plus either a psychiatrist or a child and adolescent psychiatrist member of the PPJTC. In December 1984 the committee met for the first time and selected six sites, Albert Einstein College of Medicine, Brown University Program in Medicine, Mount Sinai School of Medicine, Tufts University School of Medicine, University of Kentucky College of Medicine, and University of Utah School of Medicine, to launch this experiment in graduate medical education. With funding from the National Institute of Mental Health, the Center for Mental Health Services, the ABP, and the ABPN, a team based at the University of North Carolina at Chapel Hill School of Medicine and headed by one of the authors (C.P.F.) was selected to design and administer the evaluation.
In addition to ACGME accreditation of all three residencies, other general requirements included development of a written, cohesive curriculum, enrollment of at least two trainees per year, monthly meetings for all Triple Board trainees with the training director, and annual administration of the pediatric and psychiatry in-training examinations.
For the pediatrics portion of the curriculum, residents were required to have clinical experiences with children of all ages (premature infants through adolescents), with 6 to 8 months spent in outpatient settings and 6 to 8 months in inpatient settings. The residents were to spend 2 to 3 months caring for newborns, at least 1 month of which had to be in non—critical care settings. The adolescence experience had to include 1 month of gynecology. In addition, residents participated in a pediatric continuity clinic every 1 to 2 weeks for all 5 years.
General psychiatry requirements were 4 to 9 months of inpatient experience and 6 to 9 months of outpatient experience. The neurology, community psychiatry, consultation-liaison psychiatry, and emergency department rotations could be done in either a general psychiatry or a child and adolescent psychiatry setting. The child and adolescent psychiatry requirements included 2 to 6 months in inpatient settings, outpatient experiences with some patients that lasted at least 1 year, two consultation rotations, emphasis on a biopsychosocial approach to patient care, and experience in pharmacologic, family, psychodynamic, and behavior therapies.
The Triple Board evaluation had two distinct purposes: 1) during the pilot phase, to inform changes in program policies or operations; and 2) at or near the end of the pilot phase, to inform the decision regarding permanent status for the program. To meet these goals, the evaluation staff selected a cyclical, responsive evaluation model in which longitudinal quantitative and qualitative data were gathered from multiple sources tailored to the decisions at hand (
+3,
+4). The evaluation team collaborated closely with the PPJTC and the program directors but functioned as an autonomous group to ensure the objectivity of the study.
The evaluation was guided by a set of questions developed by the PPJTC and the program directors. The methods and results pertinent to five of these questions are reported here: Did the program recruit adequate numbers of qualified trainees? How did the performance of the Triple Board residents compare with that of their peers in traditional tracks in the three constituent disciplines? Did the program's curriculum, which combined pediatrics and psychiatry training, have an effect on the clinical reasoning of the trainees? Did the Triple Board participants seek and obtain board certification? What types of professional positions did program graduates seek, and, specifically, did they enter positions that made use of their cross-disciplinary training? Several methods were employed to answer these five questions.
Data collection began in July 1986 when the first cohort entered training and continued through June 1995 when the fifth and last cohort of the pilot project completed training. The evaluation operated in annual cycles. At appropriate times during the year, the evaluation staff collected various types of data, which are described in more detail below. Other data sources included annual site visits, residents' retreats, and annual program meetings, each of which will now be briefly described.
Each program site was visited every spring by a member of the evaluation team and a member of the PPJTC. For the first 5 years, the focus was on the program's readiness to offer the next year's curriculum. The site visit agenda included interviews of the program director, the chairs of pediatrics and psychiatry, the current trainees at that site, key faculty members, and other individuals as needed to explore specific issues identified in previous site visits. All interviews were semistructured; that is, they were guided by a list of topics, but issues raised by the interviewees were explored. At the close of the site visit, a summary meeting was held with the program director. A draft of the site visit report was shared with the program director and modified as necessary before official presentation at the annual meeting.
Yearly resident retreats were another source of program evaluation data. These were initiated after concern was expressed during the first annual meeting that residents felt isolated because participation in an educational experiment meant they had few peers. During subsequent years, each of the six programs hosted the day-long retreat at least once. One of the authors (J.E.S.) presided over each meeting, and a member of the program evaluation staff was also present. Second- and fourth-year residents' expenses were paid, although all residents who could attend were welcome, and program directors were invited to attend at their own expense.
Regular attendees were PPJTC members, program directors who all were also child and adolescent psychiatry training directors, members of the evaluation staff, and, in alternate years, either the six programs' psychiatry or pediatric program directors. Components of the agenda included program changes as described by program directors, an update of the year's evaluation findings, and any new instruments proposed by the evaluation staff. Discussions at the meeting generated changes in the programs and, in some cases, changes in the evaluation methods themselves.
Each year, data were compiled on the number of applicants to each site and their medical schools, the number of unique applicants to the program (given that one applicant might have applied to several sites), and the numbers interviewed and matched. In 1986 and 1987, each site filled its two positions by independent negotiation with its own applicants. From 1988 through 1993, slots were filled via a program-specific mini-match conducted prior to the national match. In 1994, the program joined the National Resident Matching Program (NRMP).
+
Performance in Residency Training
In addition to annual administration of the in-training examinations in pediatrics and psychiatry, the performance of each Triple Board resident was evaluated against that of his or her peers in the traditional training tracks. This aspect of the evaluation used the method of pair comparisons (
+5) and was conducted annually in four steps.
First, the evaluation staff asked each site to list a peer group corresponding to each training level of the Triple Board program at that site. A peer group was composed of the Triple Board residents and all other residents who had training experiences similar to that of the Triple Board residents at that level. For example, the first year of the Triple Board program emphasized pediatrics at all sites; hence, the peer group consisted of all program participants and the first-year pediatric house officers.
Second, a rating form specific to each site and to each Triple Board training level at that site was prepared. A subset of peers was selected randomly for a total of 10 residents per site, including the Triple Board residents. The rating form listed all possible pairs (n=45) of these residents.
Third, for each training level and site, two raters, a faculty member and the chief resident whose appraisals were most relevant to that training level, independently completed the evaluation form. For each pair, the raters indicated which resident performed more strongly that year.
Fourth, the number of times each resident was judged to have performed more strongly than his or her peer were tallied, and the residents were rank-ordered.
+
Assessment of Clinical Reasoning
From 1992 through 1994, the graduating Triple Board trainees and child and adolescent psychiatry residents at the Triple Board sites completed a specially designed examination. The exercise sought to detect effects on clinical reasoning that might be attributable to the interdisciplinary training the Triple Board residents received and specifically to detect if and how formal pediatric training affected a child and adolescent psychiatrist's approach to a complex case. A pilot version of the examination was developed in 1990—1991 by a subcommittee of the PPJTC. The final version of the examination consisted of four cases, each combining pediatric and psychiatric issues. After reading the description of each case, the residents responded to a series of short-answer questions exploring diagnosis, therapy, and general management. The examinations were scored blindly by four committee members.
+
Follow-up of Program Graduates
The evaluation team either conducted telephone interviews with each Triple Board graduate or queried the program sites to document their graduates' current professional activities, satisfaction with the program in retrospect, perceptions of the capabilities that joint training helped them develop, and plans for the future.
Over the course of the evaluation (1986—1995), 109 trainees were at some point enrolled in the program. As of June 30, 1995, 49 had completed training, 20 had left the program, and 40 were continuing in training. Fourteen new trainees were about to begin the program. Cohorts did not remain intact owing to dropouts, transfers in from other programs, and transfers between sites. For some of the evaluation studies, all residents in the program comprised the sample of interest. For other studies, the pertinent sample was the 5 cohorts of the pilot project. The 60 available training slots in the first 5 cohorts produced 49 graduates, of whom 47 completed all 5 years of training at one of the Triple Board sites.
The 20 residents who dropped out of the program opted for traditional pediatrics residency training, and the interest of these residents in child and adolescent psychiatry was considered a boon for both specialties. Because of the innovative nature of the training, this number of transfers, while disappointing, was perhaps not surprising.
+
Quantity and Quality of Applicants
Twelve entry-level positions were available at the six pilot sites through 1994. In 1995 the number of entry-level positions at the six original sites increased to 14 as Brown and Einstein were granted permission to recruit for three entry-level positions. In both 1994 and 1995, 19 applicants ranked a Triple Board program as their first choice in their NRMP. On average over the 10 years studied, each applicant applied to 2.3 sites, with 54% of applicants applying to 2 or fewer sites. In all years, sites were able to fill their PGY-1 slots outside the match when necessary.
The results indicated that the programs attracted a number of applicants sufficient to fill the available positions, a finding that remained unchanged over 10 years. The relatively large number of applicants who applied to 2 or fewer sites suggested that applicants developed interest in specific sites or regions and did not view the program as homogeneous across the 6 pilot sites.
The annual site visits explored program directors' and key faculty members' satisfaction with the quality of the applicants and the new trainees recruited. Opinions about the quality of the applicants fluctuated from year to year and across sites, with no discernible pattern. However, program directors and faculty were consistently satisfied with the quality of the applicants and the trainees recruited, and the Triple Board program was considered viable from a recruiting standpoint as long as 2 qualified trainees were recruited each year.
+
Performance in Residency Training
The rankings of the Triple Board trainees in relation to their peers at each level of training are shown in
+Table 1. The results are aggregated across all sites and years of program operation. Only the results from the discipline that dominated each year of training across all program sites appear, although at some sites the Triple Board residents studied multiple disciplines within a year of training. The large standard deviations of all these rankings indicate that the Triple Board residents were distributed across the performance spectrum.
The 85 trainees studied during their first year of training received a mean rank of 5.9; that is, the average Triple Board resident during the first year of training was ranked approximately 6th out of 10 colleagues who, during their first postgraduate year, were first-year pediatric house officers at the same institution. Similar results were obtained for the second year of training, suggesting that the Triple Board residents performed slightly below the median when compared with traditional pediatrics residents during the first and second years of training. When compared with peers in general psychiatry (PGY-3 and PGY-4), the average Triple Board resident was ranked about 5th out of 10 colleagues or at the median level. At the PGY-5 level, they were ranked about 4th out of 10 colleagues, or slightly above average.
At the site visits, residents and faculty members reported that Triple Board residents were indistinguishable from their colleagues during the pediatrics phase of their training, corroborating the results of the peer comparison study. Compared with peers during general psychiatry and child and adolescent psychiatry training, Triple Board residents were usually considered superior and were often chosen to be chief residents. It can be hypothesized that this was because individuals who chose to participate in a training program that can be characterized as stressful, diverse, and concentrated tended to be very motivated and capable.
The Triple Board residents' performance on the pediatrics and psychiatry in-training examinations is shown in
+Table 2. Although trainees were asked to take both in-training examinations on an annual basis, many did not take the psychiatry examination during PGY-1 and PGY-2.
Scores on the two examinations are reported differently and cannot be directly compared. For the pediatrics examination, the test items were drawn from the certifying examination pool, and scores were scaled based on the reference group (recent graduates who were first-time takers and American medical school graduates) that took them for board certification. The standard for certification was 410. For the psychiatry examination, scores were computed based on a national norm group that consisted of all general psychiatry residents and first- and second-year child psychiatry residents taking the examination. Individuals in Canadian residencies and those taking the examination under nonstandard testing conditions were excluded. The mean score for the norm group was 500 with a standard deviation of 100.
In pediatrics, the in-training examination performance of the Triple Board residents increased substantially from the first to the second years and then remained level for the remainder of the 5-year program. Mean pediatrics examination scores were below average for the PGY-1 to PGY-3 training levels, where a comparison with traditional pediatrics residents was meaningful, and were indicative of marginal performance on the pediatrics board examination.
In psychiatry, examination performance increased gradually over time, with the largest increase occurring at the PGY-4 level. When PGY-3 to PGY-5 Triple Board trainees were compared with PGY-2 to PGY-4 trainees in traditional psychiatry residencies, the Triple Board residents scored close to the national average on the in-training examination for all three years. These scores were indicative of successful performance on the psychiatry board examination.
+
Assessment of Clinical Reasoning
From 1992 through 1994, the examination was administered to 25 fifth-year Triple Board trainees and 41 graduating child and adolescent psychiatry residents. The results appear in
+Table 3. The mean score on the examination across all groups was 11.6 out of 16 possible points, or 73%. The reliability was 0.69 (Cronbach's alpha). Across all three administrations, the Triple Board residents' mean score was 12.4 (SD=2.0) and that of the child and adolescent psychiatry residents was 11.1 (SD=2.5).
Two-way analysis of variance indicated that the mean score of the Triple Board residents was significantly higher (P<0.05) than that of the child and adolescent psychiatry residents. There were no significant differences in mean scores by year of examination administration, nor was there a significant year-by-group interaction. This suggested that the Triple Board program had a positive effect on the clinical reasoning ability of the trainees, a result that assumed greater significance in light of their abbreviated training compared with that of the traditionally trained residents.
This result is supported by qualitative evidence from the site visit interviews, which strongly suggested that the Triple Board residents thought "differently" than traditionally trained child and adolescent psychiatrists. This observation was made at all sites, in all years, and by the faculty as well as the trainees themselves.
The progress of the Triple Board graduates in achieving board certification, based on information available to August 2001, can be summarized as follows: 18 of the 49 residents (37%) are triple-boarded in pediatrics, psychiatry, and child and adolescent psychiatry; 20 (41%) are double-boarded in psychiatry and child and adolescent psychiatry; 4 (8%) are double-boarded in psychiatry and pediatrics; 5 (10%) are boarded in psychiatry only; and 2 graduates (4%) are not certified in any of these three areas. These results suggest that while a significant portion of the group has indeed become triple-boarded, more than half have not done so.
At the conclusion of the evaluation in 1995, 22 of the 49 graduates held faculty positions, 5 were in fellowship training, 11 were in private practice, 10 held primarily clinical positions in hospitals or community agencies, and 1 was not currently employed by choice. All graduates reported that their training had been positively viewed in the employment market, and they felt that their special training gave them multiple career options.
Twelve graduates remained in some capacity at the Triple Board site where they trained, and 1 was on the faculty of a different Triple Board site. Of the graduates with faculty positions, 20 had primary appointments in child and adolescent psychiatry with a formal affiliation with pediatrics, and 2 were primarily appointed in pediatrics with a secondary appointment in child and adolescent psychiatry. Of those in private practice, 1 identified himself as primarily practicing pediatrics, the other 10 as practicing child and adolescent psychiatry.
When asked about career goals in interviews, all graduates placed high priority on direct clinical service, although several expressed the importance of working in academic settings where they could teach what they had learned. Those in academic positions saw themselves more as teachers and clinical role models than as researchers. Graduates expressed specific interest in medical problems that would benefit from attention by someone trained in pediatrics and psychiatry, such as diabetes, eating disorders, posttraumatic stress disorder, and adolescent drug addiction.
Even though it was not evident from formal position titles, the graduates' descriptions of their positions strongly indicated that they were using all components of their training. Many graduates mentioned that the phenomenon of being referred cases with both medical and psychiatric components began while they were in training and continued after graduation. Those practicing child and adolescent psychiatry felt that their pediatric training, at a minimum, facilitated communication with pediatricians.
In the 1980s it had become clear that psychiatry should become more closely linked to the rest of medicine. A response to this realization—in an attempt to augment child and adolescent psychiatry practice, teaching, and research—was the pediatric—psychiatry—child and adolescent psychiatry combined residency program. To propose shortening residency time when the reverse was common required arduous planning, diverse and multiple approvals, and an unprecedented decision to sunset the program after 10 years unless prospective evaluation data could convince the ABPN and the ABP to make it permanent.
During the pilot years, the number of programs and of trainees was limited purposefully, so the evaluated cohort was rather small. However, by all evaluation standards the program has been at least modestly successful. The integration of training between specialties seems to have helped compensate for the shortened training time. When compared with peers via pair comparisons and performance on in-service examinations, the graduates did reasonably well in pediatrics and were superior in psychiatry and in child and adolescent psychiatry. Perhaps most important was that all graduates and faculty believed that the combined residency was worthwhile and successful, and follow-up data indicated that graduates were in positions that drew on their interdisciplinary expertise.
As mentioned previously, this article describes a unique example of a multisite program evaluation in which data from multiple sources were used to make both formative and summative decisions about residency training. On the basis of the results, the ABPN and ABP decided to make the combined track permanent 2 years before the deadline, and both the number of training sites and the number of trainees per year have since been expanded. There are 9 programs with a total of 85 positions listed in the 2002—2003
Graduate Medical Education Directory (
+6).
The ABPN recently decided to facilitate a meeting of current Triple Board programs. At that time, additional data will be gathered on the graduates of the program, as well as on current functioning of the programs. For example, two topics suggested by the evaluation reported here are dropout rates and training and performance in pediatrics.
Support for this project was received from the U.S. Department of Health and Human Services, both the National Institute of Mental Health and the Center for Mental Health Services; the American Board of Pediatrics; and the American Board of Psychiatry and Neurology.