An article published in November 2007 by the Mental Health Services (MHS) Committee of the Group for the Advancement of Psychiatry (GAP) used APA data to demonstrate that early and mid-career psychiatrists now work more hours in organizational settings than in private practice (1). The article concluded that, as a result, psychiatric residency programs need to provide more comprehensive training in systems-based practices (SBP), one of the six core competencies required by the Accreditation Council of General Medical Education (ACGME). The ACGME describes SBP as an “awareness of and responsiveness to the larger context and system of healthcare, as well as the ability to call effectively on other resources in the system to provide optimal healthcare” (2, 3).
As a follow-up to that article, the MHS Committee considered how to improve residency training in SBP. In the process of considering how to approach that goal, we received feedback suggesting that there is a general lack of understanding among residents and teaching faculty of what SBP entails. This was not surprising, in light of a 2008 ACGME report (4) noting that SBP concepts “do not readily lend themselves to educational formats traditionally used in residency. Many faculty physicians currently teaching have little or no formal training in these concepts.” As highlighted by Graham et al. (5), faculty across medical disciplines frequently struggle with how to define and measure competency in SBP (6, 7).
This literature review led us to conclude that the first step toward improving residency training in SBP should be to conduct an exploratory study aimed at developing a conceptual model for SBP in psychiatry. Field testing an instrument consisting of 60 observable behaviors provided us with the data necessary to develop that model. The research questions were 1) which observable behaviors best exemplify SBP in psychiatry; and 2) what grouping of those behaviors best creates a conceptual model of SBP in psychiatry.
The study proceeded through six stages, the first three of which involved instrument development. In Stage 1, we began with an SBP taxonomy previously developed and validated by Graham et al. based upon expert multidisciplinary consensus (8). Within this earlier study, each of the ACGME’s six core expectations for SBP (2) were translated into 35 items by use of a qualitative, stepwise, 360-degree process. Each item was a measurable behavior.
For our study, we considered these 35 items to be a core of SBP behaviors that, once modified, could be applied to psychiatry. Through a consensus process, items were reviewed and edited by small work-groups for final approval by the entire committee. For example, we translated the ACGME expectation that residents “coordinate patient care within the health-care system relevant to their clinical specialty” into an item: “In general clinical settings, I am encouraged to seek input about patient care from all members of the treatment team.”
In addition to the ACGME’s six core expectations for SBP, the Psychiatry Residency Review Committee (RRC) added eight expectations for psychiatry (9). All 14 expectations are listed in Table 1 with the specific ACGME/RRC wording. To address the additional RRC expectations, in Stage 2 the committee derived an additional 25 items through the same consensus process described above, for a total of 60 items specific to psychiatry. A list of all 60 items is available upon request, each linked to the ACGME/RRC expectation from which it was derived. This process was facilitated by the fact that most of the expectations added by the Psychiatry RRC were adaptations of the core ACGME requirements to psychiatry. For example, as shown in Table 1, the RRC expectation that starts with “advocate for quality patient care” is an elaboration of the ACGME expectation that starts with the same words.
In Stage 3, content or face validity of the items was sought by submitting items for review by alumni of the Columbia University Public Psychiatry Fellowship, psychiatrists with specific training in community and public psychiatry. These doctors have expertise in the delivery of mental health services in community-based and organizational settings, where proficiency in SBP is paramount. Feedback received from this group led to refining the content and clarity of items.
The resulting SBP instrument asked the resident to rate the extent to which s/he was encouraged to perform each of the 60 behaviors during his/her most recent clinical rotation in the following contexts: general clinical settings, developing clinical treatment plans, team meetings, and written and oral communication with supervisors. Responses were rated on a 6-point Likert-type scale assessing the extent to which the resident was encouraged to perform each item.
In Stage 4, the SBP instrument was field-tested at 12 residency training programs in psychiatry, with IRB approval at each site. Potential sites were identified through contacts available to the authors. The participating sites included the following 12 programs: Columbia/NYS Psychiatric Institute, St. Luke’s Roosevelt, Weill Cornell, Mount Sinai, NYU, Bronx Psychiatric Center of Albert Einstein College of Medicine, NY Presbyterian and NYU child psychiatry programs (all in New York City), Einstein (Philadelphia), University of Texas Southwestern (Dallas), Western Pennsylvania Institute and Clinic (Pittsburgh) and University of California in Los Angeles. Since the goal was to study the relationships among the items rather than establish cause and effect, there is no indication that this convenience sampling significantly biased the results. We estimated that 200 responses would provide enough statistical power for subsequent analysis (10).
Although SBP is obviously relevant in organizational settings, it is also relevant in private practice because patients live and operate in larger systems. Accordingly, we included residents rotating through inpatient units (typical of organizational settings) and outpatient clinics (more typical of private practice).
+
Instrument Administration
Each site Principal Investigator (PI) met with residents to explain the voluntary nature of the study. Depending on the site, consent was obtained verbally or in writing. The multisite PI contacted all residents, providing a link to a secure web-based survey.
Study Question 1 (which observable behaviors best exemplify SBP in psychiatry), was addressed in Stage 5. A hierarchical cluster analysis (HCA) was used for examining the internal properties of the instrument. HCA separates items that are correlated with each another into distinct, internally homologous groups. Clusters with only two items or less were excluded from further analysis. Next, we examined item means, variances, and correlations as a measure of the interrelatedness among these items. Items with a preponderance of inter-item correlations <0.1 or >0.9 were targeted for elimination. In the former case, those items may not have been measures of SBP; in the latter case, those items may have been redundant. A principal-components analysis, a type of exploratory factor analysis (EFA), with an Oblim rotation, was conducted on the remaining items to assess their factor structure, that is, to determine whether the items measure a single construct or a set of interrelated subconstructs.
Confirmatory factor analysis (CFA) was used to test the overall fit of the model generated by the EFA. CFA is usually used as a hypothesis-testing technique on a separate data-set from that used in EFA. It tests the hypothesis that the factor model borne out by the EFA is a good fit on a new data-set. In accordance with the exploratory nature of this study, the CFA was used in a purely exploratory way to assess the overall fit of the four-factor model.
Study Question 2 (what grouping of those behaviors best creates a conceptual model of SBP in psychiatry) was addressed in Stage 6, in which the remaining factors were reviewed for behavioral themes, resulting in the Four-Factor Model of SBP in Psychiatry.
A total of 457 residents were contacted at the 12 sites, and 237 (52%) completed surveys. Among respondents, 63% were women. Respondents reported their age in 5-year intervals as follows: 25–29 (34%), 30–34 (48%), 35–39 (15%), and over-40 (3%); 64% were Caucasian, 13% Asian, 7% South-Asian, 5% each African-American and Hispanic, and 3% each reported Multiracial and Other. The breakdown of respondents according to year of training was: PGY1 (12%), PGY2 (31%), PGY3 (21%), PGY4 (21%), PGY5 (12%), PGY6 (4%). The survey respondents’ most recent rotation was reported as: Inpatient (40%), Outpatient: adult (30%), Outpatient: child (10%), Emergency Room (8%), and Other (12%).
Data analysis initially identified six clusters, one of which was dropped because it only comprised two items. Several items with low correlations were also dropped. As a result, during the next step of the analysis, a principal-components analysis (PCA), the fifth cluster was dropped. The PCA indicated that SBP could best be represented by 17 items, in four factors, defined here as Team Member, Information Integrator, Resource Manager, and Patient Care Advocate, each containing 4–6 items that accounted for 70.45% of the variance. Table 2 provides a complete listing of all 17 items, grouped by factors. Internal consistency for each scale was established, using coefficient α as follows: Team Member: 0.906; Information Integrator: 0.857; Resource Manager: 0.861; Patient Care Advocate: 0.866.
The CFA model that tested the four-factor structure indicated overall adequacy of fit (comparative fit index [CFI]: 0.934; Tucker Lewis Index [TLI]: 0.923; root mean-square error of approximation [RMSEA]: 0.078; standardized root mean square residual [SRMR]: 0.053). Therefore, the final model explained 92% of the covariance of the data. The details of the CFA are available from the authors.
Factor analysis revealed that SBP in psychiatry can be defined along four dimensions, conceptualized as a set of roles performed by psychiatrists to meet the comprehensive needs of the patient within and beyond the healthcare system. The four identified roles provide a definition of SBP that is measurable and observable and can be used to inform residency training in SBP.
Earlier work, carried out by Graham et al. (8), identified one role for each of the six core ACGME requirements for SBP, based upon expert consensus. Our work goes a step further and utilizes validated empirical methods to further delineate underlying constructs that define SBP. Graham predicted that the SBP domains might vary among specialties. Accordingly, the roles presented here differ somewhat from the roles established by Graham, as they capture the specialty-specific aspects of SBP as it applies to psychiatry.
As we mentioned in the introduction, initial feedback indicated that many psychiatrists are unfamiliar with the meaning of SBP. Confusion about the definition of SBP likely stems from its complexity: SBP has been difficult to define and teach because it is not one simple idea. The four-factor model is based on empirical evidence and provides a conceptual framework for SBP in psychiatry along four related-but-separate dimensions. The following case example illustrates how the psychiatrist can function flexibly in each of the four roles in the course of treating a given patient.
Mr. Smith, a recently-retired 67-year-old man with difficulty breathing, lives in a fifth-floor walk-up that is filled with a lifetime of accumulated objects. He lives an isolated life; he has no psychiatric history and has not seen a physician in years. When a family member tried to convince him to clean up his apartment, he became agitated. Although he can no longer comfortably climb the stairs, he is unwilling to consider moving because it would involve having to give up these objects and memories. His family prevailed upon him to see an internist, who referred him to a psychiatrist in order to better address these issues.
In this role, the psychiatrist seeks, describes, and synthesizes information from all members of the treatment team and participates in the development of an ongoing treatment plan based on that information. In the case of Mr. Smith, the psychiatrist needs to be proactive in communicating with family members, in addition to the primary-care physician, social worker, and rehabilitation and physical therapists to work with the patient to create a comprehensive treatment plan.
In this role, the psychiatrist incorporates information from multiple dimensions of the patient’s life (including housing, social, and vocational functioning) and designs interventions accordingly. Whereas the role of Team Member involves knowing from whom to collect the information, the role of Information Integrator involves knowing what information to collect and how to use it. In the case of Mr. Smith, the viability of his remaining in his current apartment could be increased by diagnosis and treatment of his respiratory difficulties, supporting his acceptance of cleaning his apartment to eliminate environmental pathogens, and helping him to develop a social life.
In this role, the psychiatrist recommends interventions that are both clinically indicated and fiscally sound. Every aspect of psychiatric practice is affected by cost considerations such as Medicare/Medicaid, SSI, medication prices, expenses for inpatient /emergency room care, and constraints of government service contracts. Mr. Smith must sort through the multiple financial and benefit issues inherent in retirement; he is facing a reduction in income, choices about Social Security benefits, and changing from his previous, employment-based private insurance plan to Medicare.
In this role, the psychiatrist addresses the individual’s treatment goals through a person-centered partnership. To this end, the psychiatrist works with and advocates for patients as they navigate issues of disability, stigma, poverty, and fragmented and inadequately funded service-delivery systems. This approach has come to be called “recovery,” defined as “the process by which people are able to live, work, learn, and participate fully in their communities” (11). Mr. Smith requires an advocate to support him in addressing specific psychiatric, physical health, and housing-related challenges. This involves understanding his strengths and needs, aligning with his personal goals and preferences, and supporting his navigation through complicated systems. This may involve optimizing pulmonary treatment and psychiatric interventions, family-based psychosocial approaches, and connections with community-based organizations that offer social-networking or housing assistance.
Fulfilling these four roles allows the psychiatrist to assist in all aspects of a person’s life, including a social component, alongside biological and psychological interventions. In delineating SBP as a core competency, ACGME clearly expects psychiatrists to know how to carry out these activities.
This four-factor model of SBP can be applied in a variety of ways. A first possible application is as a template for implementing SBP training for psychiatric residents based upon the four SBP roles identified through this study. This training could occur in four contexts: didactics, clinical rotations, supervision, and case conferences (12).
A second possible application is to inform a change in the ACGME/RRC’s definition of SBP in psychiatry, providing formalized guidance for psychiatrists in mastering SBP competency. This is consistent with the ACGME’s intent to provide definitions for all of the six core competencies that are measurable and observable.
SBP is not only applicable to publicly-funded organizational settings, but to private practice, as well. Consequently, a third application considers the implications of this four-factor model of SBP in private-practice settings. Mr. Smith may have encountered a psychiatrist in private practice, rather than in an organizational setting. The solo practitioner would decide whether to refer the patient to an organizational setting or to collaborate with other providers to create the kind of treatment plan envisioned above.
A limitation of this study is that it was carried out with a convenience sample of residency programs that included eight programs from in New York State and four programs in other states. Most, but not all, of the programs were large. Accordingly, this sample may not be representative of all programs throughout the United States.
A second limitation is that it is not clear how easy it will be for residency training directors to incorporate additional SBP training into overburdened training programs.
An additional limitation of the paper concerns the CFA. A second round of data collection would be necessary to find “test” data on which to test the CFA. This approach would confirm that the proposed factor structure could be replicated on a new data-set, thereby improving the validity of the model.
This article presents a conceptual model of SBP conceived of as four roles. We have found that these four SBP roles are easily understood by psychiatric educators, psychiatric residents, and nonacademic practicing psychiatrists. The four-factor model provides a comprehensive framework of skills needed in clinical practice. With further field-testing, the 17-item SBP Instrument could serve as an assessment tool for psychiatric residency programs to measure and document efforts to meet the ACGME’s SBP core curriculum requirement. Whether seen in organizational or private-practice settings, all patients live and operate in larger systems, highlighting the importance of training psychiatry residents to exemplify behaviors of SBP.
The authors wish to acknowledge John Encandela and Boyd Richards (Columbia University), as well as Osman Ali, Elizabeth Auchincloss, Kimberly Best, Eve Caligor, Alejandra Durango, Elisabeth Guthrie, and Bipin Subedi (site PIs) for their participation in study design and implementation.