Translating the various mandates from the myriad of governing educational institutions into functional and productive aspects of a residency program is one of the major challenges of being a training director. Often, this task is further complicated by the fact that those most directly affected, the faculty and residents, have not been voicing their concerns about the issues being addressed by the new requirements. Thus, their initial reactions tend to range from bewilderment about the need for new perspectives, procedures, or documentation to fear and hostility that the requirements will generate even more paperwork and be meaningless in practice.
This article describes the process of developing and implementing one of the more recent educational mandates, the core competencies of the Accreditation Council for Graduate Medical Education (ACGME), into a child and adolescent psychiatry residency program. Despite an initial lack of enthusiasm and grudging participation demonstrated by the program’s faculty and residents, over several years the core competencies have become an integral component of the program and have been more functional than anticipated. Currently, most of the faculty and residents agree that the core competencies provide a useful framework for conceptualizing and monitoring the program and the individuals involved.
The first steps were identifying the individuals who should be involved at each stage and composing a general sequence for the process. In this program, the training committee (division chief, training director, key program faculty, and chief resident) is the monitoring entity that makes decisions about the program and residents. The training committee’s preliminary discussion about the core competencies focused on whether the existing program decision structure should continue to be utilized or whether it would be helpful to develop new components (e.g., a core competency workgroup) to accomplish this task. For several reasons, we decided to use the existing system. Though the program is a medium-sized one, the number of full-time child and adolescent psychiatrists is small. They tend to be involved in all of the major initiatives (most are on the training committee), so it seemed redundant to appoint them to another committee. The program has several voluntary faculty members who are strongly committed to residency education and spend a significant amount of time teaching on a weekly basis. No one wanted to overwhelm these individuals or detract from their teaching commitments. There were other voluntary faculty with fewer obligations but many of them had a limited understanding of the overall program. The procedure that has worked the best for this program, after years of trying various approaches, is described below.
In terms of the core competencies, using the existing system meant that everyone was familiar with it and could focus on the assignment without being distracted by trying to figure out how to make things work. However, informative discussions were held with key faculty and the residents early in the process to educate them on the issues and plans. Other areas of preliminary discussion included the goals for the process, desired results, decisions necessary to guide the procedure, relevant issues, and types of documentation.
Identified aims included identifying the essentials of child and adolescent psychiatry; defining competency; defining the essentials in terms of the competencies; describing the competencies in measurable ways; designing and implementing an effective system of evaluation; maintaining consistency with national (e.g., the ACGME and American Academy of Child and Adolescent Psychiatry [AACAP]) and local standards (e.g., department, medical school); incorporating the competencies into the current program structure and systems; and producing a model that defines and executes competency in a manner which enhances residency education and is positively (or at least neutrally) regarded by the faculty and residents. Obviously, a primary goal of this activity was to remain in compliance with the organizations’ accrediting programs, such as the Residency Review Committee (RRC) of the ACGME (
1,
2), and the ones that certify individuals, such as the American Board of Psychiatry and Neurology (ABPN) (
3). Currently, the RRC requirements for child and adolescent psychiatry mandate the incorporation of the core competencies with at least one written competency for each competency area. The next set of program requirements, effective in 2007, will require an integration of the core competencies into the education plan with identified methods of assessment that produce an accurate assessment of a resident’s competence in the six areas (
4). Additionally, the ACGME is moving toward identifying patient care and medical knowledge competencies as specialty-specific, with the others being defined across specialties in the ACGME common program/institutional requirements. This trend probably means that programs will concentrate on patient care and medical knowledge (clinical science), while there will be more development of the other competencies across specialties at the medical school and hospital level.
After reviewing the program goals and evaluation systems within the context of the competencies, we decided that the current program content captured the essentials of child and adolescent psychiatry and addressed the competencies in terms of both program expectations and resident performance but did not use consistent language and organization. To enhance acceptance of the competencies and to minimize the creation of new systems and paperwork requirements, the training committee determined that modification of the existing documentation would be adequate with the plan that there would be coordinated competency descriptions, goals and objectives, and evaluations.
Rather than continue with the program’s original terminology, we decided to use the terminology of the ACGME (
1), in terms of the six competencies (clinical science, interpersonal skills and communication, patient care, practice-based learning and improvement, professionalism and ethical behavior, and systems of care), with the exception of medical knowledge, which was replaced by clinical science by the psychiatry RRC (
2), and subcategories (knowledge, skills and practices, and attitudes). Moving to one common vocabulary would facilitate more consistency, allow comparisons between residents and across various rotations, and would give an overall view of the entire program.
At the time the program started with the core competencies, the primary source of information was the ACGME (
1). Though more resources and information now exist for psychiatry residencies (
5,
6), the development of the competencies is still considered a local affair with very general national guidelines and suggestions. The ACGME competencies provide a broad framework for a national mandate for requisite attributes and capabilities across all specialties in medicine. The category descriptions are all-encompassing, with no specific guidelines for implementation. There is the expectation that decisions about criteria, standards, and process are made locally, with guidance and review decisions made nationally by each field’s relevant specialty organizations and certification boards (e.g., the ACGME, AACAP, and the American Academy of Directors of Psychiatric Residency Training [AADPRT]).
Having limited data available from other resources meant that the program had the advantage of developing and implementing its own approach and language. The information from the ACGME Web site (www.acgme.org) provided a useful base from which to work. It did, of course, mean that the competencies and supporting documents essentially had to be created, with the risk that different national standards and criteria, to which the program would be required to conform, would be developed later.
Another issue is that programs were expected to translate competencies into established, often entrenched, systems with defined content and organization, as well as into standards based on existing program criteria and performance. Many child and adolescent psychiatry residencies, including Emory’s, have existed for several decades with ongoing relationships and connections with various hospitals and other mental health systems. How the program worked and the expectations of the various participants had been established for some time and tended to remain stable, leading to a general resistance to change. Several decisions addressed this issue in our program. By utilizing the existing process and personnel to develop and implement the competencies, the individuals responsible for key aspects of the training systems related to the residents and the training program were very involved in decisions about the competencies. The input of these individuals primarily involved reviewing and editing rather than having to generate descriptions and other documents, so they tended to be less resistant to completing various expected tasks. It was made clear that the competencies would be defined based on existing standards and evaluation methods so that the burden of revising and retraining would be minimal.
Finally, the competencies are primarily oriented towards individual performance, enhancing the RRC requirements that regulate program functioning. The training committee decided that this situation provided an opportunity to maintain a structure of program evaluation that generally functioned well and the chance to enhance the system used to assess individual residents. Evaluation of residents doing well tended to be adequately documented, with faculty completing the "strengths" section of the evaluation form routinely when they were pleased about a resident’s performance. When the faculty had concerns about a resident, often the written summaries of performance did not match the verbal ones, with variable communication to the involved resident. Rarely were there written suggestions or plans to correct the identified problems. The hope was that having clearer, defined standards and criteria (e.g., defined evaluation points, a new practitioner level) tied to a model of describing the minimal expected performance would encourage the faculty to better document problems. Additionally, supervisors now were required to sign off on an evaluation form, confirming that they considered the resident competent, which made a stronger statement than our previous evaluation forms. The competencies also were viewed as a structure that could be employed to promote resident involvement and responsibility for their own education (
7).
After reviewing various possibilities, the committee decided that the program structure and documentation would use a basic definition of competency, based on Miller’s model (
8). He described a framework for the assessment of clinical work which consisted of a series of levels in which individuals progress from having the necessary knowledge to being able to practice independently. Competency was defined as having sufficient knowledge and the ability to utilize it effectively to perform designated tasks in a practice-like setting (
8). The amount of knowledge and skill required to be considered competent varied depending on the resident’s level of training and experience. Residents had to be at the level of a new practitioner for a designated activity in order to successfully complete a particular experience. For example, at the end of all of the inpatient rotations (multiple rotations during the first year), residents were expected to be at the new practitioner level for all knowledge and skills that were essential and unique for the practice of inpatient child and adolescent psychiatry. Being able to appropriately conduct inpatient assessments and discharge planning had to be mastered by the end of the first year, since the program’s second year consists of all outpatient rotations. This standard was defined to ensure that residents had acquired the skills needed to begin independent practice, such as having particular professional values and skills of understanding, which could be practiced in realistic settings. Residents did not have to meet this level to finish a rotation or course successfully, but the areas in which they did not meet these criteria had to be addressed during rotations with similar experiences and had to be met by the end of training.
+
Development/Implementation
The training committee reviewed the core competencies with a discussion of the history, rationale, and institutions involved, and then decided on a preliminary approach. There was a decision to have general, broader categories rather than multiple specific ones. For example, patient care encompassed all of the relevant knowledge and skills instead of specific competencies for psychopharmacology and the various types of psychotherapy. The training director then wrote a series of drafts which were subsequently evaluated and modified by the training committee. Once there was a coherent version, there was a series of meetings with the faculty (retreat, faculty meetings, and evening forums) and residents (recurring training-related meetings and individually scheduled ones) to obtain additional input which then was used to produce a final document that was distributed for final feedback. This process was utilized for all of the decisions and documentation related to the core competencies (e.g., definitions, evaluation forms).
The training committee developed descriptions for each of the competencies (Appendix 1) for the program as a whole. Each competency was described in general terms with an expected outcome and categories of knowledge, attitudes, skills, assessment/measurement, and deficiency remediation. Each category included expectations, examples of how to meet those expectations, and possible assessment methods. This general description was written to encompass the expected achievements for the resident upon completion of the program. For example, the clinical science core competency description included the medical knowledge considered essential for child and adolescent psychiatry that the resident should obtain through a variety of experiences.
The goals and objectives for each rotation and course were then reorganized and rewritten to describe and implement the competencies (Appendix 2). Each included the expected outcome and the six core competencies with sections detailing the anticipated knowledge, attitudes, and skills, with definitions, expectations, as well as methods of assessment and measurement. Each rotation and course included only the knowledge, attitudes, and skills considered relevant for that clinical or didactic experience.
The training committee, with input from the faculty, decided to revise existing mechanisms of evaluation rather than immediately implement new ones to facilitate the program’s transition to and comfort with the competency approach. The forms were revised to reflect the competencies (Appendix 3). Our program had evaluation forms that were specific to each rotation and major course, a result of faculty interest in tailoring the form to match the experience. However, this approach may not be obtainable or desirable for some programs. Another approach would be to have a master evaluation form that could be modified at the faculty’s discretion (Appendix 4). Faculty also were asked to document whether the resident met the level of a new practitioner for each competency related to that experience and whether the resident had completed various required tasks (checklist) considered essential or unique to the rotation or activity. To improve consistency among the faculty, we developed definitions of the form’s rating points (Appendix 5), as well as descriptions of the new practitioner standard (Appendix 6). Finally, the training committee identified and defined all types of evaluations utilized by the program using both program and ACGME terms (Appendix 7) as well as which competencies they addressed (Appendix 8).
When they start working with the program, residents and faculty are given a program manual which provides information on the structure and expectations of the program. Much of the manual is committed to a description of the competencies and their applications and importance in the program and resident education. It is updated annually. A significant portion of the resident orientation and of the regular training director meetings are devoted to a discussion of the core competencies, with an emphasis on their importance in providing a framework to define and measure both the program’s and individual resident’s achievement and progress. Annually, the residents and faculty are asked for their input on the core competencies and the relevant documentation in meetings (group and individual) and correspondence. They are given copies (paper and electronic) and asked to make any changes that would be helpful. Throughout the year, the training director also reminds the faculty and residents of the objectives of the core competencies and facilitates discussion of what other documentation or measurement would be helpful. Over several years, as the faculty and residents have become more comfortable and conversant with the competencies, additional forums and mechanisms of assessment have been identified. For example, the residents and faculty developed a modified 360° assessment that considered the program’s structure and systems in targeting specific key aspects of resident performance (Appendix 9).
Though not without some glitches, the process of developing and implementing competencies in the program has gone relatively easily without causing significant stress or disruption (or complaints). Several decisions were key in facilitating the process: using the existing administrative personnel and structure to start the process; discussing the issues early in the process with key faculty and the residents; having the training director write the preliminary drafts, with others providing revisions; basing the content and documentation on existing descriptions and documentation; having frequent, regular discussions with all those involved; and revising the descriptions and documentation based on the input from all participants. Seeking feedback and input on a regular basis through multiple avenues from all involved has allowed ongoing refinement, modification to fit real life situations, and continued investment by those participating.
There have been a number of positive consequences of implementing the competencies. Revising the evaluation documentation to include competency standards appears to have provided faculty with additional incentive to provide more accurate written information about poor resident performance, since, on the flip side, they also have to provide written information about competent resident performance. On a more regular basis, faculty identify problems which they document on the evaluation form, though they do tend to discuss the issue first with the training director and ask for suggestions in terms of remediation. There also has been improvement in faculty giving a resident feedback earlier in the rotation with a clearer delineation of what the problems are and what is necessary to correct them. The residents in general seem to have more awareness of what they should be learning and whether they are learning it. And both faculty and residents have been active in suggesting modifications to forms to improve effectiveness, as well as in developing new evaluation forms.
Ideally, developing and implementing the competencies would not have involved quite so much work. However, this activity has been very helpful and informative in terms of helping program participants learn to think about and approach the process and outcome of psychiatric education in a more systematic and rigorous manner while still preserving the values and priorities of the program. Next, the program plans to examine our evaluation methods, with the aim of developing a model in which our assessment strategies and techniques are valid and reliable. Initially, the program will examine these issues internally. One idea is to compare a resident’s performance during different patient interviews (live and taped) while being observed by various faculty members completing the same evaluation form, and then comparing the resident’s interview performance to the resident’s mock board performances. Also, as the competency movement progresses, there is more literature on useful and effective measurement tools with increasing interest in using methods and forms across programs and specialties which the program plans to explore.
Various aspects of this article were presented at the Annual Meetings of the AADPRT in March 2001 (Seattle, Wash.), 2002 (Long Beach, Calif.), 2003 (San Juan, PR), 2004 (New Orleans, La.), and 2005 (Tucson, Ariz.). Many thanks to all the faculty, residents and staff who helped with this process.