Advanced Search

Regular Article | June 01, 2000

Process and Product: Development of Competency-Based Measures for Psychiatry Residency

David Bienenfeld, M.D.; William Klykylo, M.D.; Victor Knapp, M.D.

Academic Psychiatry 2000;24:68-76. 10.1176/appi.ap.24.2.68

View Article Information

Resident EvaluationAssessment ScalesResidency Curriculum

Dr. Bienenfeld is Professor and Vice Chair and Director of Residency Training, Wright State University, Dayton, Ohio. Address reprint requests to Dr. Bienenfeld, Wright State University Department of Psychiatry, 2222 Philadelphia Drive, Rosary Hall, Dayton OH 45406. e-mail: david.bienenfeld@wright.edu

text A A A

Abstract

The authors describe a group effort to devise objective and measurable standards for assessing competency of psychiatric residents in a number of domains, using techniques of brainstorming, affinity grouping, and priority matrices. Results from each step were used to refine the next step. A checklist of minimum skill competencies was successfully devised and implemented. Skills were allocated to five domains: Assessment and Presentation, Diagnosis, Somatic Treatment, Side-Effect Management, Consultation—Liaison and Medical Psychiatry, and Psychotherapy. They concluded that competency-based measures can be formulated for psychiatric residency training. Although the set of competencies derived for the authors' institution may be particular to local goals and values, the general structure and the process of development are transferable.

Figures in this Article

Because the purpose of residency education in psychiatry is to produce quality practitioners of the specialty, it stands to reason that the components of that quality should constitute the curriculum of the training program. Measurement of student progress is an essential element of maintaining the relevance and effectiveness of the curriculum. Ideally, then, a residency training program in psychiatry should have clear objectives for its residents to achieve and mechanisms for assessing the attainment of those objectives.

Nonetheless, evaluation of programs and individuals tends to be performed at the level of resident activity rather than resident achievement. This trend is reflected in the current Program Requirements for Residency Education in Psychiatry of the Accreditation Council for Graduate Medical Education (ACGME;1). The requirements speak with specificity about the need to "provide sufficient experiences in" multiple aspects of psychiatric practice, but do not address the expectations of trainee accomplishments. The ACGME, however, is moving in the direction of outcome measurement, having adopted the following position:

The ACGME supports the increased use of outcomes assessment in the accreditation process and the development of an overarching model for outcomes assessment in accreditation that can be applied across the specialties (2).

Some specialties have made efforts to devise competency-based assessment schemata for residents, including preventive medicine (3,4) and family practice (5). The program requirements in internal medicine (6) and in physical medicine and rehabilitation (7) already require a number of competencies from all programs. There is no reason why psychiatric education should be less amenable to such outcome measurement.

In general, the more specific the target identified, the more likely the educational program is to achieve it. Conversely, the level of abstraction in a training goal is inversely proportional to the validity of its measurement (8). Thus, performance objectives for medical training should be broken into their smallest components: formative assessment measures steps en route to ultimate competency. If the steps are performed properly, then only the rare resident will fall short on the summative assessment (9).

Aside from the inherent educational motives in developing competency-based performance measures, there are external motivations, as well. The requirements for due process in training assessment and promotion make objective and standardizable measures desirable. Further, healthcare organizations seeking to credential alumni of psychiatry residency programs will often ask about graduates' competency to perform certain tasks. Documentation of such competencies during the course of training provides the training director with sound footing for answering such queries.

For all these reasons, the Residency Training Committee of the Wright State University Department of Psychiatry undertook to develop a set of competency-based criteria for measuring skills of our trainees. We describe herein the process by which the competencies were developed and present the checklist we devised.

METHODS

Abstract | METHODS | RESULTS | DISCUSSION | References

The body developing the list of competencies was the Residency Training Committee of the Department of Psychiatry. The Committee is composed of the associate training directors from each of the four major hospital systems; the service chiefs for Child and Adolescent Psychiatry, Forensic Psychiatry, and Community Psychiatry; two chief residents; and one representative from each residency class, in addition to the Director of Residency Training and the Deputy Director of Residency Training. A stepwise procedure was planned and followed:

RESULTS

Abstract | METHODS | RESULTS | DISCUSSION | References

Skills Lists and Groupings

The first brainstorming produced a list of 65 items. By consensus, these were grouped into 5 major categories encompassing 14 subcategories. The outline of categories is presented in Table 1.

Priority Matrix Criteria

A list of 10 criteria was created for the priority matrix. Five were considered by consensus to be of high importance; four, of medium importance; and one, of low importance. These criteria are listed in Table 2.

Priority Matrix Results

Mean priority scores ranged from 15.11 (detoxify from alcohol; provide psychotherapy for mood disorders) to 4.59 (provide primary-care medical management of psychiatric patients).

Construction of the Checklist

Using the results of the priority matrix, 12 skills were eliminated (mostly those with priority scores of 9 or less). Twenty-two items were combined with very similar others to yield seven items. Ten items were expanded for greater specificity to yield 30 items. Some items were deemed to be important but not mandatory and were reorganized into DSM-style format to require one or several of a list of skills to be demonstrated. (For example, by the end of PGY-IV, the resident must provide at least one of the following: outpatient group psychotherapy, interpersonal therapy, or behavior therapy.) Major categories were organized as: Assessment and Presentation, Diagnosis, Somatic Treatment, Side-Effect Management, Consultation—Liaison and Medical Psychiatry, and Psychotherapy. Table 3 presents the list of skills requirements and cases/episodes required, grouped by category and PGY level.

DISCUSSION

Abstract | METHODS | RESULTS | DISCUSSION | References

The list of skills was transformed into a checklist format distributed to all residents and faculty. Figure 2 illustrates the format of the checklist. It is the responsibility of each resident to obtain a faculty member's initials each time a skill is demonstrated. The resident brings the list to the Training Director at the time of his or her semiannual review, where the contents are discussed. If the resident is in danger of not meeting any of the criteria, the faculty and training director can ensure that resources are made available for the trainee to complete the expected task.

The feedback function of the checklist applies to the curriculum at large, as well. If certain opportunities are deficient, the curriculum can be modified to bring it into line with the goals that served as the backbone for the competencies list in the first place. If, for example, it is noted that more than a few residents are having difficulty using depot neuroleptics in three or more patients, then the rotation schedule can be realigned to provide exposure to appropriate patients in settings where this modality is used.

After the first year of implementation, some of these changes are already evident. The competency checklist requires that residents demonstrate the capacity to perform cognitive therapy, yet there were too few faculty to teach and supervise this modality. Accordingly, the department underwrote the training of four faculty members at a nationally respected institute of cognitive therapy, with the specific contract that they were to be trained as both practitioners and supervisors.

Very quickly, residents and faculty members were using the checklists as a device to guide case assignments. Although faculty had always subscribed to the principle that caseload should be adjusted to each resident's needs, there had not previously been a yardstick for determining that need. Within the first 2 months, residents were asking their supervisors for exposure to electroconvulsive therapy training; faculty were adjusting the assignment of admissions so that a resident who was deficient in seeing patients with delusional disorder would have the requisite exposure. The clinical chief resident modified the procedure for assigning psychotherapy cases to meet the same objective.

By the end of the first 6 months of implementation, 3 of 30 residents (10%; 1 PGY-III and 2 PGY-IV) had not documented completion of at least half of the respective year's requirements. By the end of the first year, only one PGY-I resident (3%) had failed to obtain initials in all boxes for the respective year.

A practical wrinkle in the implementation of the program was a cohort problem. Some of the residents had met the skills requirements in the year(s) before the implementation of the checklist. Indeed, the checklist was structured so that the requirements are assigned to the latest reasonable PGY level. As a result, for example, a PGY-III resident who had seen a child with psychosis in PGY-II would have to get that requirement retroactively initialed by the faculty member if that teacher was available and could remember the case. Flexibility was allowed in such circumstances.

An issue that affects much of evaluation in psychiatric education is the means of assessing skills. We have thus far used only faculty observation as the primary measurement criterion. The annual clinical skills examinations performed in our residency provide a more objective venue for assessment. In future years, or at other locations, alternative measures may prove useful or necessary.

Our residency has also trusted the veracity of residents in obtaining faculty initials. In a less trusting setting, forgery of faculty initials would be easy to accomplish. Full signatures or spot-check verifications with faculty might be initiated.

The establishment of minimal competency requirements does not imply that the skills defined are the only elements of physician competency. Rather, the requirements list describes the core behavioral skills mandatory for the competent physician. Successful completion requires knowledge and attitudes beyond the observed behavior, but there are many areas of knowledge and attitude that are additional characteristics of the competent physician (12). Our program elected to begin with skills exclusively because we anticipated a lengthy and complicated road to devising the process herein described. The dimension of clinical skills seemed to be the simplest and most concrete part of trainee competency, less vulnerable to subjectivity than measuring attitudes, or measuring volume, as in assessing knowledge.

Certainly, the ability to perform a history and mental status exam, and the ability to present a comprehensive biopsychosocial formulation does not necessarily reflect competency in empathy or rapport, which we would expect of a competent trainee. At present, our Residency Training Committee is developing a list of knowledge competencies by the same methods used to derive the skills list. Outcome criteria for attitudes and behavior will be much more of a challenge, but might indeed be feasible.

Although the information base used for generating this list of competencies is probably common to most residency training programs, regardless of sponsorship or location, the criteria used to rank their priorities are local. This is a list that works well for the values and resources of our own institution. Other programs might well require different skills or different degrees of proof of competency for each. The method by which the list was derived, however, is quite generalizable and can serve as a template for others.

TABLE 1. Categories and subcategories of skills to be ranked

View Large | Download Slide(.ppt)
Add to My POL

TABLE 2. Criteria for priority matrix

View Large | Download Slide(.ppt)
Add to My POL

TABLE 3. Competency requirements

View Large | Download Slide(.ppt)
Add to My POL

FIGURE 1.: Example of priority matrix

View Large | Download Slide(.ppt)
Add to My POL

FIGURE 2.: Excerpt from residents' checklist

References

Abstract | METHODS | RESULTS | DISCUSSION | References

Accreditation Council for Graduate Medical Education: Program Requirements for Residency Education in Psychiatry. 1994
http://www.acgme.org/rrc/psych/psy.htm,

Accreditation Council for Graduate Medical Education: Minutes of September 23, 1997

Lane DS, Ross V: The importance of defining physicians' competencies: lessons from preventive medicine. Acad Med 1994; 69:972-974 [PubMed][CrossRef]

Lane DS, Ross V, Parkinson MD, et al: Performance indicators for assessing competencies of preventive medicine residents. Am J Prev Med 1995; 11:1-8

Bell HS, Kozakowski SM, Winter RO: Competency-based education in family practice. Fam Med 1997; 29:701-704[PubMed]

Accreditation Council for Graduate Medical Education: Program Requirements for Residency Education in Internal Medicine. 1997
http://www.acgme.org/rrc/IM/internal.htm,

Accreditation Council for Graduate Medical Education: Essentials and Information Items. Washington, DC, American Medical Association, 1995, pp 199-203

McAshan HH: Competency-Based Education and Behavioral Objectives. Englewood Cliffs, NJ, Education Technology Publications, 1979, pp 49-70

McGaghie WC, Miller GE, Sajid AW, et al: Competency-Based Curriculum Development in Medical Education. Geneva, Switzerland, World Health Organization, 1978, pp 11-77

Joint Commission on Accreditation of Healthcare Organizations: Using Performance Improvement Tools in Health Care Settings. Oakbrook Terrace, IL, Joint Commission on Accreditation of Healthcare Organizations, 1996, pp 37-67

Saylor JH: TQM Field Manual. New York, McGraw-Hill, 1992, pp 99-105

Weaver CN: TQM: A Step-by-Step Guide to Implementation. Washington, DC, ASQC Quality Press, 1991, pp 137-156

Download all saved figures as a .zip file.

View Large
Download Slide(.ppt)

Add to My POL

FIGURE 1.

Example of priority matrix

View Large
Download Slide(.ppt)

Add to My POL

FIGURE 2.

Excerpt from residents' checklist