Academic Psychiatry 27:145-147, September 2003
© 2003 Academic Psychiatry
Assessing Residents' Competence in Psychotherapy
Frank L. Giordano, M.D. and
David F. Briones, M.D.
Dr. Giordano is Director of Psychiatric Residency Training, Texas Tech University Health Sciences Center, El Paso, Texas. Dr. Briones is Director of Academic Programs for the Division of Psychiatry, Texas Tech University Health Sciences Center, El Paso, Texas. Address correspondence to Dr. Giordano, Department of Neuropsychiatry, Texas Tech University Health Sciences Center, 4800 Alberta Ave., El Paso, TX 79905-2700, fgiordano{at}ttmcelp.ttuhsc.edu (E-mail).
|
ABSTRACT
|
Objective: This article describes the development of a method for assessing resident competence in the five forms of psychotherapy specified in the program requirements for residency training in psychiatry. Method: Concomitant with the rise of evidence-based medicine. There has been a movement toward evidence-based training from the Accreditation Council for Graduate Medical Education (ACGME). In the year 2000, the Residency Review Committee (RRC) for psychiatry issued specific requirements for the assessment of residency competence in five forms of psychotherapy. This article describes how a system of assessment was developed using "brainstorming" and the Delphi method to meet these requirements and gives examples of the assessment tools that were utilized. Conclusion: A simple system that does not overly tax limited faculty resources can be developed to meet the new RRC requirements.
|
INTRODUCTION
|
Three major forces have shaped psychotherapy training in psychiatric residency programs during the past two decades: the burgeoning of neurobiology and its applications to psychiatry, the impact of managed care on psychiatrists' practice styles, and the need to demonstrate accountability in a quantifiable manner.
The shift in emphasis from psychodynamics to psychobiology in residency training was described in detail during a plenary address given before the California Psychiatric Association in 1990 by Dr. Robert Wallerstein (1). Dr. Wallerstein contrasted the 3000 hours of psychoanalytically oriented psychotherapy training that was common in psychiatric residencies after World War II (WWII) to the average 200 to 600 hours recommended in a 1990 Joint Task Force Report from the Association for Academic Psychiatry (AAP) and the American Association of Directors of Psychiatric Residency Training (AADPRT) (2). He also referred to a 1990 survey indicating that both the intensity and duration of residents' experiences in psychotherapy have diminished (3).
The impact of managed care on psychiatric practice styles has been considerable and is well documented (4,5,6). Reimbursement trends favor attention to current symptoms, psychopharmacologic management, and the use of brief, focused forms of psychotherapy, when psychotherapy performed by psychiatrists is even a covered treatment. Psychiatrists have responded to these fiscal pressures with shorter visits, more medication prescription, and less psychotherapy (4).
Accountability has been a growing watchword in health care for the past two decades. With the shift of health care from public service to commodity, standard American business principles have been applied by government and other third-party payers seeking quantitative forms of accountability common in business (7).
The Residency Review Committee (RRC) of the Accreditation Council for Graduate Medical Education (ACGME) has responded to all of these forces with ongoing revisions to the Program Requirements for Residency Training in Psychiatry. In 2000, the RRC issued a significant revision, including more than 20 new and updated requirements that went into effect in January of 2001. A new requirement that has caused a significant amount of consternation among faculty members everywhere is one that mandates documentation of residents' competence in five forms of psychotherapy: brief, cognitive-behavioral, combined psychotherapy and pharmacotherapy, psychodynamic therapy, and supportive therapy. Such a requirement represents a daunting task, considering the fact that psychotherapy training has decreased by thousands of hours in most programs during the past fifty years. Whether it will be possible to accomplish this task will largely rest upon the RRC's interpretation of "competence." As Wallerstein pointed out, 3000 hours of training, practice and supervision—constituting the norm 50 years ago—usually produced a psychiatrist competent in only one of the five forms of treatment (i.e. psychodynamic psychotherapy). If "competence" were equated with "expert," it would be impossible for training programs to accomplish this goal together with the myriad of other teaching objectives a modern residency in psychiatry is required to perform. If "competence" is interpreted as knowledge of the basic tenets of the five forms of psychotherapy and adequate clinical exposure to each form that will enable residents to become comfortable using them, demonstrating those levels of "competence" may be achievable.
|
DEVELOPMENT OF ASSESSMENT TOOLS
|
The faculty at the Texas Tech University Health Sciences Center-El Paso decided to formulate the assessment of resident performance on the definition of "competence" based upon demonstration of basic knowledge and sufficient clinical exposure resulting in beginning clinical comfort with each form of psychotherapy. Additionally, we decided to make any assessment system we developed multimodal. Although a number of rating scales for competence in various psychotherapies are available, they are either cumbersome to use or have reliability problems. For example, the Therapist Intervention Rating System takes 2 to 6 hours to complete (8). In 1981, Liston, Yager, and Strauss published a study of rating resident therapy skills using a detailed assessment instrument with videotaped therapy sessions (9). Interrater agreement proved to be low, albeit greater than chance. In an attempt to deal with problems as such, the psychiatric faculty at Texas Tech University Health Sciences Center (TTUHSC) -El Paso decided to use four different and concurrent modalities of assessment for each form of therapy being evaluated: chart review, videotape, supervision, and observation of a live patient session. We anticipated that this multimodal form of evaluation, together with the ability to longitudinally observe residents' work, would assure some degree of validity to the assessment.
The assessment instruments were purposely kept simple (see appendix). The criteria to be evaluated for each form of therapy were selected by the Residency Training Committee using a "brainstorming" technique formulated on the one described in Bienenfeld's article on the development of competency-based measures (10). In developing criteria for rating, our original list was abstracted from an article by Mohl, Sadler, and Miller (11), which were augmented during the brainstorming sessions with other criteria believed to be significant by Committee members. The criteria we ultimately used were then selected from this larger group by the Delphi method. The training director prepared and presented a number of iterations of each instrument to the Residency Training Committee. Each iteration was adjusted based upon feedback from the committee members until the final iteration was determined by consensus.
As shown in the appendix, each criterion is rated on a Leikert scale of 1 to 5, with a score of 1 being unacceptable and a score of 5 being outstanding. Within a given form of psychotherapy, the same assessment form is utilized for each of the four methods of assessment. Since not all of the listed criteria can be evaluated by each method of assessment, a sixth number was added to the Leikert scale on each form. This allows the rater to specify that a specific criterion cannot be assessed using that particular method. For, example, the ability to develop an appropriate case formulation should be ratable using chart review and supervision, but would be difficult to rate with a single midtreatment videotape. Space is also provided for additional comments that raters wish to make. Signature blocks for both rater and resident are placed on each form to ensure that the residents receive immediate feedback from faculty raters.
|
METHODs:
|
Our plan is as follows: During the postgraduate year (PGY)-III and IV years, each resident will be required to present one case for each form of psychotherapy to be assessed to a psychotherapy supervisor. In our program, many of the psychiatrist faculty members are qualified to supervise and assess the psychodynamic, supportive, and combined modalities. The cognitive-behavioral and brief modalities are taught and supervised by other faculty with appropriate training in these areas. Each case will be assessed utilizing the four different methods listed above: observation of a live patient session by the supervisor, rating of a midtreatment videotaped session, chart review of therapy notes, and evaluation of the treatment by traditional one-on-one supervision methods. The assessments will be documented on the forms shown in the appendix, utilizing the criteria developed for each therapy.
Each of the five types of psychotherapy will be recorded in the residents' logbooks. Under each type of therapy, the four assessment methods will be listed with a space for the rater's signature next to each method. Residents will be expected to collect each of the 20 signatures prior to graduation. If a resident is rated as unacceptable on two or more of the criteria listed on the assessment form, that entire assessment is graded as unacceptable. Any assessment graded as unacceptable will result in the resident having to repeat that particular assessment. If the resident receives a second unacceptable grade, a remedial training plan will be established for that resident in the area that is considered unacceptable.
|
DISCUSSION
|
As is true for most psychiatry programs, we have been struggling to develop appropriate means to meet the new RRC requirements for documenting residents' competencies in multiple areas. We selected the five required psychotherapies as the most challenging place to start because we felt that other elements of our resident evaluation system that have been in place for some time are reasonably well suited to the documentation requirements for the general competencies of patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. We opted to tackle the most difficult requirement first.
The development of competency-based measures is time consuming. We are sharing the criteria sets upon which we decided because it is our belief that sharing such efforts will be vital in enabling all training programs to succeed with this work-in-progress. As we continue to gather experience for implementation, we will report ongoing progress for others to examine.
|
REFERENCES
|
- Wallerstein RS: The future of psychotherapy. Bull Menninger Clin 1991; 55(4): 421–443
- Mohl PC, Lomax J, Tasman A, Chan C, Sledge W, Summergrad P, Notman, M: Psychotherapy training for the psychiatrist of the future. Am J Psychiatry 1990; 147(1): 7–13[Abstract/Free Full Text]
- Altshuler KZ: Whatever happened to intensive psychotherapy? Am J Psychiatry 1990; 147(4): 428–430[Abstract/Free Full Text]
- Olfson M, Marcus SC, Pincus HA: Trends in office based psychiatric practice. Am J Psychiatry 1999; 156: 451–457[Abstract/Free Full Text]
- Schreter RK: Reorganizing departments of psychiatry, hospitals, and medical centers for the 21st century. Psychiatr Serv 1998; 49:1429–1433
- Schreter RK: Physician service networks and the future for psychiatrists. Psychiatr Serv 1999; 50: 415–416[Abstract/Free Full Text]
- Kongstvedt PR, editor: The Managed Health Care Handbook, 4th edition. Gaithersberg, MD, Aspen Publishers, 2001
- Piper WE, Debbane EG, deCarufel FL, Bienvenu JP: A system for differentiating therapist interpretations from other interventions. Bull Menninger Clin 1987; 51(6): 532–550[Medline]
- Liston EH, Yager J, Strauss GD: Assessment of psychotherapy skills: the problem of interrater agreement. Am J Psychiatry 1981. 138: 1069–1074[Abstract/Free Full Text]
- Bienenfeld D, Klykylo W, Knapp V: Development of competency-based measures for psychiatry residency. Academic Psychiatry 2000; 24: 68–76[Abstract/Free Full Text]
- Mohl PC, Sadler JZ, Miller DA: What components should be evaluated in a psychiatry residency. Academic Psychiatry 1994. 18(1): 22–29[Abstract]
This article has been cited by other articles:
|
|
|
|
 
K. A. Khurshid, J. I. Bennett, S. Vicari, K. L. Lee, and K. E. Broquet
Residency Programs and Psychotherapy Competencies: A Survey of Chief Residents
Acad Psychiatry,
December 1, 2005;
29(5):
452 - 458.
[Abstract]
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2003
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
