Effective January 2001, as authorized by the Accreditation Council for Graduate Medical Education (ACGME), the Residency Review Committee for Psychiatry mandated that psychiatric residency programs must demonstrate that residents have achieved competency in at least the following forms of treatment: brief therapy, cognitive-behavioral therapy, combined psychotherapy and psychopharmacology, psychodynamic therapy, and supportive therapy. (See http://www.acgme.org/RRC/Psy_Req3.asp.) This article will briefly examine methods and rituals used for demonstrating summative competency in other fields and examine methods available to psychiatric educators for assessing psychotherapy competence in residents. On the basis of this review we summarize what we believe psychiatric educators can and cannot competently assess regarding psychotherapy competencies and offer recommendations to the Residency Review Committee for issuing a realistic restatement of requirements regarding psychotherapy competencies for residents.

In a memorable monologue, the comedian Jackie Mason once dealt with the difficulties of assessing his physician's specific competencies. Paraphrased, it went something like this:

Let's say I have something wrong with my kidney and I want to find a doctor who knows something about kidneys. My friend tells me to go to Dr. Smith. He's supposed to be terrific. What do I know? I go to Dr. Smith's office. There on the wall are his diplomas. Very impressive. But they don't tell me what he knows about kidneys. What if his diploma even says "He graduated from medical school with a 90 average." That's great, but what if he only got a 60 in kidney?

Moreover, what would a 60 in kidney really mean regarding Dr. Smith's competence to deal with renal problems? Jackie Mason's dilemmas are our dilemmas.

Now, let's get personal. Imagine that you have bad chest pain; go to a hospital; get an angiogram; and are told that you need five vessel by-pass surgery. You ask around and you are told that Dr. Jones is a terrific cardiac surgeon. How do you know how if Dr. Jones is competent to perform your surgery? Why should you have any faith in his credentials? The fact is, that to be board certified in thoracic surgery, Dr. Jones had to demonstrate that he's conducted 100 operations on adult hearts, including 60 myocardial revascularizations, all closely supervised (http://www.abts.org/doc/4018#OPCASE).

Next, how do you know that your 16-year-old is competent to drive the family car? You never do know for certain, but you start by seeing to it that he or she learns the rules of the road, has a considerable amount of closely supervised instruction and experience, and, finally, passes a written test and a road test, in which a critical and paranoid driving examiner sits next to your driver wannabe from start to finish, meticulously grading each and every component of driving. The difference between this certification and that of the surgeon is that the road test never really tests your offspring under the trickier conditions you're most concerned about, for example, driving at night in bad weather or with four other 16-year-olds in the car after a Saturday night party. The teen driver is certified at a different level of expectation from the surgeon.

What broad lessons about demonstrating competency can be drawn from these and other such examples? To be endorsed as competent in a procedure, learners usually have to do the following:

How likely is it that psychiatric residency programs will ever be able to assure that residents fulfill these expectations for the five types of psychotherapy? To what extent will residency programs ever be able to assure that residents are capable of passing knowledge tests, observe others conduct the psychotherapy in a competent manner, are observed and supervised in practicing the psychotherapy, and pass appropriate performance tests for each of the psychotherapies? In what follows we use this framework to assess realistic likelihoods.

First, to what extent do we have acceptable, agreed upon definitions of what constitutes pre-requisite knowledge and competent procedural skills for each of the five types of psychotherapies? To start, we must recognize that unidimensional definitions of competence are inadequate. The teenager who is unquestionably competent to drive six residential blocks to the grocery may not yet be competent to ferry his/her younger sibling to a remote summer camp. "Competence," first of all, is contextually defined (1).

Second, competence is usually dimensional rather than dichotomous. And, we must distinguish summative competencies, abilities to perform at final, definitive levels, from formative competencies, intermediate steps of ability that have to be mastered on the way to summative levels (1,2). The thoracic surgeon at one point was competent to suture a superficial laceration, but not to patch an aorta. These complexities challenge those who seek to assess competence for any set of expectations, much less fields as complicated as the psychotherapies. Competence assessment requires measurements that have content validity (i.e., measure what we think they measure), and either predictive validity (i.e., correlate with future competence) or concurrent validity (i.e., measure qualities possessed by those of known summative competence) (3). We then have to agree on what constitutes acceptable levels of competence for each measure.

The frameworks provided by the work of the American Association of Directors of Psychiatric Residency Training (AADPRT) Task Force on Competence described in other papers in this issue are good starting points for dimensions to assess. But the details of translating these broad statements into acceptable ways of assessing competence are formidable and can get murky in practice.

For example, let's say your sister in another city requires psychodynamic psychotherapy for a particularly thorny, long standing difficulty. How would you go about finding a psychiatrist who's competent to handle your sister's specific problems?

To identify someone for your sister, you'd probably get on the phone and ask around for the best. You'd want to be certain that the psychiatrist has been well trained, suitably credentialed, and well regarded by his or her training director. This means that he or she has a decent personality and empathic capacity, is highly ethical and not likely to violate professional boundaries, is experienced and capable in treating your sister's particular problem, and maybe even has a track record of good results. You'd want to be certain that this psychiatrist is completely competent at the level of excellence. What if you got the name of a psychiatrist just out of training? To what extent will that psychiatrist's test scores, observed practice, and live examinations be able to convincingly demonstrate this person's competence to treat your sister? To what extent would the fact that someone had training at an appropriate psychoanalytic institute or cognitive behavioral therapy (CBT) institute increase your faith in their competence?

If the patient in question were the sister of someone you never met, your standards might be different. You might feel safe, if not completely enthusiastic, referring to a recent graduate of a training program of known quality. Competence, in this instance, would be characterized by presumed safety and adequacy rather than by excellence and expertise.

Unfortunately, the best tests and closest observations are not always able to detect the sorts of worrisome forms of incompetence that are likely to plague psychotherapy practice, nor to assure that program graduates all have the features of empathy, sensitivity and flexibility. To paraphrase Einstein, "not everything that's countable counts, and not everything that counts is countable." Further complicating our challenge is the slipperiness of defining competence without specific context.

Knowledge Tests: As experienced psychiatric test writers know, writing exam questions concerning psychotherapies is very difficult to do. Many questions regarding psychotherapy address easily acquired definitions of terms and concepts, and many of these questions seem trivial; even experts find writing better psychotherapy questions to be vexing. In theory, one could create patient-management problem vignettes requiring essay-type responses or use oral examination formats that require candidates to show considerable knowledge without the benefit of substantial prompting. Reliably rating these exams can be difficult. Even well constructed exams such as the Columbia Psychotherapy Test are able to measure knowledge but not skills as they are called upon in actual practice.

Training for the Practice of Psychotherapy: Even in operationally defined manualized psychotherapies such as cognitive and interpersonal therapies, moment to moment moves depend not only on the therapist's knowledge and skills regarding the specific clinical problems and specific psychotherapies but on nuances of personality in both patient and therapist, and within the interpersonal field.

All agree that the experience of developing into a competent psychotherapist, even of thinking of oneself as a competent psychotherapist, requires not only book learning but a certain amount of time, experience and maturation. Tanya Luhrmann confirmed the lore of psychotherapy training in finding that her psychiatrist-subjects didn't feel competent as psychotherapists until they'd been out of residency for at least several years (4). They weren't thinking about formative competencies. If therapist's self ratings are taken as a standard for minimal ratings of summative competence, then her observations challenge those requiring programs to demonstrate that graduates have achieved "competency" in psychotherapy by the time they leave training. Since most lay people and attorneys take the term competence to refer to summative competence, the semantics become very interesting. When you hire a plumber who's supposed to be certified as "competent," are you satisfied with formative competencies? That said, let's consider the elements of psychotherapy training that should, in theory, go toward producing competence.

To what extent do residents observe complete demonstrations of competent psychotherapy? How often do psychiatric residents actually get to watch competent psychotherapists conduct full psychotherapies, to question and discuss the tactics, and to learn by modeling? Almost never. In how many programs do residents observe a single entire psychotherapy from start to finish, let alone one of each of the five specified types? Although many programs still offer some type of observed psychotherapy (e.g., a highly select brief psychodynamic psychotherapy, portions of a psychodynamically informed assessment, or live supervision via co-led family or group therapy), that's about it. So much for "see one." To what extent is psychotherapy the sort of skill in which one can achieve competence without ever having seen a single performance?

To what extent are residents observed and critiqued regarding their actual practice psychotherapies? Before being assessed for summative competence, learners ordinarily have a chance to practice, make mistakes, and have their still-imperfect performances corrected. We all hope that psychiatric residents will, in fact, have some experience performing each of the required types of psychotherapy under close supervision. But, for the most part, the psychotherapeutic procedures themselves are unobserved. Few faculty actually watch these therapies or review full taped sessions throughout the course of treatment, or even for substantial portions of treatment. Usually, residents report to supervisors their perceptions and recollections of what they've been doing in psychotherapy (sometimes offering "process notes" that are highly processed and often skewed selections), and supervisors provide guidance according to what they hear from the residents, not according to what they personally observe. Often residents unwittingly (less so wittingly) distort what occurs in psychotherapy through omission or slanted reporting of sessions in order to make themselves appear better than they are, to conceal interactions they believe their supervisors won't approve of, or to prevent themselves from being shamed or embarrassed. Serious errors in therapy often come to the attention of supervisors indirectly, from complaining patients or from untoward events in the patient's life. When errors of this magnitude occur, unless they are due to egregious ethical violations (serious boundary problem violations, for example, leading to a resident being labeled as "dangerous"), they may be written off to inexperience requiring more training, and the resident will simply be given the opportunity for additional training, if there's still sufficient time remaining.

Through these sorts of supervision rituals, supervisors often believe that they can distinguish the smart from the dumb. Smart residents are discerning and discriminating listeners and reporters, picking up on nuances and complexities of thought, context, nonverbal communication, transference, and the like. What do we do about residents who are "dumb" in psychotherapy, those whose capacity for nuanced observations and reflections is limited? At what level do we consider them incompetent, rather than just not very good?

Assessment Problems: After residents have practiced enough to be evaluated for competence in psychotherapy and actually submit samples of their work for assessment, how competent are we to perform these assessments? Even when psychotherapies are observed and rated, experienced supervisors may not agree on what they see or on what they would consider to be competent. Liston, Yager, and Strauss studied experienced psychotherapy supervisors who examined videotapes of mid-therapy sessions of psychodynamic psychotherapies provided by advanced psychiatric residents. Even with pre-training, these raters were not able to agree on what they saw or on how competent the psychotherapeutic tactics of the residents were in the sessions they observed (5). While data on assessing more standardized therapies such as cognitive and interpersonal therapies is not entirely consistent and sometimes frankly problematic (6), some studies suggest that competence of trainees can be reliably measured against various yardsticks (7—10). The extent to which lessons from CBT and IPT can be applied to psychodynamic, supportive therapies and combined therapies is uncertain.

Even if raters could agree that competent psychotherapy was being performed during the sessions they observe, how many therapies (or individual sessions within therapies) would have to be observed if raters were to be able to certify competence? Would one or two suffice? Not if you're an educational evaluator. Modestly, most educational evaluators suggest that at least multiple different observations made by several different observers are required before one can state with any confidence that a learner has mastered a procedure (11—13). What are the odds of having residency programs rate eight or more examples of each of the five specified psychotherapies? Any single modality of assessment (process notes, case presentations, taped sessions) will have significant shortcomings. One imperfect approach to this problem is to combine several modalities of assessment to "triangulate" the specific dimension and level of each objective towards psychotherapy competence (1).

To our knowledge:

Consequently, we contend that not only is it currently impossible for programs to determine summative competence of psychiatric residents in these five types of psychotherapy, but that it will never be feasible for them to do so. Therefore, in our view it is unreasonable to ask psychiatric training programs to declare residents competent to perform certain types of psychotherapies at even a minimal summative level. To insist on certification that residents are competent to practice these psychotherapies, as the term is generally understood by the lay public, would put program directors in the untenable situation of either refusing to do so or being dishonest. Neither alternative is acceptable..

This does not mean, however, that we should throw up our hands. Programs can go a long way toward assuring that psychiatric residents have obtained formative elements of training that can contribute to their ultimately becoming somewhat competent in psychotherapy. Programs can conduct formative assessments. These activities may verify that all psychiatric residents at least have knowledge and some experience with the required types of psychotherapy.

Regarding the assessment of formative competencies, several options exist:

However, let's be clear. None of these assessments, alone or in concert, will ever suffice to demonstrate summative competence in psychotherapy. In none of these assessments is an individual actually observed performing the procedure from start to finish in a single case.

Finally, the field should define what sorts of incidents or behaviors are sufficiently egregious to be labeled "dangerous," and how many of these behaviors at what stages of residency training should define "incompetence" (15). If we are expected to define competence, we should also be authorized to define incompetence and have the field stand behind these definitions from a medicolegal perspective.

In our view, the motivations of the Residency Review Committee are to be applauded, as they are intended to strengthen the likelihood that all psychiatric residents receive reasonable training in psychotherapy. However, we maintain it is equally important for the RRC to "get real" and not foster dishonest requirements or impossible expectations upon training programs just to follow the "Pied Piper" of competence assessment. To this end, several steps should be taken:

"Residency programs must show that all residents can demonstrate knowledge about the evidence base, theories and rules of practice supporting at least the following forms of treatment: brief therapy, cognitive-behavioral therapy, combined psychotherapy and psychopharmacology, psychodynamic therapy, and supportive therapy. Furthermore, programs must demonstrate by means of patient logs, audiotape or videotape recordings and other forms of documentation that all residents have conducted at least one psychotherapy case of each type, each to a point of clinical resolution or equilibrium, under qualified supervision."

We harbor no illusions. We believe that institutionalizing, implementing and supporting just these formative requirements will not be easy. If they prove impossible to effect, training programs should not be saddled with unrealistic burdens with predictable, negative consequences. At the same time, setting forth realistic expectations may foster the intended consequences of assuring that psychiatric residents obtain psychotherapy training and increase their odds of ultimately becoming competent comprehensive psychiatrists.