Academic Psychiatry 27:136-144, September 2003
© 2003 Academic Psychiatry
Evaluating Competence in Psychotherapy
John Manring, M.D.,
Bernard D. Beitman, M.D. and
Mantosh J. Dewan, M.D.
Dr. Manring and Dr. Dewan are with the Department of Psychiatry, State University of New York, Upstate Medical University, Syracuse, New York. Dr. Beitman is with the Department of Psychiatry, University of Missouri, Columbia, Missouri. Address correspondence to Dr. Manring, Department of Psychiatry, SUNY Upstate Medical University, 750 East Adams Street, Syracuse, NY 13210, manringj{at}upstate.edu (E-mail)
|
ABSTRACT
|
Background: The Residency Review Committee (RRC) for Psychiatry has recently charged psychiatry training programs with developing methods to demonstrate competence of trainees in five areas of psychotherapy. Each program must decide what specific skills are essential for competence in each of the five listed psychotherapies. This requires determining whether those skills that are necessary are also sufficient for effective psychotherapy and whether additional specific skills are required for each one. Method: Two lists of general skills for psychotherapy are compared, one from the perspective of specific "schools" of psychotherapy and one from a more eclectic "integrative" approach. The issue of measuring competence is addressed by placing ratings of "competent" midway on a continuum from "novice" to "expert." Thirteen methods for measuring competence from the Accreditation Council for Graduate medical Education (ACGME) "tool-box" are described and reviewed with respect to applicability to psychotherapy. Examples of toolbox implementation are described based on a functioning psychotherapy evaluation program at the University of Missouri. Results and Conclusions: The authors found both theoretical as well as practical problems in measuring competence in psychotherapy. We propose that global rather than highly specific assessment methods may be more practical in these early stages of development, and we offer specific suggestions for assessment components that can currently be implemented.
|
INTRODUCTION
|
As part of a broader movement in medicine to use "outcome-based" and "evidence-based" treatments, increasing pressures from the public, government, health insurers, and certification agencies are increasingly requiring programs to demonstrate that trainees actually learn what training programs say they are teaching them. The Accreditation Council for Graduate Medical Education (ACGME) has decreed that trainees in all medical specialties must develop competence in six core areas: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice (1). The Residency Review Committee (RRC) for Psychiatry has chosen an additional five areas of psychotherapy in which trainees in psychiatry are to be certified by their training programs as competent: brief therapy, cognitive-behavioral therapy, combined psychotherapy and psychopharmacology, psychodynamic therapy, and supportive therapy (2). Some of these therapies are closely allied to or derivative of others, and there is considerable overlap in practice. This paper will explore issues affecting the dependable assessment of competence in several of these complex and varied types of psychotherapy.
We know something about which aspects of these five psychotherapies are important for effectiveness (3–13). However, we know little about which aspects are sufficient and which contain elements that go beyond what is required to facilitate therapeutic change in patients. In other words, we have yet to define clearly what the minimum is that one must do to have successful outcomes. To assess competence, we must start with two problems: what to measure and how to measure it.
|
What to Measure: Enumerating Skills Needed For Competence in Psychotherapy
|
Beyond existing research on what specific skills are essential for effective psychotherapy, the field has turned to experts to help define specific skills believed to be important for psychiatric residents. Some believe that the more specifically we can define those skills (i.e. the ability to begin and end a session on time) the more likely psychotherapy will be effectively taught and learned. Several attempts have been made to distill psychotherapy skills into their smallest identifiable units, the acquisition of which, it is hoped, will lead to the competent performance of psychotherapy (14). While this may be an effective way to teach psychotherapy, it is a leap of faith to assume that observing these units automatically translates into the best way to assess competence in psychotherapy. In contrast, many government agencies, professional institutions, and other medical specialties use a variety of outcomes to assess competence, including assuring that trainees demonstrate the successful treatment of an adequate number of specific kinds of disorders, examining rates of errors and medical complications through the venerable "morbidity and mortality" reports and conferences, and by means of other outcome-based methods.
Students of psychotherapy practice tend to distinguish basic skills believed to be necessary for all the psychotherapies, such as abilities to manage boundaries, develop a therapeutic alliance, listen, deal with emotions, and understand, from those more specialized skills considered necessary for specific kinds of psychotherapies. While extensive, lists of basic and specialized skills still include statements of proficiencies that are quite vague, posing difficulties in both teaching and assessment. To illustrate, compare the list of general psychotherapy skills (15) shown in Table I to the examples in Table II and Table III , two specific psychotherapy skill lists for cognitive-behavioral and supportive psychotherapies from the AADPRT Psychotherapy Task Force list of "psychotherapy competencies" (15). While posited as necessary for effective psychotherapy, the skills listed in Table I may not be sufficient for effective psychotherapy and certainly not for all disorders in all contexts. To increase the effectiveness of psychotherapy, one would need to add skills specific to the various individual kinds of psychotherapy, such as those shown in Table II concerning cognitive-behavioral psychotherapy and Table III, listing skills for supportive psychotherapy. And, while the lists on Tables II and III address core psychotherapies, it is evident that each derives from a different school of psychotherapy requiring a different set of assumptions and a different "headset," even during the initial assessment period. Often, trainees choose which approach to use from the outset of therapy, sometimes before fully recognizing which specific type of psychotherapy is likely to be most beneficial. Beitman and Yue (16) offer another set of general skills that, despite coming from a different, more integrative perspective, show considerable overlap with the AADPRT Task Force list. As in the AADPRT Task Force construction, Beitman and Yue include broad skills such as therapeutic listening, establishing a working alliance, recognizing resistance, identifying transference and countertransference, and termination, along with many other similar skills. Unlike the AADPRT list, they have included additional general skills which, if learned effectively, they believe should be sufficient for conducting effective psychotherapy without requiring many additional "school-specific" skills. As is evident, many of Beitman and Yue's additional general skills contain and merge into what others may categorize with skills associated with the specific types of psychotherapy. Beitman and Yue's list of skills are shown in Table IV. Beitman and Yue's skills lists are compatible with augmentation by additional skills from individual "schools" of psychotherapy (e.g., cognitive-behavioral psychotherapy, psychodynamic psychotherapy).
Simply listing skills that may be sufficient for effective psychotherapy still leaves us with the problem of deciding which skills are essential for certifying competence, whether all skills are necessary and at what point of mastery of the skills trainees are to be considered competent. These questions remain to be answered by future research and expert deliberation. As we approach these issues, the following considerations may be helpful.
|
How to Measure It: Tools to Assess Competency
|
Competent is defined as "suitable, sufficient, or adequate" (Webster's 2nd Edition, 20th Century Unabridged Dictionary). It may be helpful to think about competence on a continuum of increasing skill as described in the Dreyfus model of skill acquisition (17). Table V illustrates these concepts. Most significant is the placement of "competent" as a skill level midway between novice and expert. A trainee need not fully master all the individual skills of psychotherapy to be considered competent.
Once we agree on the skills necessary for competence in psychotherapy, how do we assess whether trainees actually possess these skills? ACGME constructed a "tool chest" of 13 "best methods" for evaluating traditional areas of competence in medical education (18). We next address these methods and comment on their applicability to psychotherapy. We will focus on the methods most likely to be useful for psychotherapy competence assessment but mention the others briefly both for the sake of completeness as well as in the hopes that mentioning them may stimulate creativity and result in innovative psychotherapy-related applications in the future.
|
Potentially Useful Psychotherapy Competence Assessment Methods from Toolbox
|
Patient Surveys
Patients are directly surveyed to assess satisfaction with services provided (e.g., amount of time spent with the patient, a therapist's skill, courtesy, and empathy or a therapist's provision of information about interview findings, treatment steps, and/or policies). Specific aspects of psychotherapy, including overall treatment outcome, working alliance, global session evaluation, and patient reactions to individual interventions, can also be surveyed (16). Reliability estimates of 0.90 or greater (1.0 is perfect agreement) have been achieved in rating hospitals and clinics but are much lower for individual trainees (18). Difficulties encountered with patient surveys are: 1) language and literacy problems; 2) obtaining enough surveys per resident to provide reproducible results; 3) the resources required to collect, aggregate, and report survey responses; and 4) assessment of the trainee's contribution to a patient's care separate from that of the treatment team.
360-Degree Evaluation Instrument
This method requires assessment by multiple people in a trainee's sphere of influence (e.g., superiors, peers, subordinates, patients and families). To our knowledge, there are no published reports of 360-degree evaluations as applied to graduate medical education. In business, military, and education settings, reliability has been estimated as high as 0.90. It is theoretically appealing to collect reliable, primary outcome data by this method. However, it may be difficult to design a single instrument for rating psychotherapy skills appropriate for use by supervisors, co-trainees, clinic staff, patients, and families. Each group of respondents has different needs, perceptions, and experiences of trainees. Several different assessments would have to be "triangulated" and integrated. Further, the administrative complexities of distributing and collecting data and integrating the results meaningfully are daunting. Nevertheless, the 360-degree evaluation has the potential to become a powerful tool in psychotherapy evaluation.
Chart Stimulated Recall Oral Examination (CSR)
A CSR exam uses the trainee's case record as the basis of a standardized oral exam of the care provided, probing for reasons behind the diagnoses, interpretations of clinical findings, and treatment plans. Examiners rate the trainee using a well-established protocol and scoring procedure. Reliability ranges from 0.65 to 0.88. In psychotherapy training "process notes" have long been the mainstay of supervision and evaluation, even though trainees' process notes rarely provide the detail and objectivity necessary for thorough evaluation. A variant of the CSR technique holds promise for psychotherapy assessment. Trainees' videotapes from several stages of treatment, such as the initial interview, selecting a focus, employing strategies for change, and termination, may provide accurate "snapshots" of their evolving skills and decision making. Trained examiners may follow standardized protocols to orally examine single cases or a "portfolio" of a trainee's work, which we discuss below. Drawbacks arise in the expense of equipment, technical ability necessary to manage the equipment, acquiring informed consent, and examining an adequate number of stages of therapy in an adequate range of patients in the various schools of therapies. To our knowledge, there are no published data on the use of CSR in assessing psychiatric residents conducting psychotherapy; however, this is one of the more promising techniques for both teaching and evaluation.
Checklist Evaluation
Checklists consist of a list of essential or desired specific behaviors, specific activities or steps that make up a more complex skill. Research supports the usefulness and reliability (0.7 to 0.8) of checklists for evaluation of patient care skills (history taking) and interpersonal and communication skills when directly observed by a trained rater and completed at the time of observation. Checklists are developed by expert consensus. Criteria for evaluating performance benefit from descriptive "anchor points."
Earlier, we presented several lists of skills believed to be essential for brief psychotherapy. While some simple behaviors may lend themselves to checklists, many behaviors, such as the ability to establish rapport or develop a therapeutic alliance, represent complex abilities that are difficult to further reduce into component skills. Therefore, we may be required to rate certain complex behaviors as gestalts, relying on overall pattern recognition ("we recognize it when we see it"). If carefully constructed and tested for validity and reliability, checklists can be effective tools for evaluating elements of live or recorded trainee therapy sessions (see "Portfolios").
Global Rating of Live or Recorded Performance
Global rating forms are distinguished from other rating forms in that (a) a rater judges general categories of ability (i.e., interpersonal and communication skills) instead of specific skills, tasks or behaviors; and (b) the ratings are completed retrospectively based on general impressions collected over a period of time, such as at the end of a clinical rotation, and are derived from multiple sources of information (e.g., direct observations or interactions, input from other faculty, trainees or patients, review of work products or written materials). Rating scales are numeric but qualitative indicators (e.g., outstanding = 1, good = 2, fair = 3, poor = 4). Written comments allow evaluators to explain their ratings. Global ratings of patient interviews directly observed by the examiners are used in the oral exam portion of the American Board of Psychiatry and Neurology certification process and as a part of many training programs.
Several problems limit the value of global ratings: 1) scores can be highly subjective, especially with untrained raters; 2) sometimes raters may grade all skills at the same level, even when performance in individual categories is variable; and 3) scores may be additionally biased when raters make inappropriately severe or lenient judgments or avoid using the extreme ends of the rating scales. On the other hand, global rating forms can be easily constructed and completed quickly. Reliability and validity improve when standards are created and "anchors" for each point on the scale are provided (i.e., examples of behaviors or attitudes).
Portfolios
A portfolio is a collection of products prepared by the trainee that provides evidence of learning and achievement related to training goals. It typically contains written documents, such as logs and transcripts but can also include video or audio recordings. It can include statements about what has been learned, statements about application, and statements about remaining learning needs and how they can be met. In general psychiatry training, a portfolio might include a log of diagnoses treated, a log of therapies used, a summary of the research literature reviewed in selecting a treatment option, a quality improvement project, ethical dilemmas faced and how they were handled, and recordings or transcripts of actual therapy provided to patients. The contents of a portfolio do not have to be standardized since the purpose is to demonstrate individual learning gains relative to individual goals. A portfolio is also one of the best tools for combining teaching with assessing continuity-of-care concerns that are central to psychotherapy. A portfolio for psychotherapy might include sample transcripts or tape segments demonstrating transference, resistance, countertransference, patients' dysfunctional thought records or cognitive distortions, vignettes describing relationships between past experiences and a patient's current symptoms, and examples of finding a focus with patients in brief therapy. Developing protocols for assessing such portfolios would be crucial to the reliability of such assessments.
Simulations and Models
Simulations used for assessment of clinical performance closely imitate reality. They incorporate a wide array of options resembling reality, allow examinees to reason through a clinical problem, permit examinees to make life threatening errors without hurting a real patient, provide instant feedback so examinees can correct a mistake in action, and can be used to rate examinees' performances on clinical problems that are difficult to evaluate effectively in other circumstances. Simulation formats have been developed as paper and pencil branching problems (patient management problems or PMPs), computerized versions of PMPs called clinical case simulations (CCX), role playing situations (i.e., standardized patients (SP)), clinical team simulations, and combinations of all three formats. Scoring rules are set by experts.
To build a simulation, clinical experts craft scenarios from real patient cases to focus on specific skills. Technical experts then create scripts for standardized patients or computer based simulations and add, when feasible, automated scoring rules. Simulations are expensive to create. To our knowledge, none is currently available for assessing the nuances of psychotherapy. However, simple clinical vignettes to assess the recognition and handling of psychotherapeutic phenomena have been incorporated into the Columbia Psychodynamic Psychotherapy Skills Test (an MCQ). We expect that the rich literature on simulation and gaming (19–21) will power the development of sophisticated and useful computerized simulations of the more subtle aspects of therapy in the future.
|
Toolbox Methods Less Applicable To Psychotherapy Competence Assessment
|
Procedure or Case Logs
Case logs document each patient encounter by setting, patient characteristics, diagnosis, and/or type of therapy. While case logs are a useful method to summarize a trainee's breadth of experience they offer little help in assessing skill competence.
Record Review
While reviews of patient records using standardized protocols may yield information about presenting symptoms, diagnoses, type of therapy performed and patient outcomes, clinical records do not easily lend themselves to the assessment of competence in psychotherapy.
Written Examination (MCQ)
The familiar printed or computer based multiple-choice questionnaire (MCQ) examination is designed to sample recollection of facts and knowledge but also to evaluate a candidate's understanding of the subject. MCQ examinations are universally used for in-training examinations and for initial certification. While useful for testing factual information about psychotherapies, the MCQ cannot evaluate subtle interactions and cannot be relied upon as a sole method to assess competence in psychotherapy. The Columbia Psychotherapy Skills Test offers an interesting and still evolving contribution.
Standardized Patients
Standardized patients (SPs) are healthy individuals or actual patients who are trained to simulate a psychological or psychiatric condition in a standardized way. The trainee examines and deals with the SP as if (s)he were an actual patient. The standardized patient approach is not likely to be useful for assessing competence in psychotherapy except, perhaps, very early in training, when it can double as a teaching tool.
Standardized Oral Examination
The standardized oral examination uses information from realistic patient cases and a trained examiner, who presents a clinical problem and then asks the examinee to manage the case. While many medical specialty boards use standardized oral examinations as the final examination for initial certification, this method has not been developed to adequately assess issues in ongoing care and psychotherapy.
Objective Structured Clinical Examination (OSCE)
In an OSCE, one or more assessment tools are administered at 12 to 20 separate standardized patient encounter stations, each station lasting 10—15 minutes. Widely used for training in the US and licensure in Canada, OSCEs are limited by the skill of the actors in their ability to assess issues pertinent to continuity of care and psychotherapy skills.
|
APPLICATIONS: THE UNIVERSITY OF MISSOURI AS A CASE EXAMPLE
|
Implementing a realistic set of competency assessment procedures in the course of a busy residency is a formidable challenge, but not impossible to do. No program has yet developed a gold standard, but several vigorous experiments are underway. The extent to which individual components and/or total evaluation packages work remains to be seen. To illustrate one substantial work in progress, the following program, instituted over the past several years, has utilized a number of the general strategies described in the toolbox, with specific attention to assessing the core psychotherapy skills valued and championed by the faculty. To demonstrate their psychotherapy competence trainees at the University of Missouri are required to develop and present a portfolio that includes: 1) a counseling self-estimate inventory (COSE), a 37-item Likert-scaled questionnaire about the trainee's attitudes and skills, rated every 3 months; 2) both a trainee and patient version of the working alliance inventory, a 12-item rating scale assessing the state of the therapeutic alliance; 3) two analyses of psychotherapy sessions in which the trainee categorizes each intervention as to mode of response as well as her/his own intention; these analyses are then compared with later sessions for the variety of kinds of interventions; 4) description of five boundary violations with specific cases using an "exploitation index"; 5) a vignette from the trainee's own cases in which a stressor became the focus of brief therapy and a vignette of psychodynamic as well as cognitive behavioral patterns evident in the same case; 6) examples from the trainee's own practice describing transference, resistance and countertransference; 7) copies of two dysfunctional thought records from the trainee's own patients and two examples illustrating cognitive distortion to display use of cognitive therapy; 8) two brief descriptions of the relationship between past experiences and current difficulties to demonstrate the trainee's understanding of psychodynamics; 9) two examples of the trainee finding a focus with two patients in brief therapy; 10) global evaluations of trainee change (a single-item global rating scale from "very poor" to "excellent") by supervisors every three or six months; 11) two vignettes in which the supervisor stops the tape or transcript and the trainee provides an empathic summary of what the patient has just said to assess ability for supportive psychotherapy.
While this portfolio is ambitious, it still does not require exhaustive assessments of every skill believed to be important for competent psychotherapy that is listed earlier in this chapter. However, it seems likely that any trainee completing these portfolio requirements satisfactorily will be at least minimally competent at what are generally recognized to be important, if not essential, concepts in psychotherapy and will, of necessity, have learned a great deal in the process.
|
CONCLUSIONS
|
Major obstacles in assessing competency in any of the psychotherapies include the fundamental difficulties of identifying and assessing the complexities of repeated interpersonal interactions with the same person over a period of time. Moreover, there may well be several effective ways to tackle any given clinical problem, and knowledge about specific, definable therapist behaviors which predictably lead to desired outcomes in specific contexts is lacking. Given these substantial theoretical and practical problems, strategies using global rather than microscopic methods of evaluating competency in psychotherapy may be more realistically achievable. Such methods rely on supervisors' abilities to recognize competence as they see it with actual patients in treatment, even though they may not be able to completely specify all of its component parts in every instance. The more these methods can be tied to actual patient outcomes, the better our accountability. Thus we see the use of checklists and/or global ratings of live interviews as the major component in assessing professional psychotherapy competence in training programs. Recorded interviews can provide added flexibility for both learning and evaluation. Cross-sectional exams have limited ability to assess the skills necessary for competence in the processes of repeated contacts with a given patient over time central to ongoing psychotherapy.
At present, an achievable psychotherapy assessment program might include 1) sequential tests of knowledge and understanding using a MCQ-based exam that incorporates patient vignettes such as employed by the Columbia Psychotherapy Skills Test and 2) use of a portfolio of recordings or direct observations of psychotherapy sessions in which the technical skills being assessed are displayed. Ideally, recordings can be obtained with patients for whom 360-degree ratings or patient surveys show significant patient improvement as well as patients who don't improve. In this way trainees may benefit from multiple sources of direct feedback, while evaluators see the trainees using techniques which actually produce or fail to produce therapeutic change in a specific individual.
As controlled trials inform us about which specific psychotherapies are most effective for which clinical problems, and which specific behaviors and techniques are essential for those psychotherapies, our expectations of trainees will become more refined. Future developments may also foster the construction of sophisticated computer models of psychotherapeutic situations which call on those essential skills. In the meantime, we will likely be best served by a broad-based portfolio containing MCQs, global ratings of ongoing supervision, recorded samples that demonstrate specific technical skills, and finally, by tolerating the ambiguity that has been integral to our profession from its inception.
|
ACKNOWLEDGMENTS
|
The authors thank Liz Wilding, Ed.D. for her help in extracting ideas and transforming them into words.
|
REFERENCES
|
- ACGME Outcome Project: ACGME General Competencies Version 1.3 (9.28.99), 2000
- ACGME Program Requirements for Residency Training in Psychiatry (effective January, 2001), 2002
- Lambert J, Bergin AE: The effectiveness of psychotherapy, in Handbook of Psychotherapy and Behavior Change. Edited by Bergin AE, Garfield SL. New York, Wiley, 1994, pp 143–189
- Klerman GL, Weissman MM, Markowitz JC, Glick I, Wilner PJ, Mason B, Shear K: Medication and Psychotherapy, in Handbook of Psychotherapy and Behavior Change. Edited by Bergin AE, Garfield SL. New York, Wiley, 1994, pp 734–783
- Hill CE, Helm JE, Spiegel SB, Tichenor V: Development of a system for categorizing client reactions to therapist interventions. J Counseling Psychology 1988; 35:27–36
- Gelso C, Carter JA: The relationship in counseling and psychotherapy: components, consequences and theoretical antecedents. Counseling Psychologist 1985; 13:155–243[Abstract]
- Horvath AO, Greenberg LS: Development and validation of the working alliance. J Consulting Psychology 1989; 36:223–233
- Moras K, Strupp HH: Pretherapy interpersonal relations, patient's alliance and outcome of brief therapy. Arch Genl Psychiatry 1982; 39:405–409[Abstract]
- Foreman SA, Marmar CR: Therapist actions that address initially poor therapeutic alliance in psychotherapy. Am J Psychiatry 1985; 142:922–926[Abstract/Free Full Text]
- Kivlighan DM: Relationship between counselor use of intentions and clients perception of working alliance. J Counseling Psychology 1990; 37:27–32[CrossRef]
- Kivlighan DM, Schmitz PJ: Counselor technical activity in cases with improving working alliances and continuing-poor working alliance. J Counseling Psychology 1992; 39:32–38[CrossRef]
- Lazarus AA: The utility and futility of combining treatments in psychotherapy. Clinical Psychology - Science and Practice 1996; 3:59–68
- Beutler L, Consoli A: Systemic eclectic psychotherapy, in Handbook of Psychotherapy Integration. Edited by Norcross J, Goldfried MR. New York, Basic Books, 1992
- Bienenfeld D, Klykylo W, Knapp V: Process and product: Development of competency-based measures for psychiatry residency. Acad Psychiatry 2000; 24:68–76[Abstract/Free Full Text]
- Mellman L, Beresin E, co-chairs AADPRT Psychotherapy Task Force: David Goldberg, Ron Reider, Ron Krasner and Lisa Mellman: Psychotherapy competencies. Further refined by Carol Bernstein and NYU faculty and Hinda Dubin and University of Maryland faculty. Received from aadprt-list@aadprt.org, July 3, 2000
- Beitman B, Yue D: Learning Psychotherapy, New York, W W Norton & Company, 1999
- Dreyfus HL, Dreyfus SE: Mind Over Machine: the Power of Human Intuition and Expertise in the Era of the Computer. Free Press, 1986; as cited by Leach D: Competencies in graduate medical education. Keynote address to AADPRT annual meeting, March 1, 2001
- ACGME Outcome Project: Toolbox of Assessment Methods, Version 1.1, September, 2000
- Meyers H, Dorsey K, Benz, E: DxR Patient Simulation Software. DxR Development Group, Copyright 1992–9. Available at http://www.dxrgroup.com
- Satish U, Streufert S, Barach, P: Assessing and improving medical competency: using strategic management simulations. Simulation and Gaming 2001; 32:156–163
- Streufert S, Satish U, Barach P: Improving medical care: the use of simulation technology. Simulation and Gaming 2001; 32:164–174
This article has been cited by other articles:
|
|
|
|
 
J. T. Frederick and K. A. Comtois
Practice of Dialectical Behavior Therapy After Psychiatry Residency
Acad Psychiatry,
February 1, 2006;
30(1):
63 - 68.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
K. A. Khurshid, J. I. Bennett, S. Vicari, K. L. Lee, and K. E. Broquet
Residency Programs and Psychotherapy Competencies: A Survey of Chief Residents
Acad Psychiatry,
December 1, 2005;
29(5):
452 - 458.
[Abstract]
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2003
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
