In 2007, the American Board of Psychiatry and Neurology (ABPN) announced that the live patient interview would be phased out of the ABPN certification examination (1). In its place, the ABPN now requires that residency programs conduct Clinical Skills Verification (CSV) for each resident during training. Through a series of directly-observed clinical encounters, residency programs must document residents’ competency in 1) the physician–patient relationship; 2) conduct of an interview and mental status examination; and 3) case presentation. The Psychiatry Residency Review Committee (RRC) of the Accreditation Council for Graduate Medical Education (ACGME) soon gave its endorsement, requiring that residents not only participate in these assessments, but pass them as a condition of graduation (2).
To facilitate implementation of these requirements, the ABPN convened a Task Force on Clinical Skills Verification Rater Training, with representation from a wide variety of professional psychiatric organizations (see Acknowledgments section). This group was charged with preparing training materials for faculty and programs to guide them in the CSV process. A series of faculty training modules developed by the group are available on the American Association of Directors of Psychiatric Residency Training (AADPRT) website (3). This report briefly describes the recommendations of the task force for faculty training and program implementation.
The ACGME focus on specific competencies and how they are assessed has become a major driver of change in medical education over the past decade, and has led to extensive discussion of how to determine and document competency throughout training. Factual knowledge is easily demonstrated with a well-constructed multiple-choice examination, but assessment of the clinical skills required in real-world practice is more difficult to accomplish. The CSV is designed to allow residents to demonstrate three of the most basic of these skills: rapport-building, information-gathering, and clinical communication, by direct faculty observation. Although a variety of additional skills are essential for residency completion, ABPN certification, and competent clinical practice, these three were identified by the ABPN as requiring direct observation of clinical encounters for adequate assessment. The following recommendations are intended to maximize the value of the CSV, not only as an assessment tool, but especially as an educational process.
One traditional model for observed clinical interviews is the “mock Board” examination, in which a patient is selected in advance for a scheduled encounter, and one or more faculty members observe a time-limited interview during which the trainee is evaluated on the essential elements of a clinical evaluation. Because it has been used for many years to meet the RRC requirement that programs conduct periodic clinical assessments, the mock Board has the primary advantage of familiarity to faculty. It provides a measure of standardization to the evaluation by limiting the range of patients that might be involved and by controlling the interview environment. Also, some programs exchange residents for these evaluations so as to provide greater objectivity.
The mock Board has a number of disadvantages, however, that make it less than ideal for CSV. Patterned after the ABPN live-patient examination that the CSV replaces, it perpetuates a number of problems that CSV has the potential to address. A major problem is that patient selection tends to be narrow and limited to patients who are intact enough to consent and cooperate, available to interview, and willing to participate—qualities not always observed in the patients routinely seen by psychiatrists in other clinical settings. Equally problematic is the use of arbitrary time-limits (e.g., 30 minutes) on the interviews, altering the nature of the interview and limiting the validity of the assessment of residents’ clinical skills in other less structured settings. In many programs, the mock Board is offered as infrequently as once a year, providing an inadequate sample size to document competence. Finally, when the mock Board is focused primarily on pass/fail assessment, there is low potential for resident education. Although the mock-Board format can be used to meet ABPN and RRC requirements, its disadvantages as an educational tool argue against it.
As an alternative, CSV readily lends itself to an interviewing workshop or class. Like the mock Board, this involves prearranged patients and time-limited encounters, but it has several advantages over the mock Board. The most important of these is the opportunity for other residents to watch the interview and participate in a detailed discussion of the patient encounter as it occurs and, later, with the resident interviewer. The patient sessions may be video-recorded to allow the resident conducting the interview to observe his or her own style and to more actively take part in the later discussion. Residents often cite these workshops as uniquely helpful to their professional development (4–7).
Although the workshop format has much educational value as a formative exercise, its evaluative function is less acceptable. It is both time- and labor-intensive for faculty, who typically must have time allotted by their departments for the activity to occur. It is intensely anxiety-provoking for residents to be observed by faculty and peers, which may affect the quality of the interview. Residents in the workshop have many opportunities to observe and comment on interviews, but few opportunities to conduct an interview. The range of patients willing to participate is even narrower than for mock Boards, because of the increased demands of multiple observers and video-recording. The workshop format is thus an important component of observational evaluation, but should not be the only experience residents have for these encounters nor the only source of information that programs gather about resident performance.
A more practical and widely applicable model is to embed these assessments in the routines of clinical care across a variety of settings. Appropriate naturalistic environments include inpatient units, outpatient clinics, consultation/liaison services, and emergency rooms. Residents routinely see new patients in each of these settings and are expected to present the cases to faculty, who are often required to personally see the patients, as well. Because the assessments are done on actual clinical encounters, there is no need to arrange patients in advance, and no consent is required. Scheduling is flexible; encounters may be planned ahead of time or as the opportunity arises.
There are numerous advantages to the embedded clinical model. Residents can be evaluated and receive feedback on a wide range of patient and interview types, many of which could not be included otherwise, such as interviews with acutely psychotic, intoxicated, or uncooperative patients, who form an important part of clinical practice. Faculty time is thus used efficiently to meet both clinical and educational goals. Since most of these encounters involve supervising faculty, the observed interviews may be used to inform overall clinical evaluations of residents, improving their quality and accuracy. Most importantly, evaluation is not isolated from clinical care. Both formative and summative feedback derived from these encounters are based on real-time patient interactions, rather than performance on an examination, providing a window into what the resident actually does in clinical practice.
Disadvantages of this model include a reduction in standardization and the need to train and motivate a large group of faculty. Limited standardization is best overcome by programs providing a large number of these evaluations with as many different patients and faculty as possible (8–10). Faculty participation may be harder to motivate, as this involves a significant change from common educational practice. Faculty training, incentives, monitoring, and accountability are essential to ensure adequate and consistent involvement. Finally, because the patient must be unknown to the resident for the CSV interview, residents may not review the patient’s chart before the interview. To avoid any deficit that this may introduce in clinical care, the faculty member can review those notes before the encounter.
The traditional 30-minute time-limit on Board or mock Board examination interviews was chosen for examiner convenience and is at variance with initial interviews residents are taught to conduct in most clinical settings (11, 12). A more useful standard is to allow the trainee whatever amount of time is appropriate to the clinical situation—shorter interviews (30–45 min) in acute settings, and longer ones (45–60 min) when a more detailed formulation and the establishment of a therapeutic relationship are desirable. Longer interviews are more likely to be complete and representative, but are less efficient for faculty and may prove to be a disincentive to their regular participation in CSV assessments. An AADPRT work-group has recommended 30–45 minutes as a reasonable balance of these considerations (13), but longer encounters may be appropriate in some cases.
The ABPN requires that residents be informed of their scores on the CSV as a mode of summative feedback. Although the process does not require that additional information be provided, good educational practice includes prompt, focused, specific, and constructive feedback, with the goal of improving future performance (4). Without objective feedback, learners tend to rely on self-assessment, which may be unreliable even in conscientious residents (5, 6). Residents benefit, however, from both positive and negative faculty assessments (7). The sharing of this information provides an opportunity for constructive engagement between faculty and trainees after all types of clinical encounters, including the CSV.
+
Number and Frequency of CSV Assessments
The reliability of observational evaluations is greatly enhanced by repeated measures involving different patients and different faculty. Studies of observed clinical encounters for internal medicine (8) and surgery (9) residents found that 8–12 assessments were required to consistently identify trainees performing at acceptable or unacceptable levels (10). Also, the number of assessments proved to be far more important than the number of items assessed. That is, frequent evaluations of a few global items were more effective than occasional ratings of many items.
Although the ABPN requires only three satisfactory CSV assessments, there is no limit to the number a program may choose to do as part of their own evaluation of a resident’s progress through the program or readiness for graduation (11). Programs should consider doing several evaluations in a variety of settings each year to ensure reliable assessment of residents’ skills and progression through training. Early assessments are particularly valuable in identifying residents with deficiencies in these areas and helping faculty to make effective educational interventions, thus avoiding the problem of a resident approaching graduation before such problems come to light. Because these are fundamental skills that can be learned early in training, a resident who does not show competency in these areas or clear growth toward that competency by the end of PGY-2 may be an appropriate candidate for remedial training.
In addition to improving the reliability of summative feedback, frequent faculty observation of resident interviews provides more opportunities for the formative feedback that directly shapes resident behavior. Residents see large numbers of patients during training, but that practice is only beneficial when good habits are reinforced and problems are identified and corrected early in the process. Frequent observation allows the feedback to focus on a small number of specific issues, which the resident can promptly address. Observation by different faculty broadens the range of feedback available to the resident and may help the trainee develop a more complete set of interviewing skills. Initiation of residents to CSV as early as PGY-1 and frequent repetition through training are recommended to maximize its usefulness as a formative tool.
The passing score for the evaluation, set by the ABPN as 5 on an 8-point scale, represents the minimal acceptable standard on that item for a general psychiatrist practicing in the community (11). An excellent or outstanding performance is not required to pass, and should be recognized with a higher score. Competency standards should not be adjusted to the resident’s level of training, irrespective of when in training the evaluation is performed. Although the assessment may take place in a specialty setting (e.g., the geropsychiatry clinic), the passing standard is that of general psychiatrist, not a subspecialist.
A critical element of trainee assessment is to identify fundamental performance dimensions and to define specific criteria for each of these dimensions. The ABPN has defined the fundamental areas of physician–patient relationship, conduct of interviews, and presentation of cases, but has not given specifics as to their essential elements. Consequently, this has been the subject of extensive discussion among psychiatry training directors and other educators. Specifically, what behaviors, interactions, or elements must be present or absent to define a competent performance?
In our experience so far with groups of program directors and other academic psychiatrists rating recorded resident interview vignettes as a group, some aspects of performance appear to have a higher level of consensus than others. There appears to be broad agreement that the trainee should be able to assess suicidal risk and substance abuse, follow historical cues offered by the patient, and respond to the patient’s affective changes during the interview. There is disagreement, however, as to whether the resident must elicit the patient’s story (as opposed to gathering information about symptoms) to achieve the minimal competency threshold, although there is high agreement that an outstanding psychiatrist does this. Similarly, there is incomplete consensus regarding the level of formal cognitive testing that is necessary for a patient who exhibits no evidence of cognitive impairment on interview.
The evaluation must be recorded on an ABPN-approved form, of which several types are available. Two of the forms can be downloaded from the ABPN website (11). Three other approved forms are available on the AADPRT website (3). Several of these forms include subscores and anchors to assist evaluators in their assessments. These aids represent an attempt to specify the exact elements comprising each global score. The subscores are not required by the ABPN or RRC. Programs may use any of these or may submit their own forms to the ABPN for approval. CSV forms must be maintained in resident files for review by accreditation bodies or the ABPN.
Despite the extensive supervision requirements for psychiatry residents demanded by the RRC, actual observation of patient care by faculty supervisors is rare (12). In part, this arises from a long tradition of off-site supervision, consisting of the trainee’s recollections of and reflections on a clinical encounter as an accurate representation of what actually occurred. At another level, however, are the problems of inadequate faculty training in what to observe during an interview, how to determine whether it meets an objective standard of professional acceptability, and how to present that information to the resident in a constructive way.
The frequency with which faculty observe and systematically assess clinical encounters (13) and the quality of those assessments (14) are related to the level of training and skill that faculty members have in standardized observation and evaluation. Several techniques to improve faculty performance have been validated (15). The first of these, behavioral observation training, begins with the expectation that faculty will increase the number of observations they make of trainees. Observational aids, such as a checklist, behavioral diary, or log of observed encounters, are provided to assist the faculty member in recording key behaviors. Evaluators are encouraged to prepare for the observation session, identifying specific goals, arranging the room ahead of time, and determining what type of record will be made.
A critical element for good learner assessment is for faculty to identify specific performance dimensions and define exact criteria for each of these. The fundamental dimensions of the CSV assessment have been defined by the ABPN, and several of the CSV forms developed by AADPRT include suggested sub-items and performance anchors to assist faculty in this area (3). It is recommended that the sub-items and anchors be used during faculty training as a starting-point for discussion and consensus-building among faculty regarding expectations of residents.
As was demonstrated in the ABPN live-patient examination, it is possible to maintain a reasonable level of interrater reliability among trained observers evaluating a single interview (16, 17). To achieve this, faculty raters require practice, familiarity with the rating form selected by the program, and regular feedback regarding how their ratings compare with those of other faculty. It is recommended that faculty gather periodically to observe, rate, and discuss sample interviews in an attempt to reach consensus. Alternatively, although less useful than an active discussion with peers, faculty may compare their own ratings of interview-videos with standardized groups of expert educators. Several such videos, together with their consensus ratings, are available on the AADPRT website for this purpose (3).
Several other measures will contribute to high-quality faculty assessments. First, because most faculty members are accustomed to assessing residents according to a relative standard, based on their level of training, the cognitive shift to an absolute standard (a competent practicing psychiatrist in the community) required for the CSV needs to be regularly reinforced. Second, interrater reliability is improved when all faculty raters within a program use the same form consistently and are familiar and comfortable with the instrument (18). Finally, in larger programs, where many CSVs are being conducted, it may be useful to compare aggregate ratings of individual faculty members with those of peers.
Finally, faculty often lack experience and skill in giving feedback, and benefit from training in how to present constructive compliments and criticisms (7). Faculty must understand the profound educational value of feedback and their duty as teachers to provide it. They need to gain skill in setting the proper climate, framing their comments, focusing their observations and recommendations on a small number of correctable issues, and eliciting reciprocal feedback from trainees.
The transition from the live-patient interview as part of the ABPN Certification Examination to the in-training CSV process provides the profession with a rare opportunity to evaluate the interviewing skills we teach and how we ensure that residents have mastered them. This is an ideal context for psychiatry educators to discuss and implement the standards that should characterize patient interactions for the next generation of psychiatrists. From a pedagogic perspective, the introduction of direct observation of residents’ clinical skills by faculty will make possible high-quality formative and summative feedback across training sites, providing a wonderful new teaching opportunity. Finally, CSV is well suited to interrater reliability and validity studies that have been all-too-rare in psychiatric education assessment.
Current efforts are focused on faculty training in direct observation and use of the prescribed rating scales. This is expected to move the field toward greater standardization of assessments and has already contributed to discussion among training directors and other educators regarding the appropriate skill-level to be expected for residency graduates. Whether this should evolve into a specific certification process for faculty evaluators remains to be determined. Interest in training has been widespread, however, and extends to such topics as presentation of feedback, remediation options for residents, and exploration of additional clinical settings for the assessments.
The challenge that we face as educators is to take full advantage of this opportunity to enhance our teaching of clinical skills and to ensure that CSV not be relegated to the status of a minimal requirement that must be met by residents and programs. The full benefits of the CSV will be realized to the degree that program directors and teaching faculty understand its potential, incorporate its principles into clinical supervision, and use it in collaboration with residents to develop essential clinical skills. We look forward to further progress in each of these areas.
The ABPN Task Force on Clinical Skills Verification Rater Training consisted of Karen Broquet, M.D.; Michael Jibson, M.D., Ph.D., co-chairs; Nyapati Rao, M.D.; Richard Summers, M.D., American Association of Directors of Psychiatry Residency Training (AADPRT); Joan Anzia, M.D.; Karen Broquet, M.D., Association for Academic Psychiatry (AAP); Eugene Beresin, M.D.; Jeffrey Hunt, M.D., American Academy of Child and Adolescent Psychiatry (AACAP); William Lawson, M.D.; Daniel Wilson, M.D., American Association of Chairs of Departments of Psychiatry (AACDP); David Kaye, M.D.; Anthony Rostain, M.D., American College of Psychiatry (ACP); Larry Faulkner, M.D.; Barbara Schneidman, M.D., American Board of Psychiatry and Neurology (ABPN); Deborah Hales, M.D., Sandra Sexson, M.D., American Psychiatric Association (APA).