With the American Board of Psychiatry and Neurology (ABPN) decision to end the live patient examination as a requirement for certification in psychiatry, it has become the responsibility of residency programs to attest to candidates' competence in 1) establishing a relationship (i.e., rapport) with a patient; 2) conducting an interview and mental status examination; and 3) presenting the case (1). The Clinical Skills Verification (CSV) process now required for ABPN certification is prescribed by only a few parameters: the resident must perform at an acceptable level in each of the three basic domains of evaluation during three separate assessments involving interviews of “about 30 minutes” with patients unknown to them, observed by an ABPN-certified psychiatrist, with no more than two assessments done by the same rater (2). The ABPN has left broad leeway for programs to choose patients, settings, numbers of evaluations, and faculty for these assessments, and has specifically invited programs to use them for other aspects of resident evaluation (2).

In our program, we view the CSV process as an opportunity to reinforce important aspects of the supervisory experience. We have made it an explicit expectation for faculty and residents, as supervisors and trainees, to participate in regular CSV assessments throughout residency, and we have worked to develop consistency in the faculty's evaluation of the clinical skills of residents. We have also begun to use CSV as one source of data to evaluate residents' attainment of pertinent core competencies (e.g., Patient Care, Professionalism, Interpersonal Skills, and Communication). In this way, CSV is similar to an Objective Structured Clinical Examination (OSCE) (3), although with some important differences. The CSV offers a way to evaluate residents at the highest level of Miller's pyramid of skill proficiency (4). Unlike prearranged, high-stakes clinical assessments (e.g., mock board exams, standardized patients), where trainees can, at best, “show how” they interact with patients, the incorporation of CSV into regular clinical duties, without the uniform time restrictions of a mock board exam or the artificial context of a standardized patient, allows residents to show what they “do” during regular clinical duties. The effect of observation on performance notwithstanding (5), the CSV, properly implemented, allows faculty to see residents do a new patient evaluation. Specifically, by integrating the CSV into the regular flow of clinical activity, we leverage a variety of clinical contacts in different settings to conduct a substantially more meaningful evaluation of how residents interact with patients, collect, and interpret clinical data. Indeed, a spectrum of teaching moments is available when the CSV is incorporated into the residency program from the beginning of training. Used in this way, the CSV functions as one type of “in-training assessment” (6), and can provide an opportunity for early recognition of the need for remediation of key clinical skills. The following points detail our conceptual and procedural approach.

Direct observation of clinical work is an excellent opportunity to provide formative feedback to residents. With regular and proper use, random sampling of clinical work can help develop, enhance, and solidify good clinical skills and highlight the need for remediation. In 2002, we introduced the Clinical Work Sampling Program for residents, medical students, and other trainees (e.g., in social work, nursing, and psychology). This program required faculty to periodically observe some clinical interaction of the trainee with a patient. It was not required that the interview be a first contact with a new patient, or even a formal history and assessment. The goal was to increase the frequency of real-time observation of a variety of clinical interactions, with immediate feedback to the trainee about his or her interview style, formal presentation skills, diagnostic assessment, formulation, and therapeutic strategies. Not all items were necessarily applicable to all clinical contacts. A daily assessment of an inpatient might only be rated on the interview and presentation, but could also be used to evaluate the trainee's assessment of the patient's current status and the therapeutic impact of the ongoing treatment plan. Multiple assessments over time provide a clear signal about the trainee's performance; the feedback provides them with concrete items on which to focus as they hone and improve their clinical skills (7). These become another set of data-points for faculty to review in formulating the summary assessment of overall competence at the end of a rotation, clerkship, etc. Clinical Work Sampling quickly became a critical component of the training experience, valued by faculty and trainees alike.

The CSV is a somewhat more elaborate example of Clinical Work Sampling. As with clinical work sampling, we embed the CSV into clinical care activities rather than setting up a staged interview (e.g., mock board exam). This can readily be done on inpatient units, Psychiatric Emergency Service (PES), ambulatory clinics, or any setting where a new patient is being seen by a resident for the first time. The attending physician notifies the resident when a new patient contact will be a CSV experience, and the resident conducts the interview under direct supervision. On our inpatient unit and in PES, the attending is in the room. In our ambulatory clinic, the attending may be in the room or may observe through a one-way mirror (in our institution, the latter requires the patient's written consent). In all cases, the resident is instructed to conduct an initial evaluation appropriate for the setting. We do not explicitly inform the patient that a formal resident evaluation is taking place, but do indicate that a faculty supervisor will be observing the interview to provide feedback to the resident, as well as to participate in the care of the patient. No patient has refused to participate in the subsequent clinical contact.

CSV is best done early and often. Beginning in the 2008–2009 academic year, we incorporated the CSV into our Intern Evaluation Clinic. This clinical experience already included periodic observation of interns' new-patient evaluations and easily incorporated the CSV. We made the training decision not to count any PGY I assessments toward the board certification requirement. In this way, we made explicit to interns and observing faculty that we do not expect interns to be proficient in the skills assessed on the CSV during their internship year. As a result, the experience is no longer seen by interns or faculty as an examination, but exclusively as an opportunity to learn and receive feedback. Thus, attending faculty are “given permission” to be more constructively critical than they might otherwise be if the assessment “counted” toward eligibility for board certification and a lower score signified a failed performance rather than a developmental stage (8).

Our goal is to obtain 6–12 CSVs for each resident over the course of each training year, guided by the evidence that 8–12 clinical assessments provide an optimal estimate of resident performance with tight confidence intervals around the average score (9). Additional assessments do not appear to add substantially more information. Fewer assessments decrease the reliability of the performance estimate. However, the evidence suggests that a range of 4–6 evaluations a year provides a reasonably reliable estimate of performance and may be a more realistic goal, particularly during the initial implementation of the CSV process we describe (9).

During the first year of implementation, after a single faculty training, the frequency of assessments was well below our goal. Among the PGY I cohort that year, 9 CSVs were conducted for 7 residents, 3 of which were at or above the ABPN “acceptable” standard, 4 below that standard, and 2 incompletely scored. None of these residents, however, was assessed to have performed at a level unacceptable for a first-year resident. In PGY II, 9 residents had 22 CSVs, 15 meeting the acceptable standard, and 7 below that. Among PGY III and IV residents (who will be required to take the live-patient examination for ABPN certification), 5 CSVs were deemed acceptable and 2 unacceptable, with several more incompletely scored).

Multiple evaluations demonstrate increasing proficiency or highlight the need for remediation. Multiple data-points increase the validity of the overall assessment of clinical competence (7, 9), and produce a clear signal that is reassuring to both the evaluators and those evaluated. It allows proficiency and consistency to be recognized and also facilitates recognition of the need for remediation. Questionable clinical skills become less a matter of hearsay, and are less likely to be allowed to slip by, and left for the training director to address as he or she considers whether to advance or graduate a trainee. In past years, when our supervision and feedback systems were weaker, we had occasions in which questions were raised about the clinical competence of a resident approaching the end of his or her residency, and yet the resident's file was populated with satisfactory, but superficial summary evaluations. Implementation of the CSV as a frequent assessment tool over the course of the residency increases the likelihood that performance problems in clinical skills will be identified early, with time for corrective action well before the end of training.

Competent residents will not necessarily “pass” a CSV assessment on every occasion. Obviously, every patient is different, posing different challenges and issues to address. At the same time, none of us is at the top of our game every day. On occasion, our ability to be proficient may be limited by internal or external circumstances. The CSV is an opportunity to recognize this and consider, with a supportive but constructively critical attending, what could be done to make the best of a difficult clinical situation.

Over the past 2 years, we have conducted a series of three education-focused Grand Rounds reviewing the literature on supervision, teaching faculty the elements of constructive supervision, and providing the message that such supervision is the privilege and responsibility of the faculty to provide and the trainee to seek. Faculty have found interrater reliability training in local and national settings to be extremely constructive. In sharing assessment responses during these training sessions, faculty become aware of their own evaluation biases and move toward agreement about the ways in which evaluations will be made. Although several ABPN-approved versions of the rating form are available, we have used a single form (10) (AADPRT CSV Form 3) throughout the department, to assist with standardization and interrater reliability among faculty. Our faculty has responded positively to the training sessions held thus far.

Implications for Conflict of Interest Among Training Directors

Concerns have been raised that the CSV presents a conflict of interest for training directors, who may be motivated to achieve a high pass-rate as a reflection of program quality or to elevate resident morale. This is particularly true if CSV is seen primarily as a summative evaluation, or merely as an obstacle to ABPN certification. The use of CSV early and often as a source of formative feedback helps to address this issue. Residents who are performing below the expected standard can be identified promptly and given opportunities to improve throughout the course of their training, rather than having deficiencies noted late in residency or even after graduation (as is now the case). It is likely that many struggling residents, with sufficient instruction and feedback, will reach an appropriate level of performance for graduation, and, in cases where that is not possible, they will be identified and transitioned out of training before graduation. This approach to CSV is true to the tenets of our education mission—assisting program directors to produce competent, and usually better than competent, psychiatrists.

Used in this way, the CSV can enhance our efforts to improve clinical supervision, resident training experiences, and the field's ability to consistently produce qualified and competent psychiatrists. We view our implementation process as easily portable to any clinical setting in which trainees, faculty, and patients are present for the clinical encounter. More than simply a way to demonstrate to the ABPN that a resident is prepared to sit for the certification examination, we see the incorporation of CSV into our regular clinical supervision as an especially effective means to ensure that our residents not only “know how” to interview a patient well, but “do” so in their routine work.

Manuscripts authored by an editor of Academic Psychiatry or a member of its editorial or advisory board undergo the same editorial review process, including blinded peer-review, applied to all manuscripts. Also, the editor is recused from any editorial decision-making.