Academic Psychiatry 26:184-186, September 2002
© 2002 Academic Psychiatry
The Hodges Psychiatry OSCE Guide and Emerging Trends in Assessment
Glenn C. Davis, M.D.
Dr. Davis is Dean of the College of Human Medicine, Michigan State University, Lansing, MI, and Director of the American Board of Psychiatry and Neurology.
Key Words: Objective Structured Clinical Examination (OSCE) • Standardized Patients
Hodges' group and the editor of Academic Psychiatry make a major contribution to the advancement of psychiatric education and assessment by the publication of this guide. The public's appetite for greater accountability in the practice of medicine and medical educators' increasing focus on performance measures of competence have grown markedly and converged in the last decade. Although orally administered clinical examinations have been around a long time, this guide by Brian Hodges and colleagues provides a much-needed "how-to" manual that supports psychiatric educators in the creation and maintenance of objective structured clinical examinations (OSCEs).
Although I will spend much of this commentary on the importance of the OSCE in the panoply of needed performance assessments in medical education, I think it is also important to comment on the quality of this manual. First, as it should be, this manual is practical; if you follow its lead, you can create an OSCE program. Conducting an OSCE, at least full-fledged OSCE, is no trivial or inexpensive task. This manual provides enough information and advice to keep expenses to a minimum. Second, the manual is comprehensive, providing information on planning, creating, preparing, conducting, improving, and maintaining an OSCE evaluation process. Lastly, the use of OSCEs in research is briefly discussed.
EMERGING TRENDS IN ACCOUNTABILITY
The dire need for medical educators to attend to the measurement of competence (particularly performance aspects of competence) has emerged in many studies in the last several decades. These studies indicate wide variations in practice; for example, in C-section rates (1), and cite recent reports by the Institute of Medicine that estimate catastrophic medical error rates in hospitals result in tens of thousands of unnecessary deaths a year (2). While most of these studies have focused on practicing physicians and complex multistep processes, the root cause of these problems is a flawed education process. Education must increasingly focus on the assessment of competence and the link between continuous learning (improvement) and the measurement of competence.
Assessment (The Measurement of Competence)
Undergraduate medical education, graduate medical education, and even certification have relied heavily on multiple-choice written examinations for the demonstration of knowledge and, to a lesser extent, competence. Although multiple-choice written exams may be suited for demonstrating the mastery of "facts" and, to a limited extent, competence in the integration of knowledge and medical reasoning, they are poorly suited for indicating competent noncognitive performance, whether that performance is psychomotor or interpersonal. Furthermore, multiple-choice tests are unsatisfactory in evaluating even the most simple aspects of performance, particularly when they require integrative skills.
Let us take the example of closed-chest cardiac resuscitation. Simulations that replicate cardiac arrest give students the actual complex challenge of resuscitation and provide immediate feedback on the likely outcome. This is essential for demonstrating competence in an arena that requires knowledge, psychomotor skills, reasoning, and the integration of all of the above in real time. Question: Would you wish to be resuscitated by someone who merely passed a multiple-choice exam on resuscitation? I think not. You would prefer to be resuscitated by someone who was tested by and passed an annual simulation test.
Assessment of noncognitive skills in medical school and residency training has relied on "attestation." Attestation is the form of assessment in which "experts" (read educators) ensure that a student is competent by "attesting" to that competence. Attesting usually takes the form of observing student performance and agreeing that it is adequate or satisfactory. Generally, attestation has relied on no formal criteria for success, mastery, or demonstration of competence. In the larger sense, a residency or clerkship director attests to the competence and completion of requirements when the resident or student graduates from the rotation or clerkship. Furthermore, a broader-based "attestation" is necessary for a graduating resident to be eligible to sit for specialty boards; the residency director attests to the fact that the resident has satisfactorily completed the requirements for graduation from an Accreditation Council of Graduate Medical Education (ACGME)–accredited program.
The OSCE offers a major advance on "attestation." The OSCE is founded on the notion that there is a set of core competencies and seeks to set up clinical situations in a simulated form that allows for the measurement of these competencies.
Link Between Education and Assessment
OSCE and other simulation tools not only replicate clinical situations and measure performance but are fundamentally linked to education. By this I mean that taking an OSCE educates as well as tests (or at least it should), and this should be an essential element of today's performance measures. Assessment must be linked to improvement of skills.
The failure of the link between assessment and education is evident when one becomes a practicing physician. One of the major challenges facing medical specialty boards as they consider raising the bar for passing recertification exams is that specialty certification has become important in hospital credentialing, in HMOs deciding to place physicians on their panels and, generally speaking, in physician employment or practice. Thus, recertification is a high-stakes examination; passing or failing it has significant consequences.
There are no effective remediation programs available for physicians who fail specialty recertification examinations. The American Board of Medical Specialties (ABMS) and its member boards are seeking to encourage a lifelong education process that involves continuous improvement. Currently, failed candidates study and retake examinations until they pass. This may be effective while recertification involves multiple-choice examinations, but what if examinations become more difficult and require performance (e.g., an OSCE) or include practice outcomes assessment? OSCE can link assessment with education, and it may be a good model for recertification as well.
One such practical application in another profession is an airline pilot's flight simulator. The flight simulator provides a high-stakes assessment but is also a device used to train (and educate). The flight simulator trains and tests to a performance criterion. We need such improvement-related assessment technologies in medicine. The OSCE is a first step toward an assessment-improvement technology.
OSCE and Outcomes
The OSCE is an intermediate step between assessing knowledge and reasoning in a cognitive examination and measuring practice outcomes. An example: a patient with bipolar disorder does not want to know that her/his doctor has passed a written examination on recognition and treatment of that illness. In fact, the patient does not solely wish to know even that the physician has passed a competence-based examination related to bipolar disorder (e.g., OSCE). The patient wants to know that the physician has the "best" outcomes in treating bipolar disorder; that is, the longest periods of remission, shortest episodes, least medications with the least side effects, early return to work, and "best" results on a whole host of other potential morbidities.
Although OSCEs provide competence-based assessment, they are process or proxy measures for outcome. In the end, research will have to demonstrate that performance on OSCEs leads to better outcomes. I say this not to diminish the importance of the OSCE paradigm, but to emphasize that a relationship between OSCE performance and clinical outcome needs to be demonstrated.
OSCE: Reliability and Psychometrics
Hodges and colleagues correctly emphasize the emergence of OSCEs over the past decade as one solution to the extreme problems of reliability presented by oral examinations. Indeed, most specialty boards have abandoned oral examinations in part because of poor reliability. OSCEs, done properly, can enhance reliability substantially.
Validity can be enhanced as well, although there is more debate about OSCEs and validity. Some argue that the standardized/simulated patient differs from the real patient in ways that challenge validity. Others argue that well-scripted (engineered and acted) cases are far more valid measures than a real patient. In my view, an OSCE generally can (and should) offer enough cases strategically selected across the spectrum of competence to be evaluated as valid and reliable measures of such competence.
OSCE and Acuity of Illness
Another advantage of OSCEs is the ability to test performance in acute care situations. For medical and even ethical reasons, testing physician performance in acute, urgent, and semi-urgent situations entails serious problems. With OSCEs, on the other hand, a whole host of competencies relating to the diagnosis and management of acute situations can be evaluated without ethical or safety issues arising. Another advantage is that an OSCE can be used to assess performance in low-frequency but high-stakes situations, such as the diagnosis and management of suicide risk, dangerousness, or dystonic reactions.
CORE COMPETENCE
It should be apparent that one important prerequisite if one is considering developing an OSCE unit is the clear delineation of the range of competencies to be tested. Hodges and colleagues emphasize the development of a blueprint. The blueprint is "the matrix that outlines parameters for the exam: content areas, knowledge, skills and attitudes, station type, and length."
A "core competence movement" has developed in U.S. medical education. The ACGME and the ABMS have been working together to delineate core competencies for all specialties. With the ACGME's attention to core competencies and the insistence that Residency Review Committees evaluate residency programs partly on the basis of core competencies, medical schools have begun to move from "educational objectives" to "core competencies" as a way of organizing the education process. In my view, this core competence movement may create a revolution in undergraduate curriculum that will, for the first time in decades, result in fundamental curriculum changes.
WHERE ARE WE GOING?
Medical education has depended on "attestation" as the means for assuring competence and quality for far too long. Although medicine in the late twentieth century dramatically expanded its scientific knowledge base, it seems to have stalled educationally. The OSCE and core competence movement may drive a clear and innovative education agenda.
It is clear to this writer that although scientific advances captured much of the public's attention over the last half of the twentieth century, the public (government and employers) has increasingly demanded that this advancing knowledge and technology be integrated into physician competence—demonstrated and proven. Physicians are the prototypes of expert workers: burdened with rapidly increasing scientific knowledge requiring interpersonal and technical skills of the highest level, expected to have a broad range of socially desirable attitudes and values, and, above all, expected to have integrated knowledge, skills, and values in the provision of care to the public. Testing whether the educational system has prepared physicians to practice at the highest levels of competence—for the physician's well-being as well as the public's—is vitally important. The OSCE is a humble but necessary step in this direction.
REFERENCES
- Stafford RS: The impact of nonclinical factors on repeat cesarean section. JAMA 1991; 265:59-63[Abstract/Free Full Text]
- Institute of Medicine: To Err Is Human: Building a Safer Health System. National Academy Press, 1999
This article has been cited by other articles:
|
|
|
|
 
N. A. Vaidya
Psychiatry Clerkship Objective Structured Clinical Examination is Here to Stay
Acad Psychiatry,
May 1, 2008;
32(3):
177 - 179.
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2002
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
| 
