Advanced Search

REGULAR | December 01, 2006

Assessment of Clinical Skills Using Simulator Technologies

Malathi Srinivasan; Judith C. Hwang; Daniel West; Peter M. Yellowlees

Academic Psychiatry 2006;30:505-515.

View Article Information

Received February 2, 2006; revised May 1, 2006; accepted May 23, 2006. Drs. Srinivasan, Hwang, West, and Yellowlees are affiliated with the University of California, Davis, School of Medicine, Sacramento, California. Address correspondence to Dr. Srinivasan, 2315 Stockton Boulevard, Sacramento, CA 95817; malathi@ucdavis.edu (e-mail).

Article

Tables

References

text A A A

Abstract

Objective: Simulation technologies are used to assess and teach competencies through the provision of reproducible stimuli. They have exceptional utility in assessing responses to clinical stimuli that occur sporadically or infrequently. In this article, the authors describe the utility of emerging simulation technologies, and discuss critical issues in simulator-based skills assessment and appropriate results analysis. Method: Based on literature search and expert consensus, the authors discuss three simulation technologies: standardized patients and the objective structured clinical examination; the integrated high fidelity mannequin; virtual clinical stations and the objective structured virtual examination. Results: The authors explore the current state of these technologies: uses, cost, limitations, and likely future applications. For instance, tele-standardized patients may test learners’ communication/management approach to challenges during tele-consultation, such as a suicidal patient several hundred miles away. Integrated mannequins may test leadership skills during psychiatric emergencies. Case-based interactive virtual clinical assessment tools may test learners’ decision-making skills or self-reflection. However, these exciting tools must be implemented systematically. Specifically, educators must define the competencies of interest precisely. Appropriate data analysis will generate dependable results, ascribing the correct proportion of outcome variability to individual learner behavior. Careful analysis and utilization of results will allow justification of the costs to major stakeholders. Conclusions: Simulation technologies offer exciting possibilities for skills evaluation and clinical practice improvement. When used creatively and appropriately, they form a useful adjunct in the armament of educators addressing the question, "Is this physician competent?"

Figures in this Article

Over the past two decades, the assessment of physician skills has evolved from paper-based testing of knowledge ("What do you know?" or "How well can you take a test?") to testing of clinical skills using trained actors and simulators (¹). The nexus of emerging technologies and statistical analysis has allowed medical educators to rapidly progress in their understanding of skills evaluation (²). Educators and the public now have the opportunity to ask the critical question "Is this physician competent?" by having physicians demonstrate their abilities in applied, observed settings.

Simulation involves the creation of exercises that are as real as possible. Typically, simulators are used to assess competencies that are difficult to observe or interpret consistently. These situations might show the varying frequency of learner responses to events, involve sensitive or private conversations, or have so many variations in clinical presentation that it is difficult to compare one learner’s responses with another’s. Common competencies assessed by simulator technologies include communication, counseling, crisis management, physical examination, procedures, leadership, teamwork, management of group dynamics, and medical decision-making (¹—⁴). Integration of simulation within other clinical/educational efforts (e.g., electronic portfolios, medical records, and online content) increases its value.

Like other evaluation modalities, simulation may be used for either formative or summative purposes. For instance, the results of tele-standardized patient interactions may be used to feed back information about practice patterns (formative evaluation) or provide remediation and encouragement. Alternatively, the assessment results may be used to determine whether a physician or trainee is competent to practice medicine independently (summative evaluation), such as with the United States Medical Licensing Examination (USMLE) Part 2 Clinical Skills licensing examination (⁵). The information generated by these assessment techniques can be used to provide feedback to the individual about his or her performance, if the relevant behavior is sampled appropriately. The assessment information can also provide institutions with information about the general practice patterns of their physicians and trainees and the effectiveness of curricular efforts.

In this article, we will discuss 1) the utility of three specific simulation technologies; 2) critical issues in skills assessment using simulators; and 3) critical issues for data analysis.

Utility of Three Specific Evaluation Technologies

We chose to discuss three technologies in various stages of educational development to illustrate common principles of skills assessment. These three technologies range from the well developed (standardized patients) to moderately developed (high fidelity integrated mannequins) to cutting edge (virtual clinical stations).

Standardized Patients

Standardized patients are actors who assess the specific clinical skills of learners. Since the 1960s, standardized patients have been used to simulate abnormal findings as formative teaching aids (⁶), to demonstrate sensitive normal (breast, pelvic exam) physical findings, or to assess specific domains of medical learner performance (e.g., communication, history, decision-making) in simulated clinic settings or classrooms. Both long- and short-format standardized patient interactions and exams have been explored (⁷). Well-trained standardized patients may evaluate rate-specific aspects of learner behavior nearly as well as clinical faculty (⁸, ⁹), but learner ratings are influenced by case order, learner fatigue, case length, gender, and other issues (¹⁰—¹⁶). Reliable individual feedback necessitates between six and 10 observations per domain. Therefore, during clinical performance examinations, learners may rotate through six to 12 standardized patient stations, have their behavior rated by standardized patients or faculty using previously constructed forms, and perform tasks between stations (¹⁷).

Now, specific technological advances have allowed a better utilization of standardized patients in medical education. First, digital videotaping has allowed for real-time remote and delayed review, storage, and analysis of standardized patient-learner interactions. Fiberoptic videocameras have reduced the size of recording equipment, allowing the learner to interact more naturally without the constant reminder of recording. Higher fidelity, directional microphones facilitate clearer audiovisual review, allowing educators to comment on tone, body language, and subtle communication cues. Enterprise computer software, such as B-Line and WebSP, have allowed integration of recording and evaluation software. High capacity T1 and fiberoptic lines allow easier high volume data transmission. Educators can review standardized patient-learner interactions from almost any location in an encrypted format through a central secure server, without compromising exam security by allowing learners to check out tapes of their performance and using intranets to limit access. However, the costs of developing, installing, and storing data using these programs can be daunting—easily several hundred thousand dollars. Developing a fully digital standardized patient-ready suite may cost upward of several million dollars, including administrative costs and lost revenue of potential clinical floor space.

Recently, unannounced standardized patients have been used to assess the habits of practicing physicians (¹⁸—²⁰). These standardized patients are inserted into a practicing physician’s clinic, usually as part of a clinical study, and present as real patients. Unobtrusive microphones have allowed standardized patients to review their clinical interactions after encounters to complete their rating forms with up to 10% to 20% more accuracy than with recall alone. Often, false electronic medical records and medical record numbers need to be created by the clinical practice to allow the standardized patient to enter the practice undetected.

In the future, tele-consultation skills may be honed or tested by using tele-standardized patients in which different clinical and communication challenges are modeled by actors at remote sites (²¹). For instance, how is a psychiatrist to meet the needs of a suicidal patient (or standardized patient) who may be located several hundred miles away? If a robotic simulator fails in the middle of surgery, how is a primary off-site surgeon going to guide the less-experienced surgeon in the operating room (OR) to complete the case?

The High Fidelity Integrated Mannequin

Patient simulators have been used to teach and evaluate procedural skills, medical management, crisis resource management, leadership, and teamwork. High fidelity or "integrated patient simulators" did not achieve mass production until the mid to late 1980s but are now used by many specialties (²², ²³). The integrated patient simulator is a mannequin that blinks, exhales CO2 as its chest falls, has palpable pulses, and can "speak" with the help of its operator. The mannequin’s software enables it to generate physical findings (heart or breathing sounds, ECG, oxygen saturation), respond physiologically to medications, or undergo procedures (intubation, cricothyroidotomy, needle thoracostomy, defibrillation, pericardiocentesis, and diagnostic peritoneal lavage). While most current patient simulators are used to teach procedural skills, their use is expanding into the cognitive specialties (medicine, neurology, psychiatry) as the mannequins become more realistic, animated, and physiologically robust. The addition of standardized patient family members or standardized nurses/therapists heightens the tension of crisis situations and allows for exploration of team dynamics. The cost of an integrated mannequin is about $250,000 for the mannequin simulator alone, not including the appropriate clinical environment construction (e.g., using real ICU ventilators, OR equipment), audiovisual/technology, or supplies for scenarios.

Competency standards for integrated mannequin simulations are just beginning to develop (²⁴—²⁸). Most studies that reported the use of simulators do not have reliable, validated, or relevant scoring systems (²⁷, ³⁵). Consequently, integrated patient simulators are used in relatively low-stakes formative assessments (²³, ²⁹—³²). Two exceptions in the literature stand out: New York’s physician re-training evaluation program (³³), and Israel’s high-stakes competency assessment during anesthesiology national board certification examination (³⁴). Other challenges using mannequins include creating reproducible clinical scenarios, accounting for equally acceptable treatment approaches, prioritization of multiple necessary actions by learners, sampling behavior sufficiently for summative evaluation, judging bias, and lack of realism of current simulators (e.g., no portrayal of anxiety, diaphoresis) (³⁵). For some examinees, the artificiality of the situation may be difficult to overcome, especially if they have not had previous simulation experience (³⁶). Competency standards for some types of skills simulators have been well-defined—such as simulators for cardiac catheterization or laparoscopic skills simulators (for remote robotic surgery). Consequently, these technologies are used both to train and test novices (²⁴—²⁶). Here, too, the cost is high: over $500,000 for the cardiac catheterization simulator and over $1 million for the robotic laparoscopic skills simulator, not including the full-time, on-site instructors and ongoing maintenance costs.

As these simulators become more realistic and "human," blending computer animation into their programming, the uses for simulation in the cognitive specialties will increase. However, cognitive specialty-specific applications will not emerge unless educators work with developers on applications specific to the needs of their learner group. These technologies are routinely used to train medical students and residents, and it is important for all educators to be aware of their current uses, potential uses, costs, and limitations. In psychiatry, applications are just beginning to be developed for the high fidelity mannequin. Psychiatry-specific scenarios will likely include management of critical side-effects of antipsychotic drugs, exploration of team dynamics in psychiatric emergencies, and management of suicidal patients and their families.

The Virtual Clinical Station

The intriguing possibility of providing high-quality, computer-generated interactive patients with programs that can assess clinical skills has led development teams to explore the use of gaming technologies, including those used for Xbox, PlayStation, Game Boy, GameCube, or Nintendo, for medical applications (³⁷—⁴¹). This technology for medicine, while still in its infancy, will likely become a staple in the armament of medical educators (⁴², ⁴³). Eventually, medical educators will be able to conduct skills assessments using Web-based virtual clinical stations or objective structured virtual examinations (OSVEs) (⁴⁴, ⁴⁵). OSVEs may include case-based learner assessments using several formats.

First, medical educators may develop interactive Web-based cases, with branching logic, that use videoclips of actors/patients about whom the learner can respond. Depending on choices that the learner makes along the way, different clinical outcomes may occur during the case. Different types of clinical information may be integrated into the case. Decision-making, reasoning, synthesis, and essay-based communication skills may be assessed. Integration with learner chat rooms, e-mail reminders, and distance mentoring for low/average performers may increase the evaluation utility. The technology to create these types of programs currently exists but does not yet have widespread circulation, often due to lack of funding for programmers to create specific applications.

Second, development teams are creating Web-generated environments in which the learner experiences a situation from the patient’s perspective. This method has been especially useful for understanding the perspective of psychiatric patients who may experience hallucinations. At the University of California, Davis, a virtual computer-generated clinic allows users to choose an "avatar" or virtual person that they use to walk through a patient clinic, as if they were in a videogame (⁴⁴). Learners can pick experiences from a variety of hallucinations, developed from schizophrenic patient experiences. These hallucinations may be auditory ("religious music" or "abusive comments" or "voices encouraging suicide"), visual ("stepping on stones above an abyss"), or both. Cases built around these experiences will allow the learner to explore interpretation of laboratory or radiology tests, conduct pharmacological administration, make appropriate decisions, take part in crisis intervention, and share the results electronically with their mentors via an electronic portfolio. Depending on the competency domain and clinical case portrayed, different types of knowledge, skills, or attitudes could be assessed (Appendix 1).

Third, intriguing possibilities arise that build upon current technologies or near future technologies. In the future, learners might be able to enter virtual worlds where they choose avatars to interact in clinical settings, such as an emergency department, clinic, or natural disaster—much like playing a cognitive-based video game (⁴⁶). Learners may take on roles as patients, health care providers, or families. As the game master, the medical educator can control the stimuli to which the learner is exposed—a neural toxin that causes psychosis, a series of emergencies, or routine clinic visits. Learners might also interact with other learners or educators posing as virtual patients via chat rooms or using voiceover IP protocols. The cases and evaluation modalities would need to be crafted with attention paid to reliability, validity, and utility. Another possibility for future development is the creation of fully generated computer simulations. In these OSVEs, computer generated artificial patients, using "artificial intelligence" technology, would be capable of generating appropriate, realistic answers to questions posed by the learner. When developed, these OSVEs would use voice recognition and natural language processing to code the comments/answers of the learners, and interactive software to proceed through the case with realistic artificial intelligence patients. The artificial intelligence patients would have a wide array of tones, affects, and subtle visual cues to indicate emotional states or concerns. Assessment could happen automatically and learner choices could be recorded for later analysis.

These technologies all currently exist, but many are in their infancy. Some of these programming technologies are well developed (avatars and interactive videogames). Much like designing portfolios and simple Web pages has become mainstream for medical educators, it is likely that within the next 10 to 15 years, normal medical educators will be able to work with programmers to design and build their own OSVEs. Currently, however, they are cost-prohibitive for the average medical educator to pursue independently, needing thousands of hours for storyboarding, script writing, designing, programming, and debugging the programs, such as with videogames like "Myst" by Cyan.

Depending on development costs, a wide array of structured environments could be generated to illustrate critical issues to which the learners must know how to respond, including some which occur with low frequency. The advantage of using Web-based learning tools lies in the ability to be used at the learner’s discretion, at any time and place where Internet/computer access is possible. Thus, unlike the objective structured clinical examination (OSCE), which must be conducted during normal business hours, the OSVE could be used at home, in a medical clinic, or even at a coffee shop with wireless high-speed access. Depending on technology development and maintenance costs, costs to administer the OSVE might be lower than the OSCE. Data might be easier to collect in the OSVE (such as keystroke monitoring), but might be more difficult to interpret. For instance, if educators wanted to interpret how long learners spent in an OSVE clinic room, they would not know if an increased length of time meant that the learners had gone to look up additional information, were pondering their response, or had gone to get a cup of coffee.

The use of avatars allows for anonymity of the learner and educator, reducing the potential for bias based on preexisting stereotypes or previous interactions. Perhaps most importantly, the OSVE may allow for multi-institutional simultaneous use and distribution—increasing the probability of collegial collaboration.

Critical Issues in Skills Assessment Using Simulators

Simulation is just one of many assessment techniques. Before deciding to utilize technology, educators should review the desired domains of assessment and determine the appropriate tool for the job (⁴⁷—⁴⁹). Any assessment modality has trade-offs in time, cost, faculty preparation, programmer/developer time, infrastructure, hardware, and staff support. Each modality has the potential to improve clinical care, if performed in a global curricular context. Educators should be encouraged to ask the following questions:

Educators should have a clear idea of what they want to assess, and determine how best to achieve appropriate sampling of that domain (¹—⁴, ⁵⁰, ⁵¹). Often, using multiple techniques to assess a learner’s understanding of a content area will lead to a more robust view of the learner’s ability. For instance, when assessing knowledge about major depression, an educator may want to test a learner’s immediate recall, factual knowledge, knowledge synthesis, or extrapolation of knowledge to other contexts. Or, the educator may want to assess the learner’s knowledge structure, reflection/self-assessment, or knowledge application/decision-making. When assessing clinical skills, the educator may want to test procedural, communication, counseling, physical examination, accuracy of self-assessment, crisis decision-making, leadership, teaching, or teamwork skills. The educator may assess attitudes, such as professional behavior and values, patient-centeredness (value, respect, listening, courtesy, comfort), and advocacy orientation. Finally, educators may assess learner habits, such as timeliness, follow-through, availability, responsiveness, consistency of clinical practice (screening behaviors, achieving practice standards, fairness in decision-making, data recording), routine self-assessment, or self-education. These competencies must be defined in an observable manner that allows assessment and the outcomes must be calibrated to differentiate between low, medium, and high performers. Appendix 1 outlines variables that can be controlled by simulation technologies during skills assessment encounters, and practical considerations for implementation. Usually, the availability of the local expertise drives the type of evaluation that educators choose to use during their assessment exercises.

Several major problems plague appropriate incorporation of new technologies into medical education settings. Broadly, these may be categorized into curricular, resource, and learner considerations.

First, technology development must be performed in the context of a local educational culture. Early adopters may rush to implement new products because of advantages over prior assessment/curricular techniques. The basic question, "What do we need to assess?" may not be adequately answered. For useful curricular development, competency assessment must be done in the larger educational context of content development, skills practice, skills assessment, and skills improvement. Attention must be paid to accountability and feedback of information to the learner.

Second, close attention should be paid to local faculty, collaborator, and infrastructural resources. An appropriate technology development team should be in place, resourced to the scope of the project. Potential hidden costs of simulation (such as software upgrades, server and facility maintenance, travel costs of actors, non-reimbursed time for the developer/educator) should be ascertained in advance. Medical educators should embrace collaboration with content experts who can provide input into critical aspects of simulation development enterprise. These collaborators include film crews, video editors, standardized patient trainers, computer programmers, graphic designers, plastics manufacturers, mathematicians, telecommunication experts, architects, and biostatisticians with qualitative and quantitative expertise.

Third, learners have a high degree of skepticism about new tasks that may increase their workload. Learners also expect medical technology to be as compelling and "slick" as commercially available products, including videogames and animated movies (such as those from Pixar/Disney and ILM). Products that fall short of these commercial ventures may be seen by learners as simplistic, unsophisticated, and nonengaging. Learners are also sensitive to how collected performance information will be used in their education. Thus, educators must find ways of ensuring information confidentiality and determine the appropriate use of assessment results (dean's letters, confidential records, appropriate feedback to learners). In isolation, assessment without feedback can have a negative impact on morale, learning, and practice environments.

Critical Issues in Data Analysis: Validity, Reliability, and Dependability

Educators need interpretable results to demonstrate the value of a new technology to their stakeholders. Once value is demonstrated, educators can negotiate for improved facilities, funding, personnel, or program expansion. Here, we explore three concepts that educators should be familiar with to demonstrate the value of their new product with certainty to outside groups: validity, reliability, and dependability.

Validity of a test is addressed by the question: "Does the technology assessment measure what the educator thinks it should measure?" Educators should think carefully about what they want to assess, and design a test that is most likely to assess a learner’s performance in that domain. In educational assessment, there may not be true gold standards against which to measure an aspect of learner performance (criterion validity). Thus, educators should think about (and possibly measure/obtain) related domains of performance (e.g., clerkship evaluations, complications of therapy) to see if those measures correlate as expected in the expected direction (construct validity). While this seems intuitively simple, educators often try to correlate tests of skills (communication with patients) with assessment of knowledge (MCAT scores), teamwork/likability (clerkship evaluations), or very distant multifactorial outcomes (like patient morbidity/mortality) and are disappointed when they find little correlation.

Reliability of a test is addressed by the question: "Does the technology assessment measure the ability of the learners in a reproducible manner?" (⁵²—⁵⁴). Understanding the reliability of a test involves understanding why the test has produced the observed results. The scores generated from any educational assessment will vary due to differences in knowledge, skills, or attitudes among the individuals being tested. However, the individual’s score may actually vary for many other reasons, for example, biases introduced because of heterogeneity of the raters, testing conditions or format. A reliable test will have reproducible results, because the educator has designed a testing situation that minimizes the sources of error (random and systematic) that could interfere with assessing the learner’s true performance. This might involve better training of raters, or having a more standardized testing environment. After obtaining the results of use of the technology, educators can use statistics to parse sources of error that interfere with test reliability. Classical statistics and the newer "psychometric" statistics provide tools to describe sources of error for major components test results: agreement after chance for multiple raters (kappa or intraclass correlation statistics), cohesiveness of domains within a set of test questions (factor analysis), cohesiveness of a set of questions as a scale (Cronbach's alpha), or stability of tested outcomes over time (test-retest reliability).

Most educators attend to the validity and reliability of their educational measures. Fewer educators routinely consider dependability (or individual generalizability) in their instrument development, study design, or analysis. In this context, dependability refers to the extent to which an educator can generalize a learner’s score from one particular assessment exercise (under a particular set of conditions) to other, broader clinical/testing situations (⁵⁵, ⁵⁶).

We provide one example to illustrate how a newer statistical technique can improve an educator’s understanding of test reliability and dependability while assessing learner skills. Assume that a group of educators wants to evaluate the ability of a group of 32 psychiatry residents to conduct distance consultations via synchronous real-time Webcasts. The educators develop an OSCE that uses a standardized patient portraying schizophrenia. Psychiatry residents are required to interview the patient via the Webcast. Specific technology and distance communication challenges are introduced to the OSCE. Over the next 2 months, educators administer this assessment to all 32 residents, averaging about four residents per week. To assess resident performance, a recording is made of each resident-standardized patient encounter. A panel of experts then divides the task of viewing the videotapes and scoring resident performance, using a 10-item survey. The survey instrument is designed to assess each resident’s performance in pre-defined competency domains related to interviewing skills. In the ideal assessment, variations in resident scores would be due to differences in interviewing skills among the 32 residents completing the OSCE. However, scores may also vary depending on which expert rates the interview; when the resident completes the OSCE (e.g., are they post-call or just having an off day?); the clinical scenario presented (e.g., type of schizophrenia, was there a psychiatric emergency with no available local help); the consistency in performance of the standardized patient; and the items included (or not included) in the survey instrument.

The classic approach to assessing reliability in this situation would be to attempt to measure each of these sources of variation/error individually (e.g., factor analysis, kappa, Cronbach's alpha). In other words, this approach does not take into account the potential interaction between multiple factors that can influence reliability (e.g., how does variation in a clinical scenario interact with learner’s fatigue?). A more robust approach to the assessment of reliability uses a variation of analysis of variance (ANOVA), called "generalizability theory" or G-theory (⁵⁵, ⁵⁶). The advantage of G-theory is that it allows educators to estimate the relative contributions and interactions of multiple sources of error simultaneously in the same analysis. G-theory also addresses the issue of dependability because it allows educators to estimate the degree to which they can be confident in generalizing a learner’s score on one particular assessment exercise administered under a particular set of conditions to other, more broad clinical situations and testing conditions. In other words, it allows an educator to determine how many testing occasions, assessment forms (e.g., different types of OSCE interview skills stations), and raters it will take to obtain dependable scores.

Conclusions

Simulator technology development has taken place during a profound shift in medical education, in which increased emphasis is placed on self-directed learning and assessment as a component of adult learning. Simulations are complementary technologies to other evaluation modalities and have major cost/benefit limitations. Major stakeholders (e.g., administration, learners) should be convinced of the value of the effort prior to embarking upon significant development and implementation efforts. As educators know, evaluation (especially external evaluation) often drives the curriculum, since learner deficiencies are demonstrated in critical assessment areas. Thus, as with all clinical skills evaluation, educators must attend to the simulation’s dependability, utility, reliability, and validity. Creatively utilized and properly analyzed, these emerging simulation technologies have the potential to improve clinical practice through educationally sound evaluation and feedback of clinical skills.

View Large | Download Slide(.ppt)
Add to My POL

APPENDIX 1. Resource Considerations in the Assessment of Clinical Skills

References

Srinivasan M, Hilty D: Teaching and helping others to learn, in Teaching and Learning in Academic Medicine: A Concise Guide for Academic Physicians. Arlington, Va, Academic Psychiatric Publishing, 2005

Keenan C, Srinivasan M: Teaching in ambulatory care, in Association of Program Directors in Internal Medicine Program Manual, 8th ed. Edited by Henderson MH. 2005

Davis DA, Thomson MA, Oxman AD, et al: Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA 1995; 274:700—705

Rossi PH, Freeman HE, Lipsey MW: Evaluation: a systematic approach. Thousand Oaks, Calif, Sage Publications, 1999

Mehta NP, Kramer DB: A critique of the USMLE clinical skills examination. MedScape Gen Med 2005; 7:76

History of SP reference, Howard Barrows and History of standardized patients, in History of Medicine

Dupras DM, Li JT: Use of an objective structured clinical examination to determine clinical competence. Acad Med 1995; 70:1029—1034

Rothman AI, Cusimano M: Assessment of English proficiency in international medical graduates by physician examiners and standardized patients. Med Educ 2001; 35:762—766

Humphrey-Murto S, Smee S, Touchie C, et al: A comparison of physician examiners and trained assessors in a high-stakes OSCE setting. Acad Med 2005; 80:S59—62

Tamblyn RM, Klass DK, Schandl GK, et al: Factors associated with the accuracy of standardized patient presentation. Acad Med 1990; 65:S55—S56

Carney PA, Dietrich AJ, Freeman DH, et al: A standardized-patient assessment of a continuing medical education program to improve physicians’ cancer-control clinical skills. Acad Med 1995; 70:52—58

Cohen DS, Colliver JA, Marcy MS, et al: Psychometric properties of a standardized-patient check list and rating form used to assess interpersonal and communications skills. Acad Med 1995; 71:S87—S89

Schnable G, Hassard TH, Kopelow ML: The assessment of interpersonal skills using standardized patients. Acad Med 1991; 66:S34—S36

Stillman P, Swanson D, Regan MB, et al: Assessment of clinical skills of residents utilizing standardized patients. Ann Intern Med 1991; 114:393—401

McLeod PJ, Tamblyn RM, Gayton D, et al: Use of standardized patients to assess between: physician variations in resource utilization. JAMA 1997; 278:1164—118

Regehr G, Freeman R, Hodges B, et al: Assessing the generalizability of OSCE measures across content domains. Acad Med 1999; 74:1320—1322

VanMents M: The Effective Use of Role-Playing: A Handbook for Teachers and Trainers (revised ed.). Edited by Kogan P. New York, London/Nichols Publishing, 1989

Kravitz RL, Epstein RM, Feldman MD, et al: Influence of patients’ requests for direct-to-consumer advertised antidepressants: a randomized controlled trial. JAMA 2005; 293:1995—2002

Glassman PA, Luck J, O’Gara EM, et al: Using standardized patients to measure quality: evidence from the literature and a prospective study. J Qual Improv 2000; 26:644—653

Hoppe RB, Farquhar LJ, Henry R, et al: Residents’ attitudes towards and skills in counseling: using undetected standardized patients. J Gen Intern Med 1990; 5:415—420

Kennedy C, Yellowlees P: The effectiveness of telepsychiatry measured using the Health of the Nation Outcome Scale and the Mental Health Inventory. J Telemed Telecare 2003; 9:12—16

Kapur PA, Steadman RH: Patient simulator competency testing: ready for takeoff? Anesth Analg 1998; 86:1157—1159

Morgan PJ, Cleave-Hogg D: A worldwide survey of the use of simulation in anesthesia. Can J Anesth 2002; 49:659—662

Aggarwal R, Moorthy K, Darzi A: Laparoscopic skills training and assessment. Br J Surg 2004; 91:1549—1558

Datta V, Bann S, Beard J, et al: Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg 2004; 199:603—606

Korndorffer JR, Scott DJ, Sierra R, et al: Developing and testing competency levels for laparoscopic skills training. Arch Surg 2005; 140:80—84

Byrne AJ, Greaves JD: Assessment instruments used during anesthetic simulation: review of published studies. Br J Anaesth 2001; 86:445—450

Murray D, Boulet J, Ziv A, et al: An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation. Med Educ 2002; 36:833—841

Murray DJ, Boulet JR, Kras JF, et al: A simulation-based acute skills performance assessment for anesthesia training. Anesth Analg 2005; 101:1127—1134

Clyne B, Gutman D, Sutton E, et al: Oral board versus high fidelity simulation for competency assessment: senior emergence medicine resident management of an acute coronary syndrome. Presented at International Meeting on Medical Simulation, Albuquerque, NM, 2004. Available at www.anestech.org/publications_abstracts.htm.

Sudikoff SN: High fidelity medical simulation as an assessment tool for pediatric resident airway management skills. Poster presentation at International Meeting on Medical Simulation. Miami, Fla, 2005. Available at www.anestech.org/publications_abstracts.htm.

Christensen J, Patel P, Skinner C: The use of patient simulators for residency competency evaluation. Poster presentation at International Meeting on Medical Simulation, San Diego, Calif, 2006. Available at www.anestech.org/publications_abstracts.htm.

Rosenblatt MA, Abrams KJ, New York State Society of Anesthesiologists Committee on Continuing Medical Education and Remediation, Remediation Sub-Committee: The use of a human patient simulator in the evaluation of and development of a remedial prescription for an anesthesiologist with lapsed medical skills. Anesth Analg 2002; 94:140—153

Berkenstadt H, Ziv A, Dach R, et al: The process of incorporating simulation-based competency assessment into the Israeli National Board Examination in Anesthesiology. Presented at International Meeting on Medical Simulation. Albuquerque, NM, 2004. Available at www.anestech.org/publications_abstracts.htm.

Boulet JR, Swanson DB: Psychometric challenges of using simulations for high-stakes assessment, in Simulators in Critical Care and Beyond: The Society of Critical Care Medicine. Edited by Dunn WF. 2004, pp 119—130

Hotchkiss MA, Biddle C, Fallacaro M: Assessing the authenticity of the human simulation experience in anesthesiology. AANA J 2002; 70:470—473

Najjar LJ: Principles of educational multimedia user interface design. Hum Factors 1998; 40:311—324

Inwood MJ, Ahmad J: Development of instructional, interactive, multimedia anatomy dissection software: a student-led initiative. Clin Anat 2005; 18:613—617

Issenberg SB, Gordon MS, Greber AA: Bedside cardiology skills training for the osteopathic internist using simulation technology. J Am Osteopath Assoc 2003; 103:603—607

Seabra D, Srougi M, Baptista R, et al: Computer aided learning versus standard lecture for undergraduate education in urology. J Urol 2004; 171:1220—1222

Williams C, Aubin S, Harkin P, et al: A randomized, controlled, single-blind trial of teaching provided by a computer-based multimedia package versus lecture. Med Educ 2001; 35: 847—854

Wiederhold BK: Virtual Healing. San Diego, Calif, Interactive Media Institute, 2004

Vichitvejpaisal P, Sitthikongsak S, Preechakoon B, et al: Does computer-assisted instruction really help to improve the learning process? Med Educ 2001; 35:983—989

Yellowlees P, MacKenzie J: Telehealth responses to bio-terrorism and emerging infections. J Telemed Telecare 2003; 9:S80—2

Humphris GM, Kaney S: The Objective Structured Video Exam for assessment of communication skills. Med Educ 2000; 34:939—945

Hulsman RL, Mollema ED, Oort FJ, et al: Using standardized video cases for assessment of medical communication skills: reliability of an objective structured video examination by computer. Patient Educ Couns 2006; 60:24—31

Parker P: The Courage to Teach: The Inner Landscape of a Teacher’s Life. San Francisco, Jossey-Bass, 1998

Pratt DD, Collins J: Five Perspectives on Teaching in Adult & Higher Education. Malabar, Fla, Krieger Publishing, 1998

Kember D: A reconceptualization of the research into university academics: conceptions of teaching. Learn Instruct 1997; 7:255—275

Mitchell M, Srinivasan M, Franks P, et al: Resident physician performance: an examination of state of the literature, and an integrative performance model. Acad Med. 2005

Srinivasan M, Litzelman D, Seshadri R, et al: Confronting unprofessional learner behavior: development of an instrument to code educator responses. Acad Med. 2004. Sept.

Crocker L, Algina J: Introduction of Classical and Modern Test Theory. CBS College Publishing, 1986

Cronbach LJ: Essentials of Psychological Testing. Harper & Row Publishers, 970

Streiner DL, Norman GR: Health Measurement Scales: A Practical Guide to Their Development and Use. Oxford Medical Publications.

Shavelson RJ, Webb NM: Generalizability Theory: A Primer. Newbury Park, Calif, Sage Publications, 1991, pp 1—16

Shavelson RJ, Webb NM, Rowley GL: Generalizabilty theory. Am Psychol 1989; 44:922—932