Medical specialties (including surgery, obstetrics, anesthesia, critical care, and trauma) have adopted simulation technology as a routine tool for measuring clinical competency in their residency training programs. The technologies used have ranged from realistic, "high fidelity" simulations involving human-sized computerized mannequins or virtual reality (1, 2) to simple, "low fidelity" simulations using a porcine eye or a piece of fruit (3, 4) for training specific procedural skills. However, these simulation technologies have rarely been adapted or used for medical student, residency, or postgraduate psychiatry training.

Psychiatry presents unique challenges for the utilization of simulation learning. The affective, behavioral, and cognitive processes involved in psychiatric care are difficult to simulate. For this reason, simulation in psychiatry has often involved the use of "standardized patients" (5), laypersons hired and trained to portray the role of an actual patient with a particular clinical problem, for teaching (6, 7) and assessment (8). Although simulation has a broad meaning in medical education (9—11), here, we focus on self-directed, computer-based simulations that ask participants to actively apply knowledge of a given topic to a simulated clinical encounter. To our knowledge, only five examples of interactive, computer-based simulation in psychiatry have been described. In 1993, the first personal computer-based case-simulation program was used in augmenting teaching in consult—liaison psychiatry. The second was a Continuing Medical Education (CME) workshop for the detection of stress disorders after a bioterrorist attack (12). In this instance, video clips of "virtual standardized patients" were observed by learners with the goal of identifying verbal and nonverbal cues and asking a limited set of questions in any order. The third example involved the use of a CD-ROM—based suicide screening tool developed by the United States Army (13). Participants navigated through 1,600 unique video clips, using an interactive voice-recognition system, to assess the suicide risk of a simulated infantryman named "Billy." Virtual standardized patients were also used to assess a trainee's ability to diagnose conduct disorder (14) and posttraumatic stress disorder (PTSD) (15). In this experimental design, trainees used a head-set microphone and voice-recognition software to interact with a computer-based representation of a virtual patient.

In this article, we describe the use of a web-based computer-simulation tool that was developed to teach residents how to obtain informed consent before prescribing a psychotropic medication. This clinical competency was chosen for simulation because it comprises a key element of patient-centered care, and it is essential for ethical and legal reasons (16). A recent study using clinical case vignettes found that only a minority of residents at seven training programs took an appropriately active approach to obtaining informed consent (17). These authors concluded that "changing residents' passive approach to informed-consent discussions might have a large impact" on improving patient—physician communication. Consequently, we sought to design an interactive, web-based simulation that would actively engage residents in the process of learning about how to obtain informed consent before prescribing an antipsychotic medication. The working hypothesis was that residents would find the simulation tool easy to use, and that their confidence in obtaining informed consent would improve significantly as a result of this brief training experience.

Study Subjects

Eighteen residents in the Massachusetts General Hospital/McLean Adult and Child Psychiatry Residency Training Programs (MRTP) were invited to participate in this pilot study of web-based simulation. As an incentive, they were told that upon completion of the study they would be entered in a raffle for one $50 debit gift card. Under an IRB waiver, prior approval to collect and analyze learners' responses to the vignette was sought and granted from the Massachusetts General Hospital Institutional Review Board. Sixteen trainees (89% of those solicited) participated in the pilot study (Adult Psychiatry residency training program: five PGY3s and four PGY4s; Child Psychiatry residency training program: four PGY4s and three PGY5s.).

Materials

Using a software programmer (author TG) and a commercially-available web-hosting service, custom web-based simulation software was written for this project. The software we created represents a flexible infrastructure that could easily be adapted to host many different scenarios. Here, we focus on a prototype scenario that can assess physician competence in obtaining informed consent before prescribing an antipsychotic medication. The simulated patient was a 22-year-old woman college student with psychosis. The design process involved the identification of specific learning goals associated with informed consent in the administration of antipsychotic medication, and subsequently, the creation of a tree diagram showing all the nodal points involved in reaching these learning goals. Senior clinicians reviewed the tree diagram for completeness and helped to write a script for each nodal point. The software program measured performance-based completion of five critical elements of informed consent. These items include discussion of: 1) the expected benefits of taking the medication; 2) the relevant and significant risks, side effects, and adverse reactions of the medication; 3) the probable consequences of not taking the medication; 4) the alternative treatments; and 5) the checking for understanding (i.e., whether the patient was able to appreciate the information conveyed by the psychiatrist during the consent process, given the patient's level of health literacy) (18, 19).

Participants accessed the CSAT over the Internet, using a standard web-browser. After securely logging into the CSAT website, participants began the simulation by watching a brief Flash-based video that introduced the clinical scenario. Learners were instructed that they were about to begin a simulation in which they would be engaging a patient in a conversation about the initiation of an antipsychotic medication. Participants were then presented with a series of menu options from which they could choose their next action (see screenshot in Figure 1). Learners had the opportunity to select from a multiplicity of lines of inquiry related to the discussion of side effects, treatment alternatives, checking for understanding, and the scheduling of follow-up visits. Examples of possible actions included "discuss weight gain," "discuss alternative treatments," and "check understanding of [the patient's] illness." Each selection was followed by a video clip showing the clinician and simulated patient completing the selected task. Some menu options were contingently available (e.g., a learner could not "check for understanding of metabolic syndrome," without first selecting, "discuss risk of metabolic syndrome"). To replicate the time-constraints of an actual patient encounter, a virtual timer was set at 12 minutes and displayed the elapsed time.

After the allotted 12 minutes had expired, or the participant indicated they were finished with their questions and explanations, the program provided a feedback screen regarding the appropriateness of the viewers' actions. In an effort to provide targeted remediation for errors of commission and omission, participants were provided with an on-screen written explanation of why each error was either incorrect or necessary (if omitted), as well as a link to a PDF of a peer-reviewed manuscript that discussed the topic. For example, if a participant forgot to discuss metabolic syndrome with the simulated patient, during the feedback session, the participant was linked to an article discussing the pathophysiology and long-term implications of the disease.

Outcome Measures

Physician Performance

To assess physician performance in obtaining informed consent, the CSAT program counted the number of required elements each participant completed.

Confidence Scale

To reliably assess changes in confidence with respect to obtaining informed consent for the use of psychotropic medication, the wording of a scale found to have high internal consistency and developed by Smucker and associates (20) to assess practitioner confidence in treating back pain, was modified. The phrase, "to effectively obtain informed consent when prescribing new psychotropic medication," was substituted for their original phrase, "to effectively assess patients with low back pain," in each question. Participants rated their agreement with each of the following four statements by means of a Likert scale that ranged from 1 ("strongly disagree") to 5 ("strongly agree"): 1) "I lack the knowledge needed to effectively obtain informed consent when prescribing new psychotropic medication;" 2) "I know exactly what to do to effectively obtain informed consent when prescribing new psychotropic medication;" 3) "I am very comfortable obtaining informed consent when prescribing new psychotropic medication;" and 4) "I am well prepared to obtain informed consent when prescribing new psychotropic medication."

System Usability Scale (SUS)

The System Usability Score is a standard measure of software usability adopted by the computer industry (21). Participants assigned a score from 0 ("strongly disagree") to 4 ("strongly agree") to each of 10 statements that ask how easy it was to understand and interact with the website. Sample items from the SUS include: "I thought the system was easy to use;" "I imagine that most people would learn to use this system very quickly;" "I felt very confident using the system;" "I would like to use this system frequently;" and "I found that the various functions in this system were well integrated." Following the scoring guidelines outlined by Brook and colleagues (21), a standardized usability score was created (range: 0—100). High usability is thought to be indicated by an overall score >80. The SUS has not been validated for individual item-analysis.

Procedure

Each participant completed the Confidence Scale before interacting with the web-based simulation. Participants then accessed the web-based simulation, using any available Internet-connected computer, after which they again completed the four-item Confidence questionnaire and also completed the System Usability Scale.

Participants selected actions that addressed a mean of 4.4 of the 5 required elements for informed consent (11 participants selected all five elements, one selected four elements, and four selected three elements) on the physician performance measure.

Statistically significant improvements were found on all four items of the Confidence Scale from pre-test to post-test (Table 1).

Participants reported a high level of system usability with the web-based simulation tool, as indicated by a median score of 86.5 (IQR: 80.6—90.5) on the System Usability Scale.

This pilot experience demonstrated the feasibility of integrating web-based patient-simulation tools into adult and child psychiatry residency training programs. Web-based simulation allowed for the assessment of a psychiatric trainee's ability to select actions necessary to provide informed consent to a virtual standardized patient. It also provided simulated practice and screen-based feedback to improve this competency. Data from this study suggested that the CSAT was effective in increasing trainee confidence in completing an informed-consent task.

A major advantage of web-based simulation—such as the CSAT—is that it permitted asynchronous learning; that is, the simulation could be accessed by multiple learners at the same time, from any location, at a time that was convenient for the learner. This is in marked contrast to traditional simulations that use standardized patients in synchronous interactions where the actor and student(s) are required to be present at the same place and time.

There are several ways in which asynchronous, web-based learning may be advantageous. First, asynchronous simulation training allows programs to meet training and education needs of individual residents in real time. This ability to practice a particular skill the day before—or the morning of—an expected patient encounter has the potential to enhance the confidence and performance of the learner in the clinical setting. Second, it also allows trainees to practice and to learn new skills without risking the safety of actual patients. Third, in the rich clinical-training environment of an inpatient unit or a supervised outpatient clinic, residents can practice a cognitive skill with supervisory feedback in close proximity to the point of care.

This efficiency and flexibility in training may enhance the educational approach to core competency assessment. Physicians generalize much of their professional training, which is fundamental to practice-based learning of core competencies (16). Incorporating simulation training of several different learning modules may generalize to problem-solving and cognitive formulation of other clinical scenarios, even if those skills are not directly addressed by a specific simulation. For example, completing a simulation related to informed consent for one type of medication may improve a learner's ability to obtain informed consent for other types of medications or procedures.

Although the data in this study suggested that CSAT was effective in increasing user confidence in obtaining informed consent and that it was highly usable, this pilot study has several limitations. First, the residents selected for this study were small in number and did not represent a random sampling of trainees. All of the residents were PGY-3s or above and therefore had at least 2 years of clinical experience as a physician. As such, the selection of these residents likely contributed to an underestimation of the difference between pre- and post-exercise confidence measures.

Also, the confidence measures used were not specifically designed or validated for measuring confidence in obtaining informed consent for an antipsychotic medication; they were an adaptation of an existing confidence scale. However, to date, there are no published scales that have been specifically designed for assessing provider confidence in obtaining informed consent. It should also be noted that the Confidence Scale measured trainees' self-reported subjective sense of confidence in obtaining informed consent, but it does not provide a specific measure of the trainee's actual ability to complete the task.

The data also suggest that the CSAT showed high levels of system usability in this pilot study. It should be noted, however, that the SUS was designed as a general tool for assessing software usability, and is not specific to web-based simulation tools. Furthermore, SUS scores may have been influenced by the study population. Most PGY-3-to-PGY-5 trainees are part of a "net-savvy" generation (22); that is, they are comfortable using computers and completing educational interventions online. Such high usability scores may not have been found if this pilot study was conducted in a more senior cohort of psychiatrists.

Finally, this study leaves unanswered the question of how best to incorporate this type of web-based simulation into an existing residency-training curriculum. Other studies have suggested that web-based simulation is most useful when integrated into traditional didactic curricula and/or environments in which trainees are likely to use the highlighted skill (23). In the next phase of this project, the MRTP will use the CSAT simulation tool to augment didactic curricula and supervision. PGY-1 residents will begin by completing a CSAT simulation assessing their ability to obtain informed consent for antipsychotic prescription. The chief resident on the medical-psychiatry inpatient unit will then deliver a didactic teaching presentation about informed consent at the commencement of the clinical rotation. Residents on this rotation will then complete a second simulation focused on informed consent for the prescription of antidepressant medication. Each resident will review his or her simulation performance with the chief resident and/or their attending (given that much of the literature in medical education simulation finds that most of the learning actually occurs in the debriefing session after the simulation exercise itself (24)). A similar process will be repeated for PGY-2 residents on their psychotic-disorders unit rotation. This study design would permit a more objective examination of the efficacy of the CSAT intervention.

Future versions of this tool will include additional case vignettes focusing on informed consent for SSRI medication prescription and will feature patients of various ethnicities with low health literacy, as low-health-literacy populations are particularly vulnerable during the informed-consent process (25). The degree to which performance improvements within simulations translates into improved patient care will be assessed through chart reviews and/or follow-up CSAT simulations.

Web-based simulation of patient—physician interactions can serve as a useful adjunct to the traditional educational methods used to teach psychiatry trainees; however, not every clinical encounter can or should be simulated. The goal of a web-based simulation is not to recreate all aspects of a clinical encounter, but to simulate and examine the participant's performance within a narrow slice of a clinical encounter. As such, simulations should be designed with well-defined and achievable objectives for a limited clinical encounter. Within this narrow focus, the educational utility of web-based simulation may extend beyond psychiatry and graduate medical education. As the Accreditation Council for Graduate Medical Education (ACGME) is looking toward more reliable and valid assessment tools, this methodology might be applicable not only to psychiatry, but to general-medical education (e.g., informed consent is required of all specialists). Web-based simulation of this type could provide assessment of knowledge, skills, and attitudinal acquisition across the continuum of medical education, from medical-student education, through residency, and into postgraduate continuing education. This includes assessment of American Board of Psychiatry and Neurology (ABPN)-required core competencies in trainees; augmentation of didactic material taught in Continuing Medical Education programs; and assessment of established physicians in post-licensure maintenance of certification.

The authors acknowledge Jenna Billian and Steven Locke for their invaluable assistance with this project.

At the time of submission, the authors reported no competing interests.