The explosive expansion of computer technology over the past 20 years has left medical educators racing to find novel applications for computers in medical training. Computer assisted instruction (CAI) is found throughout medical training programs in the form of computerized lectures, online learning modules, and patient simulations. The general utility of computers and the temptation to favor novel ideas often lead educators to assume that CAI is superior to traditional teaching methods. However, when asked to justify the time and resources needed to develop and maintain electronic educational tools, educators have little objective evidence to support this accelerating trend. Furthermore, there are no clear strategies for development or testing of educational computing, leaving residency directors to their own devices when deciding how and why to apply CAI to their own programs.

There is currently a rapidly proliferating body of work examining these questions in the medical education literature, most of which casts CAI in an overwhelmingly positive light. The prevailing opinion asserts that there are multiple benefits of CAI over traditional pen-and-paper or lecture-style learning. Many educators feel that today’s learners prefer, or even demand, that their educational material be online (1). Proponents state that CAI enables learners to review material faster than “hard copy” (2) and that CAI enables educators in a hectic and unpredictable clinic environment to provide a standardized educational experience when specialized instructors may not be available (3, 4). Paradoxically, CAI developers also claim that it provides a more individualized educational experience through greater interactivity. In addition, computer technology increases the efficiency of the educational setting by automating evaluations and assessments (5) and helping educators organize and increase access to learning materials.

Nevertheless, there are several concerns raised in the literature as well. Critics point out that the existing literature is redundant and primarily descriptive, enumerating individual CAI interventions, but providing minimal or no data supporting their efficacy (6). Manuscripts that describe CAI tend to be written by computer-savvy developers who have invested a significant amount of time designing the CAI intervention and therefore are biased towards its advantages. Experimental approaches to the objective study of CAI vary widely, making it difficult to make comparisons with or draw generalizations from the literature. These studies usually test the final product of a long development process and rarely provide guidance for someone interested in creating new CAI methods. Furthermore, the current literature reviewing CAI and its evaluation exists in a diverse range of sources, from the educational journals of various medical subspecialties to broader educational forums, such as Academic Medicine. Even if it were possible to keep track of this multitude of sources, it is unclear how well this information generalizes to the field of psychiatry. Psychiatry has unique educational requirements; interviewing skills, mental status findings, and therapeutic approaches can be difficult to transfer to a computer format.

This debate calls into question not only the utility of CAI in our field but the best way to approach the issue in an empirical manner. It is important to examine the literature with consideration for the unique needs of the psychiatry educator who may or may not have expertise with computers. What information helps this individual evaluate CAI and decide how it is best implemented in his or her own academic setting? There are several important questions to this end: 1) How is CAI best studied in a naturalistic educational setting? 2) What are the barriers to assessing CAI in this setting? 3) What outcome measures have validity in studying CAI? 4) Are there other factors that limit the utility of CAI in psychiatry education? To date, attempts to answer these questions have usually centered on attempts to prove that CAI is “better” or “worse” than traditional learning techniques. However, it is also important to ask if such stark distinctions are beneficial, or even possible, given the drastic differences in the two formats and the difficulty conducting standardized research in a real-world educational environment. Is there a more efficient approach to studying CAI than strict comparisons to other methods?

In order to review the relevant literature systematically, a Medline search was conducted, with the combined fields of “computers” and “education/medical” or “education” and “psychiatry,” yielding 213 hits. This list largely comprised manuscripts relating to patient education. The abstracts were reviewed and articles not applicable to psychiatry education were excluded. The topic of distance learning, or “telepsychiatry,” has a broad literature worthy of a separate review and therefore was considered out of the scope of this review. Similarly, as there is already an up-to-date review of “cybersupervision,” it would be redundant to echo that here (7). Articles were excluded if the primary topic concerned computer applications in direct patient care, HIPAA compliance, computer security, ethical use of computing resources, information technology training, or CAI in a nonpsychiatry specialty. The remaining articles were read in their entirety and the above criteria were applied to the full text. Since the journal Academic Psychiatry is only fully referenced on Medline since 2003, a separate search was conducted on that journal’s Web site with the terms “computer” and “education,” and the resulting articles were reviewed as above. Five additional relevant manuscripts were identified in the citations of the above articles. After this process, a total of 20 papers were reviewed.

It was useful to classify these articles based on methodology. A system has been previously described in the nonpsychiatry literature that separated CAI studies into one of three categories: media-comparative, demonstration, or analytical (6). Media-comparative articles compare a CAI intervention to other teaching media and, as such, always include some form of data on which to base the comparison. Demonstration articles are the educational equivalent of case reports; a single CAI intervention is written up as an example of what can be done in a computer-based format. They may or may not include some means of assessment. Analytical articles evaluate an aspect of CAI or the literature as a whole. They are rarely primary studies and are usually editorial commentary or literature review.

As direct comparisons of old versus new teaching methods, media-comparative studies have the most face validity as a way to examine the efficacy of CAI. There were five media-comparative articles in the psychiatry literature (Appendix 1). Three of these are comparisons of a specific educational intervention versus traditional teaching methods. All include only cursory descriptions of the development of the program in question, focusing instead on content and format. These studies use very different experimental designs, making comparison difficult, but focus on attempts to create a controlled trial of CAI versus traditional educational formats. Overall, they demonstrate no consistent advantage or disadvantage of CAI compared with traditional learning. Even within individual studies, the performance of CAI versus controls varies depending on the outcome measure. For example, in a comparison of a multimedia teaching module on anxiety and panic disorder versus an equivalent lecture, the CAI group scored higher on a graded mental status exam generated after the session, but both groups scored equivalently on a knowledge-based post-test (8). In a comparison of a CAI module versus small group learning in a medical ethics course, the overall scores on a multiple-choice final exam were equivalent between groups, although the CAI group outperformed the control group in questions regarding patient confidentiality. In a different arm of the same study involving a scored standardized patient exam, the CAI group outperformed the control group in bioethics content, performed equivalently on communication skills, and underperformed the control group in patient satisfaction (9). Thus, it begins to appear that validating the use of CAI may be more complicated than simple comparisons to other media, but that some content or environments may lend themselves better to adaptation to CAI than others.

The remaining two media-comparative manuscripts deal more with the integration of computer resources into the psychiatry education process. One outlines the switch from a paper-based to an electronic system for evaluating faculty and trainees (5). This study documents increased compliance with return of evaluations after switching to the electronic format, as well as a general preference of students for the electronic format (76% favored electronic evaluations, 21% felt they were equivalent, and 3% favored the paper format). Faculty, on the other hand, were more resistant to the change; 25% continued to use the paper evaluations, and of those that used the computer format, only 75% preferred it to paper. This manuscript provides a useful look at integration of technology into the residency training program. The major limiting factor here was Internet access and institutional firewall problems, highlighting the fact that conversion to computer-based systems requires an investment in adequate computer resources and support from the information technology resources of an institution. This was also the only manuscript reviewed which discussed the financial cost of the intervention, a real-world issue that is often the limiting factor in a program’s ability to adopt new technology. Unfortunately, although the authors estimate the cost of development and maintenance of the program, they do not compare this cost to the traditional system.

The last article in this category compares the use of print versus electronic media for literature review over the last 5 years (10). It includes a brief survey of residents and faculty asking participants to estimate methods of information-gathering (print versus video versus Internet) now and 5 years ago. The survey found a significant increase over the 5-year span in use of the Internet as an educational source and a significant decrease in the use of print media. This confirms the general feeling that today’s psychiatrists are rapidly moving toward the Internet as the primary source of literature review. Included is a review of the literature on electronic journalism and a useful discussion of the pros and cons of this method of information dissemination.

A careful look at the methodology of the above studies highlights the difficulty of conducting comparative studies in a naturalistic educational setting. None was able to create truly equivalent experimental and control groups. Notable variations exist between the groups in specific educational content, educational format (e.g., small group versus lecture), and length of time spent with the educational material. There is also significant difficulty randomizing students to comparison and experimental groups. The best design was able to randomize student groups and deliver interventions of equivalent length at the same time of day, but the traditional group received a lecture while the CAI group used small groups of three to four students working together per computer station. This adds a confounding factor that the CAI group was a more interactive experience than the lecture, which can itself influence learning (8). In the comparison of Web-based versus live lectures, students were allowed to choose which they attended, introducing selection bias. Although this study reports that shelf exam scores after the CAI intervention were higher, it actually compares the aggregate scores of the entire class the year before the intervention with the entire class in which the intervention occurred, thus comparing a class which had only live lectures with a class which had mixed use of live and Web-based lectures. It is difficult, then, to feel that this result clearly favors CAI, as the second set of exam scores includes no breakdown of student attendance or preference of live versus Web-based lectures (11). These problems are not the result of careless experimental design, but rather indicate the need to ensure adequate education to all students. Nevertheless, they make it difficult to look at the results of these studies with confidence, and they highlight the difficulty of using this approach to examine CAI.

An important finding in these articles is that although they often collect both user preference/satisfaction data as well as objective measures of knowledge gains, these two outcomes do not necessarily agree. Students who learned bioethics with a CAI module felt just as prepared to deal with issues in ethics, confidentiality, and “doctor-patient communication issues” as students who learned in a small group setting. These students were also equally satisfied with the overall course as those that did not have the CAI intervention. However, during the standardized patient exam the CAI group scored higher than small-group discussants on bioethics knowledge and lower on patient satisfaction (9). Similarly, students who used a CAI module to learn about anxiety and panic self-rated their knowledge and skill significantly lower than those who learned from a traditional lecture. However, the two groups showed no difference in scores on an objective post-test, and the CAI group scored significantly higher on a graded mental status exam (8). Thus, user satisfaction and self-assessment of knowledge gains, though often used as a proxy for educational efficacy, are not necessarily reliable in this role.

There were seven demonstration articles reviewed. Of these, only two provided outcome data (Appendix 2). Two of these were different analyses of data from the same CAI module (3, 4). Data collected in demonstration articles usually consist of an accounting of the number of participants who completed the entire intervention and a satisfaction survey of those participants. Some collect general demographic data, which are useful for better understanding CAI users. A Web-based module on addiction education made openly available via an advertisement in Psychiatric News found that one-third of participants were not psychiatrists and that two-thirds of participants used the module for very general reasons, such as “curiosity” or wanting “to learn any clinical topic on the Internet” (3, 4). This suggests that open availability of CAI attracts “Web surfers” rather than clinicians focused on a specific clinical question. User satisfaction of these demonstration studies is generally positive, though the fact that they enlist volunteer participants who chose to use the CAI programs is a source of selection bias.

The remaining four demonstration articles describe novel CAI interventions without any structured form of evaluation (Appendix 2). They provide interesting descriptions of CAI applications, but a lack of assessment makes it difficult to form an opinion about the utility of these interventions. Similarly, lack of description about actual development of the interventions makes it difficult for readers to reproduce the technology in question.

Two manuscripts describe CAI modules with intriguing applications to training and provide partial blueprints for development of CAI for the motivated educator (12, 13). The remaining two describe novel applications of computers to the psychiatry training program. A description of an electronic database to track trainees’ schedules and progress through the program and notify trainees and faculty of deadlines and missed evaluations is interesting but largely out of date now that such programs are commonplace in the modern residency education office (14). More timely is the description of an “electronic filing cabinet” to facilitate access to important articles in the literature. It describes the development, infrastructure needs, and copyright issues of such an endeavor and would be a useful start for an administrator interested in replicating this work (15).

The remaining eight articles fall in the category of analytical articles, meaning they are review articles or editorials (Appendix 3). Many are intended to summarize the state of current technology for an audience with less expertise. For example, Thomas Kramer and Robert Kennedy ran a column entitled “Educational Computing” in Academic Psychiatry between 1996 and 1999. This column generally discussed basic concepts in technology to bring the novice user up to speed. Topics included “Software Reviews” and “Useful Websites for Psychiatrists” (16, 17). Most of these columns were focused on basic computer topics applicable to direct patient care; only “Software Reviews” dealt with psychiatry education and thus was included in this formal review, but the others are a useful reference nonetheless. The literature also contains several discussions of how technology is changing psychiatry in general, including clinical care, research, and education (18–20). These also primarily give an overview of general principles of computers and the Internet, targeted toward the technological novice and contributing little to the specific question of efficacy or utility of CAI.

A review of technological applications in psychotherapy training was mentioned above as an excellent source for information on cybersupervision (7). It also contains an interesting discussion on applications of technology to psychotherapy training, including a discussion of basic principles of CAI in general. This review is certainly recommended reading for any educator interested in CAI. Drawing mainly on literature from nonmedical education sources, it provides multiple examples of studies in which CAI was found to be equivalent or superior to traditional learning. However, it also notes that many of those studies have been criticized for the same difficulties of randomization and selection bias found in the media-comparative articles discussed above. It identified only a few primary studies of CAI in psychotherapy training and was unable to draw conclusions from this small set.

There have been several efforts over the years to summarize the existing literature or available resources in psychiatry education. Locke and Rezza (20), in 1996, wrote a commentary summarizing the history and application of technology to mental health education, treatment, and research. Attached to this article is a thorough appendix listing Internet and software resources available at that time (20). It is a frustrating demonstration of the speed of technological progress that most of the resources on this list are now inactive, outdated, or have become so commonly used that there is little value in placing them on such a list (e.g., MedLine, FREIDA). Similarly, the novel computer technology of the 1990s is the standard or even obsolete technology of the 2000s. There are several reviews that cover the early literature very well, but they have limited application to modern conditions (19, 21).

Many of the same questions and complications with adapting CAI to medical education and validating its use are echoed in the general medical literature outside of psychiatry. There has been noted to be a preponderance of demonstration articles without clear outcome studies (6). Although descriptions of individual CAI modules abound in almost every specialty, there is little to be gained from reading more than a few of these. The literature largely ignores the resource intensity of developing and maintaining adequate CAI interventions, and rarely is the financial or work-hour investment discussed. There are widespread difficulties creating methodologically sound comparison studies of CAI versus traditional learning. As in the media-comparative studies discussed above, it is very difficult to control for all of the variables in a head-to-head CAI versus traditional educational interventions. Confounding variables include instructor variability, content variability between the CAI and traditional format, and differences in pedagogical style both between and within the test groups (22). No paper identified by this review described an adequate method of constructing these studies. Some argue that it is because of this that the media-comparative model of CAI versus traditional learning methods is inherently problematic and should be replaced by attempts to examine different approaches to CAI and achieve a better understanding of how these methods apply to different learning goals (23).

An example of such an evaluation strategy is found in a series of articles using the Medici program, a CAI tool designed at the University of Adelaide, Australia, to help teach anatomy to second-year medical students. The initial study compared four different CAI or traditional teaching models: an online didactic tutorial (essentially a computerized lecture), a “problem-solving” module with multiple-choice questions, a “free text” module (using open-ended questions with typed answers, scored by the computer), and a traditional lecture. Comparisons were made via pre- and post-test scores on a written exam. All groups improved their scores after the educational intervention, but only the CAI didactic group showed a statistically significant improvement over both the control group and the free text group. This was contrary to the expected result that the free text version, being the most interactive option, would be the superior educational intervention (24). It was hypothesized that this may have been due to the fact that the multiple choice exam was focused on recall rather than application of knowledge, a task for which straightforward didactics may be better suited. A follow-up study used a similar design but used an exam with more open-ended questions. Again, all four groups showed improvement over the pre-test scores, but in this study the free text version was the only group that significantly outperformed the control group, and it did not significantly outperform the other two CAI interventions (25).

There is also evidence that there may be variations in how individual students perform on different computer testing modalities. In a comparison of CAI multiple choice versus CAI open-ended test scores, there was a significant positive correlation in the bottom 50% of the class, but a negative correlation in the top 50%. The authors of this study suggest that the different CAI formats measure different aspects of student knowledge (26). The take-home message from these studies seems to be that there are differential learning or assessment effects in different CAI formats, and there may be no single superior format for all situations. As in traditional learning, thought must be put into matching the presentation of information with the desired assessment and use of that information.

There are some useful resources in the general medical literature that can inform psychiatry’s approach to CAI. Notable among these is a very useful step-by-step guide for developing Web-based learning modules with a mind toward adult learning theory in the Journal of General Internal Medicine (27).

Based on this review, the current literature does not offer much direction to the clerkship or residency director curious about the pros and cons of integrating CAI into the educational setting in psychiatry. There is no successful model for conducting comparative studies in this field. Attempts at such studies are very time and resource intensive, and it is not clear that data generated from such a study would generalize to other educational settings, given the unique demographics of each training program. The literature also offers little insight into practical necessities, such as financial cost, computing support infrastructure, and faculty time needed for development and maintenance of CAI.

Despite these limitations in the literature, it would be both pessimistic and unrealistic to conclude that there is no place for CAI in psychiatry education. On the contrary, this review highlights some of the diverse and creative applications for CAI in the psychiatry training setting. The question remains of how to study and quantify the benefits this innovative technology has to offer. There remain other avenues in the literature that may offer some insights into this question.

Telepsychiatry and cybersupervision, for example, are very different from the type of CAI reviewed here but are still important models for integrating technology into the educational setting. A valuable review of both clinical and educational telepsychiatry was recently published which points out clear benefits of this technology, including increased access to clinical care and specialists, high patient and learner satisfaction, and high reliability when used as an evaluation tool. This review also identifies many of the same gaps found in the CAI literature, namely a lack of information about financial costs of implementation, as well as a paucity of studies demonstrating clear comparative or outcome data (28). Also of note is a recent article providing helpful guidelines to educators interested in integrating distance learning technologies into their training programs (29). The cybersupervision literature highlights some pitfalls of incorporating technology into psychiatry training, such as complicating communication between teacher and learner, and disrupting the therapeutic process. Nevertheless, continuing miniaturization of relevant technologies supports the idea that computer-mediated live supervision may eventually become less disruptive than other forms of live supervision (7). One could also look outside the medical field altogether to see how other professions have approached this issue. The Journal of Computer Assisted Learning examines CAI from a general educational standpoint, and there are several other similar publications. It would take the average psychiatry educator too much time and energy to follow these sources, but they may be a valuable resource for those with focused interest in this topic. In the future, a thorough review of these sources may help bridge the gap between the two disciplines.

There remains the problem of how best to approach CAI from a research perspective. Most efforts at validation utilize the media-comparative model to construct head-to-head comparisons of new versus old teaching methods. As demonstrated above, this model is difficult to construct and execute and so far has yielded little useful data. It is possible that a completely different approach would be more useful. The dramatic difference between CAI and traditional learning calls into question whether it is possible to construct groups that are truly equivalent for a controlled trial. Given that the studies that have come close have demonstrated mixed results depending on content area and outcome measure, it seems a lot of work to try to come to a binary “better” or “worse” answer on an issue that is probably much more complex than that. In addition, overall assessment of a final product does not give much insight into what parts of that product work or do not work. Significant time and resources are invested in developing a CAI module; if evidence showed it underperformed other teaching methods, developers would likely want to improve it rather than abandon the entire project. The current media-comparative model does not offer any insights into determining strengths or weaknesses within a module. It may be more beneficial to study the component parts of CAI interventions to develop a set of guidelines to steer future development. In this way an educator could show that a CAI module was constructed from validated “building blocks” that are known to be effective.

For example, CAI modules often include “branch points” or “hyperlinks” that are not part of the primary educational content but exist for interested students to explore a given topic. How often do students actually use these links, and what is the probability that a given piece of material will be seen if it is placed on the side in this manner? CAI modules often use video segments to create a case-based format or highlight important points. There are anecdotal observations that students may get “bored” during videos and skip sections, and there have been suggestions that there is an “attention ceiling” past which students will not watch anymore. If so, what is the optimal length of a video segment? These are small questions that would not require extensive control groups or hours of development of a single module. One could design a module and vary individual aspects between well-controlled and randomized experimental groups. This approach could potentially yield valuable information about what facets of CAI work or do not work in given situations. Ultimately, such an approach would give the average residency director valuable tips on how to construct their own CAI module.

Similarly, there is a dearth of information on real-world logistics of integrating technology into the residency. There is the intriguing possibility that CAI may save both money and faculty time, as well as improve organization and efficiency of educational departments. On the other hand, there are undefined costs of development and maintenance of these interventions. Simple cost/benefit analyses of CAI, including guidelines on time and costs of development, could be of enormous value to an educator lobbying for support for a CAI intervention. Similarly, a clearer understanding of how technical support issues, such as computing speed, firewall, and off-site access limitations, affect the utility of CAI would be helpful.

Computers and CAI have already been widely integrated into the medical setting. It is impractical to think that this trend can be reversed. However, it is obviously undesirable for computers to totally replace traditional teaching methods. Both psychiatry and teaching are, after all, fundamentally humanistic endeavors. Therefore, it may be helpful to stop thinking of the question as “CAI versus traditional,” as if there would be an ultimate victor, and instead think of CAI as another useful tool available to the modern educator. Those skilled in traditional teaching methods know that there is no single “best educational method” for every situation; they vary their teaching based on the environment, material to be taught, and individual needs of learners at any given time.

In order to help medical educators understand how to use CAI in this manner, here are some suggestions: 1) focus future CAI studies on component parts of CAI rather than entire modules; 2) perform such studies with carefully controlled and randomized experimental groups so that it is clear how to interpret the data; 3) publish comparative analyses of cost, resource intensity, and time commitment of CAI so that educators have realistic expectations of the resources they will need; and 4) include clear details about the development of the CAI intervention so that others do not have to “reinvent the wheel.” Hopefully, these suggestions will lead to a more organized and efficient study of this technology, enabling educators to make informed decisions about how to apply this intriguing and promising technology.