In July 2005, the University of Colorado Department of Psychiatry Evidence-Based Medicine (EBM) Project began to investigate whether formal educational interventions could help residents develop a positive attitude toward EBM, acquire EBM knowledge and skills, and facilitate the daily use of EBM with patients in a psychiatric outpatient residency-training site. We developed our curriculum and teaching approach by reviewing the world literature, three published EBM curricula (1–3), and additional EBM curricula presented at American Association of Directors of Psychiatric Residency Training (AADPRT) meetings. In December 2006, we found no published comprehensive EBM curricula from any specialties reporting effectiveness data.

Two common approaches to EBM education found in the literature include the use of problem/case-based learning methods (4–9) and adult learning theory (10). These approaches emphasize active learning to link knowledge and skill with clinical practice. In 1995, a systematic review of 102 trials (8) revealed that didactic approaches had little effect on physicians’ clinical practice. A recent 2004 meta-analysis (9) reviewing 18 studies using standalone EBM teaching methods demonstrated that these methods improved EBM knowledge, but failed to change attitudes toward EBM or foster clinical use of EBM with patients. Table 1 reviews empirically validated, clinically integrated teaching methods (9, 11–14), utilized by nonpsychiatric colleagues, fostering clinical use of EBM.

We believe our study may be unique as the first prospective psychiatry study designed to research the cumulative effectiveness of a curriculum with educational interventions. The goal is to significantly increase resident knowledge, skills, and clinical use of EBM with psychiatric outpatients. The project includes curriculum development, implementation of four educational interventions, development of two attitude questionnaires, development and validation of a psychiatry EBM knowledge exam, and curriculum evaluation using a pre/post-intervention design. This article describes the curriculum, study instruments, and preliminary evidence about the program’s effectiveness using data from 37 psychiatry residents who voluntarily enrolled in the study.

Psychiatry-EBM Curriculum

The resident curriculum is progressive over 3 years (postgraduate years 1–3) and follows a traditional EBM approach of teaching the 6As (Assess a patient, Ask a clinical question, Acquire the information, Appraise the information, Apply the information with a patient, Assess the outcome with the patient) (3–5). Residents learn and practice assessing a patient, asking an EBM question using the PICO-QQ format (Population, Interventions, Comparison group, Outcome, Question type, Quality of the study), acquiring the information using online library searching of textbooks and the world literature, appraising the literature using EBM mathematics, applying the literature with a patient, and assessing patient outcomes. All courses are highly interactive and use computers, mini-lectures, problem/case-based learning, and resident teaching.

The PGY-1 course consists of three weekly, 2-hour sessions, designed to introduce EBM and create interest in the 3As (Assess a patient, Ask a question, Acquire the information), while avoiding EBM mathematics. The content addresses the question, “What is EBM and why use it?” as well as assessing, asking, and acquiring articles from online searches.

The PGY-2 course meets weekly and consecutively for 6 hours. Two sessions are devoted to practicing the 3As. Residents choose one randomized controlled trial to evaluate. Three sessions focus on critical appraisal skills and an introduction to EBM mathematics (1–6). The last session covers outcomes and application of information with patients, with consideration of patient values and preferences.

The PGY-3 course consists of 25 75-minute classes. The first 15 sessions solidify use of the 6As and EBM mathematics. Faculty members model an ideal case for two sessions. In 13 sessions, residents present their own cases to practice and teach all 6As. With faculty support, residents lead critical appraisals of therapy, use EBM mathematics, and learn to use outcome measures. The remaining 10 sessions use the 6As with practice guidelines, systematic reviews, and a wider variety of outcome measures. Practice guidelines are critically appraised, using the AGREE (15) assessment. Meta-analyses/systematic reviews are critically appraised using guidelines described elsewhere (4–6, 16).

Clinically Integrated Teaching Activities

Use of EBM in daily psychiatric care is strongly emphasized in the PGY-3 outpatient year. Midway into the PGY-3 course, we add four educational interventions designed to facilitate residents’ clinical use of EBM.

Intervention 1: Index Case and EBM Reminder Survey

Residents in PGY-3 participate in a 3-hour session. In the first hour, the resident sees a patient called the “index case.” In the subsequent hours, the resident discusses with faculty any knowledge gaps in using the 6As. Together they set individualized learning goals. By the end of this session, the resident will have selected an outcome measure, from a rating scale book (17) or CD, which will be used with the index patient. The resident is also introduced to the EBM Reminder Survey and completes the first survey detailing usage of EBM practices during the session.

Intervention 2: Outcome Measure at the Index Patient’s Second Visit

During the second visit with the index patient, the resident explains, negotiates, and begins use of an outcome measure with the patient. Using the EBM Reminder Survey, the resident details EBM activities during this and all subsequent patient visits.

Intervention 3: Four Additional Resident Cases: EBM Reminder Surveys

After completing the initial index patient visit, the resident chooses four additional cases. A staff member from medical records attaches the EBM Reminder Survey to the front of the additional charts for the next five visits of each patient. After each patient visit, the resident completes a survey. Over many months, each resident can complete a total of 25 surveys.

Intervention 4: 1-Minute-Preceptor

Once PGY-3 residents begin working with their index cases, four EBM attendings, who precept all PGY-3 residents, begin using a 1-minute preceptor “microskills” (14, 18, 19) approach, which is supportive of EBM. The attending tries to elicit a commitment from the resident regarding a case formulation and treatment plan, probe the resident’s thinking and evidence supporting all decisions, teach something new about EBM, reinforce what was done correctly, and correct mistakes.

Environment for Implementation

A critical component of curriculum implementation involves availability of faculty knowledgeable in EBM. With only one expert EBM teacher, we formed a four-member EBM faculty self-teaching group, designed to help all faculty learn EBM and prepare to co-teach EBM courses. We also scored exams and supervised in the outpatient clinic using microskills. These attendings all work in the outpatient department, a major teaching site for all residents. Close resident-faculty working relationships may have influenced the 100% willingness of 37 eligible residents to enroll, with only one resident opting out of the clinical component of the study. Senior psychiatric research and clinical faculty volunteered to take, and completed, an early version of the EBM exam, revealing a very receptive environment.

Curriculum Study

Based on the literature and prior experience teaching EBM to residents, we assumed that residents would have little prior exposure to EBM. We hypothesized that attitudes toward EBM and EBM knowledge, as measured by the Colorado Psychiatry Evidenced-Based Medicine Examination (CP-EBM Exam), would increase progressively with this curriculum. After six sessions, we expected residents would do more searching using PICO-QQ. After 25 sessions, we hypothesized that R3 scores on the CP-EBM Exam would substantially increase. Furthermore, we hypothesized that application of EBM with patients would not significantly increase until PGY-3 residents had a 3-hour teaching session, saw patients, completed the EBM Reminder Survey, and received feedback from attendings using the 1-minute preceptor. To evaluate these hypotheses, we conducted a prospective pre/post-intervention study examining changes in resident attitudes/beliefs, knowledge, and clinical use of EBM at multiple time points.

Instrumentation

Three instruments were prospectively designed to assess program impact: the EBM Attitudes and Beliefs Survey; the EBM Reminder Survey, designed to prompt and track resident use of EBM with patients; and an EBM knowledge exam.

The EBM Attitudes and Beliefs Survey is a 72-item questionnaire assessing prior computer use and experience, prior experience with EBM, perceived importance of using EBM, perceived confidence in EBM skills, and perceived barriers or facilitators to EBM use. The attitude questionnaire is administered before and after each course and after clinical teaching interventions.

The EBM Reminder Survey is placed on the front of patient charts, prompting residents to use EBM during clinical care. Nine questions ask residents to detail their use, or nonuse, of EBM practices after each of 25 visits. Investigators are blinded to which residents complete the surveys and any identifying patient information, except for the initial index patient visit.

The Colorado Psychiatry Evidenced-Based Medicine Examination (CP-EBM Exam) is a 14-question exam assessing EBM knowledge and skills. Based on the Fresno Test of Competence in Evidence-Based Medicine (20), the psychiatry version and scoring rubric were developed in consultation with Dr. Ramous. The 14 open-ended questions attempt to reproduce the 6As clinical thinking used during a patient encounter. The exam is administered to residents with an open time frame of 60–120 minutes.

To validate the exam and scoring rubric, at random we chose 8 out of 20 exams previously completed by PGY-4 or PGY-5 child fellows not enrolled in the study but receiving EBM instruction from the first author. Initially, raters independently scored the same exam and compared results on each of 14 items. We discussed all items on which our scorings disagreed. After three rounds, using three different exams, we agreed on all answers. Interrater reliability was obtained using our four investigators, scoring two rounds of five additional exams. Raters were blinded to resident and resident year. One common exam was embedded in each group of five exams for each rater, and raters were blinded to the common exams. Reliability was estimated using average interrater correlations on the total score and by the intraclass correlation. The interrater reliability for the two rounds of five tests was 0.95, and the intraclass correlation was 0.93. Given these high reliabilities, we were comfortable having a single investigator score resident exams. Preliminary EBM scores are reported by number of sessions, while pre/post-improvement in scores is reported in the Results section.

To validate the exam, psychiatry EBM experts were identified using the AADPRT list of e-mail addresses. All experts identified themselves as EBM teachers in their respective psychiatry residency training programs. Five experts volunteered to complete the exams via the honor system following the same test conditions offered to residents. Two investigators scored the expert exams. With 226 as a perfect score, four experts scored in the range of 211–221, and one expert scored 178.

The CP-EBM Exam, scoring rubric, and both survey instruments are available from the first author.

Sample

After 1 year of implementation, preliminary results of our attitudes and beliefs survey consist of data from four resident groups, totaling 37 residents in all 4 years. Table 2 details the number for each year, the number of residents who took and completed the attitudes and beliefs survey at different points in time, and the numbers from our sample that were available for our preliminary analysis.

Analyses

Attitude questionnaire responses are summarized in the Results section, using parametric statistics and pre/post analyses employing paired t tests. Given the limited sample with full data, results should be considered preliminary and cautiously interpreted.

Preliminary analysis of performance on the CP-EBM Exam’s knowledge portion was based on a group of seven PGY-2 residents. Results of these exams were analyzed by comparing means across groups based on number of EBM sessions attended using ANOVA. A paired t test estimated improvement in knowledge for the seven residents with exam scores prior to receiving any EBM training and after completing six EBM sessions.

EBM Attitudes and Beliefs Survey

Before any EBM session, all residents reported using the computer at least once a day at work and a few times a week at home. Residents in PGY-1 indicated more frequent computer usage than either those in PGY-2 or PGY-3 who had completed six EBM sessions. Likewise, a larger percentage of PGY-1 residents had some prior experience with epidemiology/statistics/EBM, some formerly working as research assistants or researchers.

In the questionnaire domain of importance of EBM for patient care, six items were analyzed. These items assessed reading and understanding the psychiatric literature, self-questioning about optimal patient care after each diagnostic assessment and routine visit, finding and applying the pertinent literature to the patient’s care, using risk/benefit calculations to make treatment decisions, and using measurable outcomes or rating scales to follow patient progress. Paired t tests of differences in level of importance, measured on a 5-point scale (1=not important, 5=extremely important) before and after six sessions (PGY-1/PGY-2) and before and after 25 sessions (PGY-3) showed no statistical differences. Residents rated that use of the literature, skills in finding and applying the literature, and self-questioning about best patient care to be “very important.” Residents found using risk/benefit calculations to make treatment decisions and rating scales to follow patient progress “important.”

In the questionnaire domain of residents’ level of confidence for performing 26 EBM skills, mean responses of residents prior to completing any sessions (PGY-1 and PGY-2), after six sessions (PGY-1–PGY-3), and after 25 sessions (PGY-3 and PGY-4) indicate an increase in confidence on all skills. Paired t tests were conducted to explore whether changes after six sessions (PGY-1 and PGY-2) and between 6 and 25 sessions (PGY-3) were significant. For significant results, see Table 3 . Because sample sizes are small, these results are tentative at best.

Fourteen residents completed data prior to and after completing six sessions. The mean reported level of confidence increased significantly (mean_pre=1.29, mean_post=2.29, p=0.01) only for their ability to “construct a well-formulated (six-part PICO QQ) clinical question.” Because only four residents had complete data after 6 and 25 sessions, no preliminary statistical analyses are appropriate. Even in this sample of four, an examination of areas where level of confidence exceeded 2 standard deviations in the pretest group, four is considered noteworthy. On average, residents participating in 25 sessions gained more than 2 standard deviations, compared with their level of confidence after participating in six sessions, in their level of confidence to perform six skills: understand the basic statistical concepts; clinical epidemiology and study design/methodology; determine the effectiveness of a treatment intervention by calculating risk/benefit numbers; write/summarize a concise, thoughtful, one-page critical appraisal of a therapy; determine if a relevant study can be applied to a patient; and utilize librarians and EBM supervisors as needed.

This questionnaire also included 14 items about EBM practices. Some questions were anticipated to change based on postgraduate year alone (e.g., number of patients seen each day), whereas others were thought to be subject to change as a result of improved EBM skills (e.g., number of article searches each week). Comparing residents after six sessions, only the number of background searches done each week significantly changed, with the number of literature searches declining. After 25 sessions, residents showed an increase of 2 pretest standard deviations in the “number of treatment decisions changed due to EBM in the last month,” from a mean of 0.75 to 2.5 (with 0=none, 1.5=1–2, 3.5=3–4, 5.5=5–6, 7=7 or more). These results are summarized in Table 3 .

Colorado Psychiatry EBM Exam

A perfect score on the Colorado Psychiatry EBM Exam is 226 points. We scored 17 resident examinations. Thirteen residents were in PGY-2, who received six EBM teaching sessions; two residents were in PGY-4 or 5, who received 10 teaching sessions; and two residents were in PGY-3, who received 25 teaching sessions. Descriptive statistics for groups receiving six sessions (13 PGY-2), 10 sessions (2 PGY-4–5), and 25 sessions (2 PGY-3) indicate that means increase with number of sessions, as expected (106.7 after six sessions, 144.5 after 10 sessions, and 195 after 25 sessions). This sample of 17 residents was too small for statistical analyses (n=13, 2, and 2, respectively), and therefore our data are only suggestive of trends we expected.

Because we are reporting preliminary data, we only scored seven out of the 13 PGY-2 EBM exams that were taken. A paired t test was estimated to assess improvement in resident knowledge after participating in six EBM sessions. Of seven residents, scored with both pre- and posttest data, performance improved significantly from a mean of 37.0 (SD=23.9) to a mean of 102.6 (SD=49.9), a significant improvement (t=2.814, p=0.03).

Our experience suggests that an effective, comprehensive EBM curriculum can be developed and delivered to psychiatric residents. Data on the effectiveness of the curriculum is generally favorable, although the limited number of residents completing questionnaires and exams requires that all results be interpreted as very preliminary. The effects of our clinically integrated interventions in PGY-3 are awaiting further data and analysis.

Keeping these limitations in mind, we discuss our results according to our initial assumptions and hypotheses. First, our assessment of prior exposure to EBM confirms our assumption that current residents in PGY-2 and PGY-3 have little experience. On the other hand, current residents in PGY-1 have greater experience with epidemiology/statistics/EBM and more experience working as research assistants/researchers. We are not sure whether this is a local phenomenon or whether it reflects a national trend of increasing EBM teaching in medical school. It has clear implications for the starting point in the curriculum of the future, because it reinforces the need for a knowledge examination to appropriately match teaching to resident needs.

Current attitudes data provide no obvious progression in perceived importance of EBM in patient care. This unanticipated result may be explained by the high level of importance ascribed to EBM prior to participation in the program. Changes in resident level of confidence reflect our hypothesis with specific areas of confidence paralleling curriculum topics.

Results from our attitudes survey indicate that when combining confidence and knowledge data, six sessions appear sufficient to introduce residents in PGY-2 to EBM and facilitate their knowledge and use of literature and PICO-QQ searching. Likewise, 25 sessions appear sufficient to extend PGY-3 residents’ level of confidence in their ability to understand EBM concepts/study design, determine the effectiveness of a treatment using EBM mathematics, write a critical appraisal, apply EBM with a patient, and regularly utilize librarians and EBM supervisors as needed.

Knowledge of EBM, as measured by the Colorado Psychiatry EBM Exam, indicates that even a limited, six-session curriculum can result in substantial improvement in searching skills. These trends in knowledge acquisition lead us to cautiously suggest that the sequence and depth of instruction appears appropriate.

The final tasks and results of our efforts to facilitate resident use of EBM skills with patients in daily practice are awaiting future data on the effect of these components.

We would like to thank Gretchen Guiton, Ph.D., for her invaluable support with survey designs and statistics.