In 2003, the U.S. Institute of Medicine released a report commissioned by the National Institute of Mental Health that noted, “The number of psychiatrist researchers does not appear to be keeping pace with the unparalleled needs that currently exist in clinical brain and behavior medicine” (1). A broader recognition arose, around the same time, that mechanisms need to be developed to facilitate research training, beginning in medical school or early in residency, rather than afterward (2). In addition to the decrease in the number of psychiatrists pursuing research, there is a general lack of training and knowledge among teachers of psychiatry residents about evidence-based medicine (EBM) and how to teach research literacy (RL) (3–6).

In response to these needs, a conference series was designed with R13 funding to assist training directors in achieving these goals. This was provided through a forum dedicated to teaching basic EBM and RL skills to directors of psychiatry residency programs. This “train-the-trainer” series began in 2006 as a day-long meeting held on the day before the annual meeting of the American Association of Directors of Psychiatric Residency Training (AADPRT), and continued through 2010; thus, the gatherings are referred to as “pre-meetings” throughout this article.

The focus on EBM and RL skills was intended to serve a dual purpose. First, it was anticipated that training psychiatry residency faculty in such techniques would have a downstream effect, whereby they would engage their residents with these same methods, and those residents would, in turn, become more interested in pursuing careers in research. Secondly, in focusing upon basic, rather than higher-level statistics or study-design issues, which can often alienate those less experienced in research, the series was intended to make EBM and RL something that everyone would feel comfortable pursuing as Life Long Learning and Maintenance of Certification (MOC). This latter issue is of growing importance, since MOC is required every 10 years of all those who have graduated since 1994 (3–6).

The purpose of this article is to present an assessment of the series by examining data from evaluations collected at the conclusion of each pre-meeting, as well as data collected from an online survey distributed before the very first pre-meeting, in 2006, and between each subsequent pre-meeting, in order to produce an overall longitudinal impact assessment. Funding ended shortly after the 2010 pre-meeting, so the longitudinal impact of that pre-meeting could not be included in this article.

There were expectations that this program would result in several specific outcomes, namely, that participants would 1) enjoy the program overall; 2) improve their knowledge on the topics presented; 3) utilize learned information through teaching and distributing material in their institutional training programs; and 4) add more training in research and EBM to their own residency training programs. This study was meant as an open clinical assessment to stimulate both more controlled research and changes in teaching habits that have primarily gone untested in residency training until now.

In addition to teaching EBM and RL techniques throughout all 5 years, each pre-meeting included content that highlighted a particular mental illness or psychological condition. These included the treatment of complicated grief (2006), memory problems (2007), depression (2008), schizophrenia (2009), and attention-deficit hyperactivity disorder (2010). Although it was likely that training in these topics was already included in participating institutions’ curricula, the subject matter provided a means through which new teaching modalities relating to EBM and RL could be introduced. Working within familiar topics also allowed for the introduction of statistical concepts important in understanding the research literature, such as judging validity and reliability of measurements (2006); sensitivity, specificity, and likelihood ratios (2007); how to assess papers on meta-analysis (2008); t-tests, ANOVAs, and LOD scores (2009); and study designs (cohort, case–control, cross-sectional; 2010).

The general goals of the pre-meetings were to simulate activities that were available to residencies of various sizes and to give participants hands-on experience with being taught in those ways. Thus, each pre-meeting consisted of a mix of small-group activities (groups ranged in size from 15 to 20 and 30 to 40) and large lectures involving up to 150 participants. The meeting planners used as many opportunities as possible to not just have participants observe but to practice and, in some cases, to teach others by presenting “team” findings to each other within the small groups.

Small-group activities included learning to critically appraise the research literature by reviewing articles together and using worksheets to identify the components of each article. In reviewing each article, there was a special emphasis on discussing what impact it might have on the clinical care of present or future patients. Small-group facilitators also gave mini-lectures, which provided an opportunity for participants to ask focused questions on specific areas of study. These small groups were sometimes divided into smaller “teams” of 8–10 people so that attendees could work together to solve a problem or answer a question. For consistency, small-group facilitators were trained through e-mails, several conference calls, and a face-to-face meeting the evening before each pre-meeting.

In the large-group setting, several lecturers gave PowerPoint presentations in order to review the existing literature and new research. Other large groups reviewed a specific topic or activity, such as statistics or study design, that was about to be practiced in the small-group “teams.” Another large-group activity involved a demonstration on how to search for articles in PubMed and how to use the Cochrane Database of Systematic Reviews. It was assumed that many attendees already had some knowledge of these tools, so these firsthand demonstrations were intended to present additional useful tips.

A final technique used in both large and small groups was presenting specific instruction on how to use the books and CDs that were distributed to attendees each year. The CDs provided at each meeting contained all of the materials used in that year, so materials could be reproduced for teaching purposes at attendees’ home institutions.

The 2010 pre-meeting warrants special mention, as it was conducted differently from the previous 4 years. The primary goal of every pre-meeting was to teach new instruction techniques; however, in 2010, a special effort was made to do this by introducing the concept of Team-Based Learning (TBL) (9–12). This was in addition to the traditional format used in the 4 previous years, which the authors are calling Teaching in Large and Small groups (TLS). Unlike the traditional classroom lectures utilized in TLS, which places greater emphasis on the acquisition of knowledge, TBL allows more time for participants to apply knowledge, rather than simply acquire it, and involves participants working together in small teams to answer questions and solve problems. In order to measure the difference in impact on learning that TBL might have had, a randomized controlled study (RCT) was conducted. Using a process of randomization during pre-meeting registration, attendees were assigned to one of the two teaching formats, with approximately 50 attendees receiving the day’s material using TBL and 64 being instructed in the TLS format. Evaluations tailored to each curriculum were collected at the end of the TBL and TLS instruction periods. The closing session was retained as a time to bring groups together for a lecture and summary on the dynamics of TBL and the teaching resources it can provide.

Data for this study were collected via two sources: Anonymous pre-meeting evaluations, which were distributed and collected on-site at the end of pre-meeting, and an online between-pre-meeting survey, managed via SurveyMonkey.com, which was sent to all AADPRT members before the first meeting in 2006 and between each subsequent pre-meeting in order to study longitudinal impact. The evaluations and surveys were submitted to the Institutional Review Board (IRB) of SUNY Upstate Medical University and Keck Medicine of USC for consent approval in 2006 and 2008, respectively. Both instruments were granted an exemption from participants’ consent because they were being used to assess an educational program.

Based on a Likert scale of 1–7, pre-meeting evaluations asked respondents to rate the conference overall (1: Poor; 7: Outstanding), establish self-perceived pre- and post-meeting level of familiarity with the material presented (1: No knowledge; 7: Thorough knowledge and use), and report the likelihood of their using the material introduced (1: Definitely won’t use; 7: Will definitely use) (7).

Those who responded to the between-pre-meeting survey were asked whether various teaching methods or skill areas learned during the pre-meetings were used within their home programs. Of particular interest was the use of journal clubs and whether or not they were required, didactic training in the critical appraisal of literature, basic research-methods programs, training in evidence-based medicine, and residents’ participation in faculty research (see Table 1).

Because of the descriptive nature of the collected ordinal data, the evaluation and survey results are presented here as relative frequencies (percentages) of specific ratings on the 1–7 Likert scale (7, 8). In many cases, ratings between 1–3 and 5–7 have been collapsed in order to present percentages of respondents who rated certain items below or above average. Because 177 of the approximately 180 general psychiatry residency training programs in the U.S. attended at least one of the five pre-meetings, there was no control group for this study.

The pre-meeting evaluation response rate each year was high, with close to 96% of attendees returning evaluation forms in each of the 5 years. There was a consistent core of about 70%–80% of the approximately 180 general psychiatry residency training programs in the U.S. at all five pre-meetings, and over 98% of all general psychiatry residency programs were represented at at least one of the pre-meetings.

Data from these evaluations revealed that, regardless of year, the majority of respondents endorsed the use of material and topics presented at the conference series. Each year, the overall conference rating was routinely moderately high, with at least 90% of respondents reporting ratings between 5 and 7 (scale of 7). Every year, the majority of respondents reported a moderately high rating for their likeliness to use the material that had been covered in the pre-meeting. Overall, 86% of respondents reported that it would be likely (rating of 5 to 7) that they would use the material covered in large groups, and 81% of respondents reported the same ratings for material covered in small groups. As was anticipated, at the end of every pre-meeting, respondents reported an improvement in their understanding of the material presented. The majority of respondents rated their knowledge between 3 and 5 before each pre-meeting, and, at the end of each pre-meeting, the majority rated their knowledge between 4 and 6.

While the addition of the RCT changed the format of the 2010 pre-meeting slightly, data from that year reflect identical trends as the previous years. Respondents in both the TBL and TLS groups reported a moderately high likelihood to utilize what they had learned, and moderately high satisfaction with the conference overall, with 91% of respondents rating it between 5 and 7 (scale of 7). Although conference attendees were randomly assigned to each group, survey respondents in the TBL group reported having less of an understanding of the subject before the pre-meeting than those in the TLS group, but, after completion of the day’s instruction, the rate of knowledge increase between both groups was the same. Whereas the same general trends were observed with respect to former pre-meetings, in one key rating, respondents seemed to slightly favor the TBL over the TLS curriculum: TBL respondents reported a greater likelihood of using the conference material; 89.1% reported it would be likely that they would use the material (rating of 5 to 7), as compared with 83.8% of TLS respondents who reported the same ratings. This outcome coincides with numerous articles showing statistically significant results that have found TBL to be both a more desirable and an equally or more effective method of teaching and learning (9–12). More details of the difference between TBL and TLS will follow in a subsequent article about the 2010 pre-meeting.

Although the response rate for the conference evaluations was consistently high, the response rate for the online between-pre-meeting surveys was considerably lower, with between one-third and two-thirds of attendees responding. Response rates for individual survey questions were also highly variable. A number of respondents exited the survey at early stages, and, since answering one question was not required to move on to the next, some respondents skipped certain questions. Thus, in some cases, only 30%–50% of the total number of people responding to the survey may have answered any one question. It is for this reason that the N values in Table 1 and Figure 1 are particularly small.

Several supplemental survey questions were added to the final between-pre-meeting survey that was distributed between the 2009 and 2010 pre-meetings. These questions sought to gauge the overall impact of the conference series by asking attendees to what extent they had implemented what they had learned over the previous 4 years of the pre-meetings into their training programs.

As shown in Table 1, despite differences in the number of people who took the survey, 92.6% of respondents in the between-pre-meeting period 2009–10 reported that their residency training programs included EBM teaching in their curriculum, compared with only 75.2% in pre-2006. As illustrated by Figure 1, a total of 82.4% of survey respondents indicated that they had integrated more EBM teaching into their didactic content curriculum as a result of attending at least one pre-meeting, whereas a smaller percentage had incorporated the techniques into individual supervision (39.2%), group supervision (22.3%), and clinical rounds (34.5%). Of the respondents who reported that EBM training had been added to their residency training programs, 59.3% had added training to the first postgraduate year (PGY-1); 65.5% had added training to PGY-2; 62.8% had added training to PGY-3; and 43.3% had added training to PGY-4.

As mentioned above, the topics covered in 2006–2009 were the treatment of complicated grief, memory problems, depression, and schizophrenia. Survey results showed that roughly 48% of respondents reported they had enhanced their curriculum by adding at least some EBM and RL teaching within those specific content areas as a result of attending at least one pre-meeting.

Finally, each year, a portion of the budget was set aside to provide take-home materials to attendees. These included textbooks and CDs containing research articles, slideshows, and other items presented or used in preparation for the pre-meeting. The last of the questions in the final between-pre-meeting survey examined how this material was utilized once attendees returned to their home institutions. More than 70% of respondents had made the textbooks and CDs a part of their establishment’s library, and a notable portion (32%) assigned chapters and articles to be read by residents.

The vast majority of attendees described the conference overall as “very good,” “excellent,” or “outstanding.” Whereas most individual sessions of the pre-meeting were rated equally as favorably, the material in the large-group lectures was rated as slightly more useful than what was presented in the small-group breakout sessions. Each individual instructor’s teaching performance might have had an effect on the reported usefulness ratings; however, since most instructors received at least an above-average rating, any difference in their impact was likely negligible. Also, respondents were overwhelmingly confident in their retention of the “take-home” message and their ability to institute and incorporate that information into various aspects of their home institutions’ residency programs.

The between-pre-meeting survey data demonstrate that, although not all survey respondents necessarily added new EBM and RL teaching relating to specific pre-meeting topics, the conference series was useful to them in providing teaching enhancements to their training programs. These included teaching tools, case examples, methodologies for use in existing journal clubs, and didactics on research methods (Table 1 and Figure 1).

Although these preliminary reports are very encouraging, there are weaknesses in the evaluation of these data that must be addressed. Data were collected via an ad-hoc process, and with very limited funding to conduct a formal evaluation of the full 5-year program. The survey instruments, which needed some modification each year depending on the topic, were only beta-tested by conference facilitators and presenters before distribution to attendees; thus, a true validation process with attendees was not conducted.

A low response rate has proven to be a major issue in the analysis of between-pre-meeting survey data. With only one-third to two-thirds of attendees answering the survey between pre-meetings, it has not been possible to accurately assess the impact this conference series has had on all participating programs. In order to acquire a higher response rate, the authors recommend distributing hard-copies of between-pre-meeting surveys at the beginning of the next pre-meeting, rather than using an online survey, since the hard-copy evaluations had such a high response rate.

Another major setback in the assessment of the conference series’ impact is the anonymous nature of the collected data, which limited the authors’ ability to compare the program’s impact within and between specific training sites. If this program is continued, the authors recommend that institutions’ responses be tracked from year to year through the use of anonymized identification codes. Another setback with the anonymity of the data is that it was impossible to track which institutions did not respond to or only partially completed the survey. Those non-responding institutions may have employed additional curricular components at rates different from those who replied. The existence of such a difference would imply an inconsistency in training standards, yet it was impossible to investigate this further without anonymized tracking of responses.

Also, without extension of this funding beyond 5 years, it was difficult to establish effects on residents, both in terms of the use of EBM strategies in clinical practice and in terms of increased rates of research track selection. Most of the residents who may have benefited from the lessons that their training directors took home over the 5 years have only just finished their residency training or have only been out in practice for 1–4 years. Further study dedicated to tracking specific program changes and the status of residents after completion of their training can help to highlight the longitudinal impact of this program.

Another flaw in the evaluation of these data are the lack of a control group. An annual survey collecting a formal response from each institution and residency program, including those that did not attend any of the pre-meeting conferences, would have been ideal for measuring group differences between attendees and non-attendees. However, the popularity of the program meant that over the 5 years of pre-meetings, approximately 177 of the nation’s 180 general residency training programs were represented at least once, and thus the creation of a “non-attendee control group” was impossible.

Despite the shortcomings in the evaluation of these data, the authors are confident in saying that those who attended these pre-meetings gained useful and effective knowledge and used that knowledge to improve their institutions’ training programs. The series provided attendees with useful content, teaching techniques, and assessment tools for curriculum development in research literacy and evidence-based medicine.

The authors thank Christianne Joy Lane, Ph.D., who assisted with data analysis.

This publication was supported by NIH/NCRR SC-CTSI Grant Number UL1 RR031986. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.