In 1991, the American Psychiatric Association (APA), through its Assembly and Board of Trustees, began to develop Practice Guidelines for the treatment of DSM-IV psychiatric illnesses. "Practice Guidelines" are systematically developed, standardized documents that present patient care strategies to assist psychiatrists in clinical decision-making. Having had great success in standardizing the diagnostic process with the Diagnostic and Statistical Manual, the APA hoped to provide clinicians with pragmatic considerations to aid treatment planning.
The process by which these guidelines were established differed markedly from those commonly used by authors of textbooks. Although many textbook authors discuss empirical research as they suggest strategies for treating psychiatric illness, the Practice Guidelines are notably more comprehensive. For this study, we used the nine guidelines in effect at the time. These were the guidelines for 1) Psychiatric Evaluation of Adults; 2) Eating Disorders; 3) Major Depressive Disorder; 4) Bipolar Disorder; 5) Substance Use Disorders: Alcohol, Cocaine, Opioids; 6) Panic Disorder; 7) Schizophrenia; 8) Nicotine Dependence; and 9) Alzheimer's Disease and Other Dementias of Late Life. Since then new guidelines for Delirium, HIV/AIDS and Borderline Personality Disorder have been issued, and ones on Posttraumatic Stress Disorder/Acute Stress Disorder and Management of Suicidal Behaviors are in development.
The APA Steering Committee on Practice Guidelines selects a small number of individuals as a work group on the particular disorder. The work group chairs and Office of Quality Improvement staff develop preliminary outlines, which are continuously revised. A comprehensive literature search obtains relevant articles, which are coded by study design. Evidence tables for each treatment are compiled. Four drafts are constructed during the development process, involving participation by the Steering Committee, some 50 expert reviewers, the Board of Trustees, the APA Assembly, Joint Reference Committee, Council on Quality Improvement, Council Chairs, Commission on Psychotherapy by Psychiatrists, Committee on Women, District Branches, individual members, the American Journal of Psychiatry, and 100 representatives of related organizations.
The evidence base for Practice Guidelines is derived from two sources: research studies and clinical consensus. Where gaps exist in the research data, clinical consensus is garnered from the extensive review described above. Both research data and clinical consensus vary in their validity and reliability for different clinical situations; guidelines specify the nature of the supporting evidence for specific recommendations so that readers can make their own judgments regarding the utility of the recommendations.
Just as the DSM series has found widespread acceptance, it may be anticipated that the APA Practice Guidelines will be widely accepted as well. For this reason, and because the guidelines are inherently informative, psychiatric educators should convey this material to trainees. The purpose of our study was to efficiently and effectively teach the guidelines to residents. To that end, we introduced the guidelines into our residency curriculum in a novel fashion, using small teams of residents to teach each other. The challenge was to determine the best way to construct teams, institute the learning paradigm across multiple training sites, and assess this strategy by using pre- and post-test outcomes.
Our teaching method differed from most residency or medical school teaching methods. Little research has been done on its efficacy. It is, in essence, a student group preparation process followed by an intergroup teaching approach.
Many teaching styles have been employed with medical students and residents, including lectures, classroom, case-based methods such as vignettes or videotapes of live patient interviews, short stories and commercial movies, teacher or student-led journal clubs, teacher-led bedside teaching, rounds and case conferences, and teacher or student-led seminars.
An important teaching technique is student-to-student teaching. This method can be difficult and challenging for teachers who do not control these sessions yet have to steer the learning process. Our literature review found only one report of student-to-student teaching in medical education (
+1).
Although useful in psychiatry clerkships (
+2), videotape learning is usually passive, with little interaction between the teacher and trainee. The absence of videotapes based on the guidelines precluded this technique. Computers can be successfully used by medical students to learn psychiatry (
+3). Statistically significant improvements in post-test scores were shown in studies using personal computers for teaching medical students liaison psychiatry (
+4), histology (
+5), pharmacology (
+6), and pathology (
+7). Although a computer-based approach was a possibility for teaching the guidelines, we lacked the expertise and resources for designing such software and implementing such a strategy. Problem-based learning (PBL) is a method that was innovated at McMaster University in 1960s as a response to the widespread criticism of conventional medical education. Data from individual studies suggest that PBL is a useful method to teach ethics (
+8) and psychopathology (
+9) and to impart skills in forensic psychiatry (
+10).
Although film dramas, documentaries, television shows, and other visuals are powerful teaching tools, none of these methods seemed to suit our purpose. Various films such as
The Breakfast Club or televisions series such as
Buffy the Vampire Slayer have been used to depict suicides, suicide attempts and their risk factors (
+11), and adolescent issues (
+12). Other visuals, such as charts and pictures, can also be effective medical student teaching aids (
+13,
+14).
Journal clubs (
+15), blind role-playing (
+16), the use of short stories (
+17), traditional bedside teaching (
+18), the Objective Structured Clinical Examination (OSCE) (
+19,
+20), and "mini-boards" (
+21) have all been introduced as teaching methods in psychiatry. Yet none of these techniques seemed to offer us an efficient means for teaching the guidelines.
Given our 12- to 18-month study period and the distribution of residents across multiple sites, we perceived that a student-to-student teaching method might best achieve our goal. To that end, we established interyear, intersite practice guideline teams. Members of each team would teach fellow residents, who were not from that team, at lunchtime meetings at their training site. We would assess the efficacy of this method by pre- and post-study tests of knowledge. No Institutional Review Board review was required for our project, and no decisions about advancement in the program were made on the basis of performance.
Thirty-seven residents participated in the small-group cross-site teaching exercise. The trainees work primarily at three academic medical centers located approximately one hour by car from each other. Of these 37 residents, 27 participated in the pre- and post-test examination: nine from PGY-1, seven PGY-2, eight PGY-3, and three PGY-4. Except for vacation or illness, no residents were allowed to opt out of the project.
About one-half of our residents are at a Veterans Affairs Medical Center, one-third at an urban medical center hospital, and about one-sixth at a state hospital. Although all residents meet together on Thursdays for a teaching day, there was not enough open time to plan and execute our practice guideline teaching strategy on that day, given the considerable breadth of the residency curriculum. Therefore, each guideline team contained residents from each PG year and from each site. We assigned more residents to the lengthier guidelines and fewer residents to the briefer ones. Each guideline team had a minimum of three residents assigned to it. For example, the Nicotine guideline team had three residents, from PGY-2, 3, and 4, each from a different site. The Bipolar Affective Disorder guideline, which is much longer, had five residents assigned to it: two from PGY-1 and one each from PGY-2, 3, and 4. All three sites were represented within this team as well.
+
Teaching Process and Materials
The program director selected a PGY-3 team leader to coordinate each guideline team. We preferred PGY-3 rather than PGY-4 residents because graduating residents tend to focus on job or fellowship arrangements for the last few months of residency. Each team leader was chosen on the basis of his or her interest in the project, perceived leadership skill, and perceived grasp of what would be expected in terms of implementing the project. These decisions were made by the program director (D.G.). There was no monetary or other type of inducement for the team leader other than the honor of being chosen for the task. The PGY-3 team leader was expected to assemble a synopsis of his or her particular guideline. To divide the work pragmatically, the team leader assigned segments of the guideline to each team member to summarize. The work was divided such that it took each resident about three to five hours to summarize his or her part of the practice guideline. The team leader then assembled the entire synopsis and edited it with the help of the training director. Each team leader spent about five hours on the task.
The nine synopses formed the basis of the resident-to-resident teaching. Prior to the actual presentations, each resident prepared a computer slide presentation of the guideline synopsis. This took about three hours. In this way, no one resident was burdened with an undue amount of work. The residents were motivated to participate in this task because it was viewed as an important residency-wide teaching and learning exercise strongly endorsed by their training director, vice-chair, and chair. They believed that learning the guidelines would help them be better prepared for the Psychiatry Resident In-Training Examination (PRITE) and American Board of Psychiatry and Neurology (ABPN) examinations.
Every other Friday, at lunchtime, at each site, one member from a given guideline team presented it to all on-site residents. For example, on a given Friday in February, three separate presentations on the Panic Disorder guideline were offered—one at each site. Each presenting member constructed his or her own slide presentation of the guideline for his or her session. A copy of the team synopsis was distributed to each audience member. Although each presenter had a slightly different slide presentation from the other two presenters at the other sites, they were all developed from the same synopsis and audience members at all the sites had a copy of the same synopsis in front of them during the presentation.
Most of the residents had to learn how to use the software and hardware, which they did not find difficult, were eager to learn, and perceived as enhancing their computer literacy.
+
Test Methods and Data Analysis
To determine the efficacy of this program, we gave a pre- and post-program examination. A one-hour, 50-question multiple choice test was administered prior to establishing the practice guideline teams. After development of the fourth draft of each practice guideline and before its final approval, APA Work Group members develop continuing medical education (CME) questions based on the guideline. These questions were used in our pre-test examination and were converted into clinical vignettes in our post-test reexamination. Slightly more questions were chosen from the longer guidelines and slightly fewer from the shorter guidelines. One of us (N.A.V.), with extensive experience in test design, chose the questions at random. Using Case and Swanson's criteria for examination questions, she converted the content of the pre-test item stems into clinical vignettes. In this way, rote recognition and memory of the pre-test questions minimally influenced post-test outcome. An independent statistical consultant reviewed the test methodology and concluded that scores on the pre-test exam were equivalent to scores on the post-test exam. The difficulty and discrimination indices for each question were calculated during the item analysis of the test. Each of the questions was determined to be a valid question.
Twenty-seven of the 37 participants (73%) took both the pre- and post-study examinations. Of the 10 that did not take the post-program exam, 1 (PGY-4) had graduated from the program early, 5 (3 PGY-4, 1 PGY-3, 1 PGY-2, 1 PGY-1) were on vacation, and 2 (PGY-3, PGY-1) were sick. None of these 10 participants' pre-test scores were significantly different from those of their PGY peers.
The pre- and post-study examinations were administered and scored by using the medical school's standardized Scantron answer sheets. For each exam, each resident received a raw score, a percentage correct, and a standard T score. Individual resident scores were then aggregated by PG year. Pre- and post-study test scores were compared with a paired-samples t-test.
In this survey, about half the participating residents were questioned about their subjective feelings and attitudes concerning the program some nine months after its completion. Because some participating residents had graduated and others were unavailable, we were able to survey only 17 of the 36. The survey was divided into two parts. In the first part, participants were asked to grade different components of the program with an A, B, C, D, or F. In the second part, they were asked if they agreed or disagreed with a given attitude about the program.
Of the 27 completers, 12 were women and 15 were men. There were 3 American medical graduates and 24 international medical graduates. Four residents had one or more years of psychiatric training in another country prior to their matriculating in our program.
+Table 1 shows pre- and post-study test results for the 27 residents taking both exams. The scores presented in
+Table 1 represent percentage of correct answers on the 50-question, multiple-choice pre-test and the 50-vignette multiple-choice answer post-test. The first-year residents increased their scores from 52.4% to 79.6%; the second-year group increased their scores from 63.3% to 79.7%; the third-year trainees increased their scores from 70.3% to 82.4%; and the fourth-year residents increased their scores from 68.6% to 84.3%. Each resident participating in the study had a statistically significant increase in examination scores from pre-study to post-study.
+Table 2 shows the results of the follow-up survey taken about 9 months after the end of the study period. Several items stand out. The program director's participation in the overall process is felt to be quite important by the trainees. Although the actual work of making the synopses and the PowerPoint slides was not felt to be of the highest value, the vast majority felt that the project did enable them to learn the guidelines well and felt that their presentations and the other resident group presentations were of high quality. Notably, almost three-quarters of the participants felt that what they learned from the guidelines project was consistent with what they learn in their other classes and, perhaps most important, consistent with what goes on in their clinical work in their hospitals and clinics. A substantial majority, 76%, felt that the resident-to-resident teaching method was a useful strategy, and more than 83% felt that the cross-team teaching method that we introduced was worth their time and effort.
This study demonstrates that an important body of knowledge can be instilled into a psychiatry residency by using a novel teaching approach that was synchronous with the department's clinical and academic programs. This approach has many advantages. First, teaching in teams increases the likelihood that less convinced residents will "buy into" the importance of the material. The message is reinforced by having a respected PGY-3 resident as the team leader. Second, a tone of friendly competition and pride in the teamwork product develops, which enhances the work product used for teaching. In our study, the preparation of the synopses was the focal point for the team product. Different residents put varying amounts of time and energy into their contribution; total effort ranged from 5 to 25 hours. Yet almost all of those surveyed felt that the effort was well worth it. The team leader role helped to even out whatever inconsistencies there might have been between work products. In this system, the role of the team leader was extremely important. The incentives and motivation for participation revolved around forces of peer pressure, the desire for recognition and respect, and a sense of belonging, competitiveness, and "school spirit." The trainees knew they were participating in a demonstration project connected to the American Psychiatric Association, and this enhanced their desire to participate and gain recognition for their school and program.
The issues of residency cohesiveness and learning new, generalizable skills (PowerPoint slides and Word outlines) were a "perceived value" that we thought would be of interest to the trainees, but this hypothesis was not borne out by the follow-up survey. Almost all residents perceived that the APA Practice Guidelines would eventually become de facto standards for future PRITE and ABPN exams, and possibly even hospital and clinic standardized work formats. Thus, they believed that their smaller efforts now might yield high benefits later.
Another reason the pre- and post-study scores were significantly different may have to do with the process of knowledge acquisition itself. In the process of condensing a prepared text into a synopsis, the information becomes one's own. The synopses were abridged versions of the guidelines. They state conclusions and leave out the substantiating studies. Thus, we had some 37 residents actively translating the unabridged practice guidelines into their own condensed versions. This may have played a significant role in good post-study scores.
We did not assess the impact of this learning exercise on actual clinical performance, an area for future research. To what extent does this kind of learning translate into treatment planning on the wards and in the clinics? Are residents better able to construct comprehensive interdisciplinary treatment strategies as a result of this learning? Our follow-up survey addressed whether what residents learned in the project was consistent with what was expected in the clinical situation, but more needs to be determined about whether the residents actually became better treatment planners. This could be done by a systematic review of charts or by looking at supervisor ratings of treatment planning skills.
Thus, this study focuses on two important issues in teaching. First, resident-to-resident teaching can be quite effective. Perhaps the old saw that "the best way to learn something is to have to teach it" applies to the results of this research. Second, the knowledge gained by this method can be assimilated in a way that allows it to be applied to novel situations, as was demonstrated on the trainees' examination by clinical vignette in the post-study test. We believe that training programs may want to consider more resident/ peer teaching as an integral part of their curricula.
There are also important limitations to our study. The sample size of some our PGY groups was small; for example, the PGY-4 group had only 3 residents. Furthermore, the mix of ages and sociocultural backgrounds of our trainees may not be representative of all programs, and thus some caution must be used in extending the results of this study to all psychiatry residency programs.