Academic Psychiatry 25:68-76, March 2001
© 2001 Academic Psychiatry
The Design and Evaluation of a Group Research Experience During Psychiatric Residency Training
Anita H. Clayton, M.D. and
Adrienne E. Sheldon-Keller, Ph.D.
Dr. Clayton is at the Department of Psychiatric Medicine, University of Virginia, Charlottesville, Virginia. Address reprint requests to Dr. Clayton, Department of Psychiatric Medicine, Box 800623 HSC, University of Virginia, Charlottesville, VA 22908. e-mail: ahc8v{at}virginia.edu
|
ABSTRACT
|
A weekly, year-long research seminar, led by two clinical research faculty members, provides psychiatric residents with an opportunity to integrate clinical and research experience and "hands-on" familiarity with clinical research. The functional goal is production of an IRB-approved clinical research project design. The group approach is less intimidating for the residents, allows for individual flexibility while maintaining the project's momentum, is congruent with other seminars and courses, and encourages a cooperative approach to research. Evaluations indicate that the residents value the experience and that they increase their knowledge of the requirements and techniques of clinical research.
Key Words: Research Training • Group Experience
|
INTRODUCTION
|
"ppreciation of research methods..." is one of the ACGME [Accreditation Council for Graduate Medical Education] requirements for general psychiatry residency programs (1). Similarly, provision of a suitable research environment and opportunity to participate in clinical research are requirements for accreditation of training programs in internal medicine. Nonetheless, our experience suggests that research is most frequently incorporated into training primarily through didactic seminars only. Cooke (2) observed that inspiration and opportunity are keys to "the future success of clinician–scientists<..." Inspiration, he argues, rests primarily on close association with mentoring faculty. Opportunity requires the creation of research experiences within clinical training programs "without necessarily requiring more time," by promoting programmatic flexibility.
The development and evaluation of research experiences have been reported for other medical training programs. Thompson et al. (3) detail the logistical problems they encountered and overcame in designing a research experience for residents in obstetrics and gynecology. In their setting, the research experience for residents consists of each resident's independently conducting a study under the supervision of a faculty member. A potential problem was highlighted in the report of a survey of structured resident research training in physical medicine and rehabilitation (4), which found that use of mentors outside of regular department faculty negatively affected the level of resident research activity.
Much of the literature on research experiences for residents describes programs designed to accomplish limited or specialized goals. For example, Hillman et al. (5) describe a 1-week research course designed to encourage careers in radiology research. DeHaven et al. (6) report on the development of a research program in a family-medicine residency to encourage community-based research. Coleridge (7) reports on a research opportunity for residents in osteopathic medicine designed to teach residents to write a research paper.
The research experience designed by the Internal Medicine Clinical Research Consortium faculty (8) for residents in internal medicine at the University of Pittsburgh is the closest in goals and design to the seminar we present here. A single research project involving "most of the 66 PGY-2 and -3 internal medicine trainees and seven general-medicine faculty ... (p 219)" provided the venue for training in research. This training program, which extended over 27 months from identification of the research topic through submission of a manuscript, emphasized hands-on experience in a specific, large project. Didactic instruction was apparently not a regular component of the program. The involvement of the residents varied both according to the interest and time of the individual and according to the stage of the research project. The authors conclude that, partly because of the fragmentary nature of residents' participation, "it is unclear what knowledge and skills residents gained regarding research methods (p 222)."
As noted by de Groot and Kennedy (9), research requires the expansion of the residents' definition of clinical relationships from physician–patient to include investigator–subject. Just as comfort with the physician–patient relationship is only built through repeated structured exposure and supervision, so we believe that comfort with the investigator–subject relationship can only proceed from a carefully mentored, comprehensive experience in clinical research. As primary supervisor of outpatient psychiatric experiences for residents, the first author of this paper became increasingly concerned about the lack of hands-on research experience for most residents. This seems to run counter to the scientific basis of modern medicine, the expectation that physicians will use clinical research findings in clinical decision-making, and the commitment of supervising clinical faculty to active research. With this impetus, we designed a research experience for residents that allows close collaboration with research faculty and participation in the full course of a clinical research project, while promoting increased familiarity and comfort with clinical research literature.
Our program has a total of 8–10 full-time PGY-III general-psychiatry residents and PGY-IV combined internal medicine–psychiatry residents in the outpatient clinic for a full year as part of our university-based residency program. There are 12 full-time faculty psychiatrists and 40 part-time faculty members. The model described by Thompson et al. (3) was less appropriate for our setting because of the relatively small number of full-time faculty psychiatrists and the fear of overburdening our clinical population. Conversely, the work of Blake et al. (4) recognized how difficult it is to provide a uniform experience with many outside/part-time mentors. The size of our program offered the opportunity for a cohesive experience for small groups of residents working regularly with committed full-time faculty. This situation supported our feeling that the research experience had to be led by full-time departmental faculty in order to maintain a priority equivalent to always-pressing clinical demands.
We wanted to provide a comprehensive research experience for our residents to include the goals of the specialized programs described above and more. We hoped to include a hands-on experience, to provide an opportunity for scientific thought and writing, and to critique published research while encouraging careers in psychiatric research.
The research experience reported by the Internal Medicine Clinical Research Consortium was a possible model, but it involved many more residents than our seminar and, as a result, did not provide a uniform experience for each resident, nor did it include didactic instruction.
This article describes the development, evaluation, and critique of an experiential seminar designed to introduce general-psychiatry residents to the theories, methods, and logistics of clinical research through a mentored group research experience.
|
OVERVIEW OF THE SEMINAR EXPERIENCE
|
Logistics
General-psychiatry residents and combined internal medicine/psychiatry residents are assigned to Outpatient Psychiatric Services full-time during the PGY-III and PGY-IV years, respectively, and part-time during the last year of residency training. A total of 12–15 hours per week are devoted to seminars on various clinical topics, with 22–25 hours per week spent in patient care and supervision. One seminar hour each week is devoted to the research training experience, with unstructured time varying with associated assignments. Attendance at the weekly research seminar is mandatory. The seminar has been offered yearly since 1994, with learning objectives, techniques, and outcomes constantly critiqued and revised by the seminar leaders in response to evaluations of each year's seminar. During the first years of the seminar, we attempted to have the residents design a research project during their full-time year in the clinic and then conduct the project over the following year, when they were in the clinic only part-time. This proved to be unsatisfactory for the residents and the seminar leaders, because coordination and prioritization were constant struggles. The required experience is now completed in a single year, with optional continued participation available in the following year.
Seminar Goals and Learning Objectives
In designing the seminar, a high priority was placed on specific knowledge of sound research methodology and, in particular, the ethical requirements of research with human subjects. Less emphasis was placed on the mechanics of statistical testing, although the underlying theory of inferential arguments is discussed in detail. The functional goal of the research seminar is to produce an original clinical research project design approved by the Institutional Review Board (IRB). The seminar is divided into five major components: orientation to clinical research, selection of a research topic, design of a research protocol, submission of an IRB proposal, and preparation for data collection. Each component has identified learning objectives and determined duration and method guidelines. Table 1 presents the learning objectives and summarizes the duration and methods for each component.
Components of the Seminar Experience
Orientation to clinical research.
The first month of the seminar reinforces the value of clinical research and demystifies the process of conducting clinical research. Residents read "Sounding Board: Picking a Research Problem, the Critical Decision," by Ronald C. Kahn (10). We review and discuss the objectives of the research seminar: critical evaluation of clinical and research data, experience in hypothesis-development/testing and research design, exposure to statistical analysis, stimulation of creative research ideas, logistics of data collection, presentation of clinical research data, and "ownership" of a research project. Residents review a sample protocol format, project-development algorithm, and the seminar plan (Table 1). Cooperation, discovery, satisfaction, and attention to detail are emphasized for a meaningful, scholarly research experience. The intrinsic connection between accomplishing a research project and communicating the results to the community of scientists is also emphasized, both theoretically and pragmatically. For example, the authors designed Figure 1 for use in the seminar as an orientation tool and to stimulate discussion of the ultimate goals and objectives of clinical research.
Selection of a research topic.
Seminar participants spend approximately 2 to 3 months brainstorming about clinical problems and resulting questions and producing a list of potential research ideas. The seminar leaders encourage the residents to begin their thinking from the basis of clinical experience: what clinical issues do they find particularly interesting or difficult? With what particularly interesting or difficult patients are they dealing? What unusual illness presentations, developments, or responses to treatment have they observed? Each topic is reviewed for clinical relevance, practical feasibility in the clinic/ departmental setting, and interest of the house staff. A preliminary literature search for the three most promising ideas provides the basis for consensual choice of a research topic to investigate further. Residents divide themselves into groups on the basis of interest in the topic area, perform a MEDLINE search, obtain and read relevant articles, and summarize their findings to the entire group. One of the three areas is chosen as the topic by consensus. A more intensive and detailed search of the literature is then performed. Relevant articles are obtained and read by each resident in anticipation of potential hypothesis-generation and discussion. Over the years, this part of the seminar has become an increasingly valuable experience for the residents. What began as a necessary foundational task has evolved into an experience of careful review of the literature, lively discussions, and spirited defenses of favored topics. Recently, several residents have written case reports for publication during this portion of the seminar, stimulated by investigation of a particularly interesting clinical presentation.
Design of the research protocol.
The seminar focuses for approximately 2–3 months on design of the research protocol. A primary hypothesis and secondary hypotheses are developed. With faculty supervision, the residents design a research study, usually a prospective, double-blind experiment. The seminar leaders guide the residents in a detailed design of the research protocol, including planning the statistical analysis. At this point in the seminar, several weeks are typically devoted to primarily didactic presentations of the principles of inferential statistics and sampling methods.
Preparation of IRB protocol.
The medical writing is divided among the residents—some volunteer, others are assigned by the seminar leaders. Progress of the work is monitored weekly. Seminar participants work individually or in pairs to produce drafts of the justification and literature review, research design and methods (Institutional Review Board format), consent form, and summary study schedule. Drafts are reviewed by the entire seminar group. The seminar leaders have come to recognize this exercise as the core of the learning experience for the residents. The preparation of a protocol that is suitable for submission to the IRB requires the residents to attend carefully to the methodological and ethical details of the study. This portion of the seminar typically occupies approximately 4 months, with multiple drafts circulating among the participants. The preparation of the protocol by the group also frequently generates discussions with other faculty members involved in research, providing the residents with a practical focus for learning from the experience of senior researchers. The protocol is submitted to and reviewed by the IRB, providing seminar participants with a "real-world" critique of their product.
Preparation for data collection.
The year's experience ends with 1–2 months' attention to the logistics of data collection. The residents design the necessary clinical research forms, learn research techniques such as the Structured Clinical Interview for DSM (11), and work with the seminar leaders to design computerized databases for the study.
Year 2 follow-up: data collection, analysis, and preparation of a manuscript.
The seminar leaders continue interaction with interested residents into the following year. Typically, residents who are interested in academic careers, particularly those looking toward a post-residency research experience, are motivated to continue the study and carry it forward. At the end of the first year, the seminar leaders typically lead the group to a consensus as to who will participate in the project, and in what manner, in the coming year, and what implications that will have for authorship of papers and ownership of data.
Evaluation
Approximately 30 residents have participated in the seminar since its inception in 1994. The number is approximate because variations in training patterns for individual residents have meant that some have had to join the seminar in progress or leave before the end of the year.
We have used two simple quantitative measures to support the detailed qualitative evaluation at the end of each course year. Our goals for the quantitative measures were 1) participation in the quantitative analysis by the majority of seminar participants; 2) straightforward interpretation that did not rely on inferential statistics; 3) high replicability; 4) low demands on faculty time; and 5) information to support the qualitative evaluation. The first quantitative measure is the evaluation that the residents complete yearly for each formal seminar experience. This measure focuses on how "useful" the residents estimated the experience was in their training, how much they "enjoyed" the experience, regularity of attendance, reasons for not attending, and suggestions for improving the seminar. The questions on perceived usefulness and enjoyment are answered on a five-point Likert scale from 1=little value to 5=very much. Attendance is answered by endorsing one of four options: consistent, intermittent, spotty, dropped it.
Over 90% of participating residents have completed this measure for the research seminar. The mean scores on the five-point rating of the usefulness and enjoyment of the seminar were identical: 3.6, with a range from 2 to 5. The score on the four-point rating of attendance was 4 for all respondents, due at least in part to the success of the seminar leaders in mandating attendance. Two respondents provided additional comments: one suggested that the seminar be optional, and the other noted that the seminar was interesting, but of little perceived relevance to that respondent, who did not anticipate engaging in research. These two comments indicate that one implicit goal of the seminar, to educate residents as to the necessity for all practitioners to understand research, has not been fully accomplished.
The second quantitative measure is a brief questionnaire (see Appendix 1A and Appendix 1B) which, beginning in 1996, has been administered before the seminar and immediately after the seminar, to assess changes in residents' knowledge related to seminar content. This questionnaire was designed by the authors to reflect their priorities in teaching the seminar. The questions encompass statistical concepts (five questions), ethical issues (three questions), and methodology (two questions). One question asks for a definition based on required reading, and one question asks for the residents' personal opinion of clinical research.
Pretests, completed by 13 residents, reflect the diverse training experiences and exposures to research with which the residents come to this seminar experience. The mean percentage of correct answers was 54%, varying from 23% to 87%. Within the three major subsets of the questionnaire, means were 50% for statistics (range: 0–80%), 57% for ethics (range: 33%–95%), and 67% for methods (range: 32%–85%). The fact that the lowest percentage correct was achieved for the statistics questions was expected. Of more concern is that residents seem more familiar with research methodology than with the ethics of research with human subjects. This reinforces the value of the preparation of a proposal for IRB review as a significant part of the seminar work.
Fourteen residents have completed the posttest questionnaire. The mean percentage of correct answers was 73%, varying from 53% to 87%. Within the three major subsets of the questionnaire, means were 69% for statistics (range: 20%–100%), 77% for ethics (range: 45%–95%), and 72% for methods (range: 44%–81%). Although we have to be cautious in drawing comparative conclusions, it does appear that the residents who have had the seminar have gained knowledge in all three areas, with the greatest average increase in score from pretest to posttest occurring for ethics (20% increase) and statistics (19% increase).
The qualitative evaluation is a goal-attainment evaluation and is based on the seminar leaders' judgment of the extent to which the goals of the seminar were achieved. This judgment is formed through 1) an end-of-the-year debriefing session with seminar participants; 2) the results of the quantitative measures; and 3) a yearly evaluation and planning meeting of the seminar leaders. Although this evaluation is the most difficult to document for publication, it is often the one that most influences the design and conduct of the seminar. For example, the goals of the seminar initially included having each group of residents participate in data collection for the research study designed by the preceding group. But our experience suggests that residents have low motivation for recruiting patients for a study "owned" by a previous group of residents. Expectations have been revised accordingly. These and other issues arising from the evaluation of goal attainment are presented in the Discussion section.
|
DISCUSSION
|
Our experience supports the benefit, feasibility, and difficulties, of involving house staff in an experiential research seminar. Success appears to be related to high expectations and involvement of the faculty leaders, incorporation of the seminar into the clinical experience, availability of additional resources (e.g., clinical research nurses), faculty encouragement, and monitoring of the structured timetable for the research seminar. This model provides a group mentoring opportunity in clinical research, ensuring guidance and resources for the residents while allowing for independent, creative design of a project. Resident evaluations of the seminar suggest that further interest in research was stimulated by the research seminar experience. All of the house staff reported feeling significantly less intimidated by research and the scientific literature.
Some limitations of this model are evident. Faculty need to remain ever-committed in their efforts to keep to the seminar schedule. The residents' perception that research required more time than was warranted by the benefits to them personally was the primary problem involving work performed outside of the scheduled seminar.
This seminar requires cooperation among the residents to complete the seminar task. In this respect, the project resembles faculty research projects (also done by a group), but without a single driving force (the principal investigator). There is usually at least one resident each year who is eager to fill the leadership vacuum. Conversely, occasionally a resident appears not to pull his/her weight. The developing group and interpersonal dynamics require careful monitoring by the seminar leaders, with occasional guidance offered to the residents in bringing problem behaviors to awareness and/or dealing constructively as a group with participants whose behavior is creating problems. This opportunity to observe and work with group dynamics is a secondary benefit of the research seminar experience.
In conclusion, our experience suggests that training in clinical research may be incorporated successfully into a psychiatric residency program utilizing the above model. The group approach offers certain distinct advantages, including 1) being less intimidating for the residents; 2) allowing for flexibility in demands on individuals while maintaining the momentum of the project; 3) operating within an organization that is familiar to the residents and congruent with other seminars; and 4) encouraging a cooperative approach to research as a team that is an appropriate preparation for research projects in the "real world." The evaluation information indicates that the residents value the experience and increase their knowledge of clinical research, particularly in the areas of statistical concepts and research ethics.
|
REFERENCES
|
- American Medical Associates: Graduate Medical Education Directory, 1994-1995. Chicago, IL, American Medical Association, 1994
- Cooke TDV: Research education in residency programs. Clin Invest Med 1992; 15:222-233[Medline]
- Thompson RJ, Benrubi GI, Kaunitz AM: Resident research in obstetrics and gynecology. J Reprod Med 1994; 39:635-638[Medline]
- Blake DJ, Lezotte DC, Yablon S, et al: Structured research training in residency training programs. Am J Phys Med Rehabil 1994; 73:245-250[Medline]
- Hillman BJ, Maynard CD, Stanley RJ, et al: The development, first year's experience, and promise of a program to encourage radiology research careers. Invest Radiol 1992; 27:192-197[Medline]
- DeHaven MJ, Wilson GR, Murphree DD: Developing a research program in a community-based department of family medicine: one department's experience. Fam Med 1994; 26:303-308[Medline]
- Coleridge ST: Teaching residents to write a research paper. Med Educ 1993; 93:936-940[Abstract]
- Internal Medicine Clinical Research Consortium: House staff team research in the ambulatory setting: it can be done. J Gen Intern Med 1995; 10:219-222[Medline]
- de Groot JM, Kennedy SH: Integrating clinical and research psychiatry. J Psychiatry Neurosci 1995; 20:150-154[Medline]
- Kahn RC: Sounding board: picking a research problem, the critical decision. New Engl J Med 1994; 330:1530-1533
- Spitzer RL, Williams JBW, Gibbon M, et al: Structured Clinical Interview for DSM-III-R-Non-Patient Edition (SCID-NP, Version 1.0). Washington, DC, American Psychiatric Press, 1990
This article has been cited by other articles:
|
|
|
|
 
L. W. Roberts and M. P. Bogenschutz
Preparing the Next Generation of Psychiatric Researchers: A Story of Obstacles and Optimism
Acad Psychiatry,
March 1, 2001;
25(1):
4 - 8.
[Abstract]
[Full Text]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2001
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
