There are numerous methodological problems in the published psychopharmacological literature. Some of these problems, including examples from published papers, are described elsewhere (1). Sponsorship bias and its effect on the wording of the report, particularly on the wording of abstracts, has been demonstrated (1). These problems are not always apparent to a superficial reader, particularly not to one who focuses only on the abstracts. In our experience, many practicing clinicians do not consider these problems important, thinking that these are “minor technical details.” In this article, we will describe our effort to improve this situation by teaching systematic critical appraisal of individual publications.

A series of workshops teaching critical analysis of published papers on psychopharmacology was conducted at three national psychiatric meetings in the Czech Republic and Slovakia in 2009 and 2010. The participants were psychiatric residents and other young clinical psychiatrists from both countries. The same three tutors (authors of this article) with experience in psychopharmacology were present at each teaching session to direct discussion and provide feedback to the participants. Each workshop analyzed two different papers that were sent to the participants before the event for individual study. One of the six papers selected by the tutors for analysis was industry-sponsored; five were independently funded.

The rating instrument was an appraisal sheet consisting of 23 items developed by the authors. The appraisal sheet was based, in part, on a similar instrument developed at the Center for Evidence-Based Medicine at the University of Toronto (http://ktclearinghouse.ca/cebm/teaching/worksheets/therapy); 22 items were questions exploring various aspects of the papers, with three possible responses: Agree, Disagree, or Undecided. A fourth option (“Not Applicable”) was added after the first workshop. One additional item was an unstructured question inquiring about further problems in the study and asking for comments. Full text of the appraisal sheet with an example of the results is shown in Table 1.

During the workshops, each paper was evaluated, one item at a time, by each individual participant, using the appraisal sheet. A direct-recording electronic voting system was used to collect and record the results. The text of the item was projected on a screen, and each individual endorsed one of the four answers (Agree, Disagree, Undecided, or Not Applicable), using buttons on a small recorder that was provided for each participant. After all the participants voted, the results in terms of frequencies for each of the four answers pertaining to the item being evaluated were instantly displayed on the screen and automatically recorded for future off-line analyses. At that point, a discussion of the particular item and its evaluation ensued. The tutors shared their views and provided feedback including the key ratings. The voting implied informed consent to participate in the workshop and was voluntary. Participants in the first workshop were supported by a pharmaceutical company (see Acknowledgment); the rest of the series was without any industry sponsorship.

Evaluation of one paper from a session with the highest number of trainees was selected as an illustrative example, and the following variables were assessed: 1) items with the highest agreement with the key rating (i.e., responses endorsed by the tutors), measured as a percentage of “correct” responses; 2) items with the highest rates of disagreement among the trainees, measured as a frequency of non-unanimous (discordant) ratings; and 3) items with the highest number of “Undecided” responses, where, in fact, the key rating indicated an “Agree” or “Disagree” response.

Six papers were appraised in the course of three workshops. A total of 58 trainees participated. The methodological quality, as well as the clarity of the writing, varied greatly across the papers. The appraisal of one representative article was further analyzed (Table 1).

The results showed that:

We have constructed an instrument for the appraisal of psychopharmacological publications and used it as a teaching tool in a series of sessions with psychiatric residents and other trainees. The instrument (appraisal sheet) highlights the principal methodological features that are important for the interpretation of the results of psychopharmacological trials.

The principal purpose of the exercise was to help the trainees to think critically about what they read. Using the systematic approach imposed by the checklist format of the appraisal sheet, they discovered for themselves various methodological problems in the publications that were scrutinized during the workshops. One of the principal functions of the tutors was to demonstrate how these problems affect the interpretation of the results. The discussion format and voting stimulated active participation of the trainees. We witnessed the result that active involvement, facilitated by the structured assessment tool and followed by immediate feedback and thought-provoking discussion, may enhance the learning process. Finally, our observations underscore the importance of teaching statistics and methodology.

Our feasibility study presenting the experience with this new teaching method has certain limitations. No meaningful statistical analysis of the results of the ratings was feasible because the numbers of papers and participants were too small. The results shown here thus serve as an illustrative example and do not allow for any generalization. Furthermore, we did not test the participants before and after the workshops to estimate the educational impact of their experience. These issues can be addressed in future research.

We thank Eli Lilly & Company for sponsoring the first of the three workshops. Drs. Mohr and Höschl were supported by the research projects CNS 1M0517 (from the Czech Ministry of Education) and MZ0PCP2005 (from the Czech Ministry of Health).

Dr. Mohr has received speaking honoraria and/or consulting fees and travel support from AstraZeneca, Bristol-Myers Squibb, Eli Lilly, Janssen-Cilag, Krka, Servier, and Pfizer. Dr. Höschl has received speaking honoraria and/or consulting fees and travel support from Bristol-Myers Squibb, Eli Lilly, Janssen-Cilag, Gedeon Richter, Krka, Servier, and Lundbeck International Neuroscience Foundation. Dr. Volavka has received speaking honoraria and/or consulting fees and travel support from Eli Lilly and Merck.