Aprevious report entitled "Maximizing the Benefits of Peer Review" (1), which was written by the senior authors (M.P., D.W.H.) , emphasized the interface between the peer review function and each of the following administrative processes: credentialing/privileging, competency assessments, performance evaluations, and disciplinary actions. We suggested that an equitable peer review process, incorporating input from and feedback to the medical staff, can improve clinical competence and performance in general and can effect positive changes in specific areas of practice.
In their "Commentary on ‘Maximizing the Benefits of Peer Review’ " (2), Schmetzer and Wernert stated: "While we agree with the values espoused by Patel et al., we believe that it will take more than a good process … to bring about such a shift in values. … Instead, we would argue that residents in psychiatry … must have course work specific to the theories and benefits of administrative activities such as peer review built into their training."
The Accreditation Council for Graduate Medical Education’s (ACGME’s) Special Requirements for Residency Training in Psychiatry has also challenged psychiatric residency programs to "provide sufficient experiences in psychiatric administration … including supervised experience in utilization review, quality assurance, and performance improvement" (3).
In response, we devised a peer review workshop for PGY-3 and PGY-4 psychiatry residents.
Our peer review workshop, which began and ended with a 3-hour classroom discussion, lasted 6 weeks. In the intervening 5-week period, the participants designed and implemented their own peer review programs.
The residents were from three institutions: a large state hospital, a VA hospital, and a community hospital with a 28-bed psychiatric unit. Prior to the first session of the workshop, the residents were asked to identify foci of attention for possible improvement at their respective institutions. After receiving suggestions from residents such as "improve the conditions of the call room," we offered the following broad examples of issues for consideration: safety, documentation, treatment planning, and aftercare issues. We also offered as examples typical problem areas in most psychiatric hospitals: suicide and violence risk assessments, other high risk conditions, restraint usage, and patients on special precautions. Finally, we suggested certain issues that are the subject of Joint Commission on Accreditation of Health Care Organizations, Centers for Medicare and Medicaid Services, and other agency reviews: proscribed abbreviations, cultural competencies, informed consent, and restriction of rights.
During the first hour of the first workshop session, the article by the senior authors, discussed previously, and the peer review format in use at their hospital were distributed. Definitions of and distinctions among peer review, competency assessment, and performance evaluations were stressed. A pretest on didactic information of the course was administered (Appendix 1). The remainder of the first session emphasized the purpose and utility of the peer review process as an instrument of change. The residents identified site-specific problem areas and began to formulate peer review questions to address them.
While there was ready agreement among the residents at each particular site regarding the types of issues they wished to address, the nature of the issues varied considerably from one institution to another. We believe that this is a function of differences in the nature of the practice, patient populations, and documentation requirements at the different sites.
During the second week, the resident groups created the content of the peer review forms for their individual sites. This effort was coordinated by the chief resident at each site in communication with one of the senior authors (M.P.). It involved input from all residents at each location, including PGY-1s and PGY-2s who were not involved in the workshop. We specifically did not correct or amend their products. The peer review form created by the residents rotating at the hospital where the senior authors practice is presented in Appendix 2.
In week 3, the residents at each site reviewed a patient’s chart for purposes of establishing interrater reliability. The same chart was reviewed by all residents at a particular site. Their results were reviewed as a group with their chief resident.
At one site the residents noted little agreement in their ratings of question 8: Does the discharge note mention dose, frequency, and other instructions for discharge medications? During the discussion that ensued, it became apparent that some residents scored "yes" if the information was documented by either the attending psychiatrist or the nurse, while others scored it positively only if it had been documented by the doctor. The residents decided on their own that the responsibility for this important documentation was uniquely that of the physician as the clinical team leader.
During week 4, a similar exercise was done with a different chart, and interrater reliability was notably improved. In the fifth week, each resident peer reviewed the chart of a patient who was on the caseload of another resident.
In week 6, the residents reconvened with Dr. Patel. Residents from each site, in turn, explained how they identified their specific peer review items. During this discussion, the residents noted certain similarities in focus at each institution. Each of the three sites included questions regarding suicide and violence assessments, physical examinations, and medication consents. All residents agreed as to the importance of these considerations in the practice of psychiatry.
Differences were also addressed. The residents concluded that these often reflected current site-specific emphases at their respective institutions. For example, question 11 (Appendix 2), addressing the need for a follow-up appointment within 5 days of discharge is an important focus of attention at our institution.
A discussion of discrepancies in scoring led to an appreciation of the need for clarity in communications. For example, questions 2 and 6 ask for an assessment of adequacy of the physical exam and the suicide and violence assessment. Other sites included questions regarding adequacy of the mental status exam and progress notes. The residents immediately understood that they were not in agreement as to what constituted "adequacy" in these areas and a lively discussion followed.
The relevance of quantitative versus qualitative elements was also raised by the residents. Should numerical scores be used? If so, should the scores be weighted? What differentiates "good" treatment and documentation from "superior"?
Near the end of the second group session, the 10-question test (Appendix 1) was given again and the answers were discussed. Mean scores on the pretest were 6.34 for PGY-3s (N=10), 5.6 for PGY-4s (N=7) and 6.0 for both years combined. The most frequently missed questions on the pretest were 2 (missed by all) and 9 (missed by 15 of 16). Posttest results were 8.3 for PGY-3s (N=10), 6.8 for PGY-4s (N=9), and 7.8 combined. This improvement is statistically significant (p<0.001). Question 2 remained the most missed question (14/15). We hypothesize this is because the question relates to various processes, such as credentialing and privileging, to which the residents had not yet been exposed. Question 9, the explicitly "legal" question missed by 15/16 in the pretest, was missed by only two of the residents on the posttest. We attribute PGY-3s outscoring PGY-4s on both tests to the fact that many of the PGY-3s had previously rotated through our hospital and had gained some prior exposure to these concepts. None of the PGY-4s had such prior experience.
The workshop ended with a discussion of its utility and the residents were asked to complete an anonymous written assessment. The feedback was universally favorable. The most frequent comments from the residents were that the workshop experience improved the quality of their clinical skills, especially in documentation and assessments of risk factors.
We noticed, however, that the level of discussion among the residents rose to a new level of appreciation of administrative techniques and skills. In so doing, our residents unknowingly iterated the findings of Balla et al. (4) that participation in peer review groups enhanced among attending psychiatrists a reflective practice that achieved new understanding of clinical work.
The residents were unanimous in recommending that the workshop be repeated on a yearly basis at the beginning of the academic year with the previous year’s PGY-3s serving as mentors at each clinical site. Most importantly, in our view, the residents decided to continue monthly peer reviews of each other’s charts and suggested the results be submitted to the attending psychiatrists to whom they were assigned for discussion during their evaluations.
In 1980, the American Psychiatric Association (APA) published an action paper on "Psychiatrists in Administrative and Policy Making Positions in State and Other Governmental Mental Health Systems." Commenting on this article, Borus (5) wrote: "It suggests that the administrative ignorance and organizational passivity of many in our profession provide a convenient rationale for bypassing psychiatrists for important administrative roles, leading to the psychiatrist director becoming an ‘endangered species.’ "
We suggest that a well-designed peer review process, more so than any other administrative function, incorporate managerial decision making skills and techniques in a manner that can be readily appreciated by attending psychiatrists. Additionally, some authors have recommended what the ACGME now requires—that such skills and techniques be part of the psychiatric residency curriculum.
We elected to teach these skills in a peer review workshop. For a relatively small investment of our time, we achieved a modest improvement in what the residents know about managerial/clinical issues and perhaps an even more significant improvement in how they think about them. We believe that our workshop process has potential and practical application for other psychiatry residency programs that are also attempting to respond to these challenges.