Turnball J, Carbotte R, Hanna E, et al: Cognitive difficulty in physicians. Acad Med 2000; 75:177—181
The College of Physicians and Surgeons of Ontario, the licensing body for that province, has instituted an intensive diagnostic program for physicians identified by quality assurance committees as having deficiencies. The Physician Review Program (PREP) is a full-day evaluation consisting of multiple-choice questions, standardized patients, and chart-stimulated recall. A single summary grade is computed based on the results of all these tests, and on the basis of their scores, subjects are categorized into six levels of competency. Levels I and II are associated with no or minor deficiencies; III and IV indicate moderate to major difficulties (serious concern about competence); V is unsafe to practice without direct supervision; and VI is unsafe to practice in any setting. On the basis of the PREP results, continuing medical education experiences are prescribed.
About 30 of 26,000 physicians are referred every year, and it had been noted that for some of these doctors, extensive remedial work did not return them to competence. As a result, a neuropsychological test battery was added to the evaluation to determine whether there were underlying cognitive deficits or mood problems. This assessment took 1.5 hours, and scores were produced for each of the following: verbal problem-solving, visual—spatial problem-solving, memory, and tracking/concentration, as well as a summary score. The latter score represented significant deficits in a single domain or lesser deficits across several domains. Interrater agreement on 12 cases was 0.82.
Twenty-seven physicians were tested in this study. Seven of these showed moderate or severe cognitive difficulty. Mood disturbance was identified in 3 of the 27, and 1 had both mood disturbance and cognitive problems. The authors noted that these physicians' scores on the Profile of Mood States was well below the average for the norm group, which consisted of non-distressed college students, suggesting that these physicians may have been reluctant to acknowledge psychological distress.
Six of the 19 physicians (32%) who were categorized as III, IV, or V on the PREP had moderate or severe cognitive impairment. Of the eight who scored well on PREP (categories I or II), seven had no, minimal, or mild cognitive impairment.
The authors concluded that neuropsychological screening may be an important component of physician assessment programs if it can be demonstrated that scores on such measures are predictors of the effectiveness, or lack of effectiveness, of educational interventions. If competency cannot be restored, then appropriate disability arrangements could be made.
Colliver JA: Effectiveness of problem-based learning curricula: research and theory. Acad Med 2000; 75:259—266
Colliver reviews eight studies on the effectiveness of problem-based learning (PBL) curricula published from 1992 through 1998. Three review articles were published in 1993, and although different methodologies were used, all reached the same conclusion: that PBL had little or no effect on student achievement. The author argues that a major educational intervention like PBL should have relatively large effect sizes; that is, students in PBL groups should differ markedly from students in standard curricula.
All of the eight new studies compared students in PBL and in standard curriculum tracks. Three involved randomized assignment to tracks, and five were nonrandomized. The three randomized studies showed no effect of PBL on the NBME Licensure Examinations or on measures of diagnostic reasoning and clinical problem-solving. Although the nonrandomized studies reported some effects, these were readily attributable to selection biases and the use of outcome measures that were closely related to specific PBL activities.
Colliver suggests that although PBL is supposedly based on educational theory, close examination suggests that the links between that theory and its basic research are loose at best and hence don't provide a strong rationale for this pedagogical method. He does suggest that PBL curricula may provide more enjoyable medical school experiences than do standard curricula, but they have not been demonstrated to be more effective in terms of either knowledgebase or clinical performance.
Peters AS, Greenberger-Rosovsky R, Crowder C, et al: Long-term outcomes of the New Pathway Program at Harvard Medical School: a randomized controlled trial. Acad Med 2000; 75:470—479
This study was published after Colliver's article and reports on a follow-up study of graduates from Harvard Medical School who were randomly assigned either to the New Pathway Program (NP) or to the standard curriculum. The NP, a preclinical curriculum, had several objectives: promoting a sound grasp of the basic sciences, developing facility in and positive attitudes toward self-directed learning, and developing competency in integrating psychosocial and humanistic concepts with biologic principles in patient care. PBL was selected as the primary pedagogical method for meeting these objectives, although the NP also included such activities as lectures and laboratories.
The subjects were 63 NP students and 62 traditional-curriculum students who entered medical school in 1985 or 1986. Program evaluation took place at the end of the second and fourth years of medical school and four years after graduation. At those times, the NP students were found to be more positive in the humanistic domain, but there were no significant differences in basic science knowledge or clinical problem-solving.
This study reports on long-term follow-up at 12 and 13 years after entry into medical school. It was hypothesized that "NP graduates would show more positive attitudes toward and behaviors related to humanistic medicine, lifelong learning, and issues related to learning in a social environment than would control[-group] graduates." (p. 471)
Of the 125 graduates, 100 (50 NPs and 50 controls) were located who agreed to participate in the evaluation. A survey instrument containing items that measured attitudes and self-reported behaviors in three domains (humanistic medicine, lifelong learning, and social learning) was administered in a 20-minute telephone interview. A final open-ended question asked the respondents to describe two or three lasting effects of their education during the first 2 years of medical school.
In terms of humanism, 40% of the NP graduates, vs. 18% of traditional graduates, were engaged in careers in primary care or psychiatry (P<0.05). The NP graduates felt better prepared to practice humanistic medicine and felt more confident in managing patients' psychosocial problems than did their counterparts.
In the lifelong learning domain, there were no significant differences between the two groups in reported behaviors or attitudes. In the social-learning domain, NP graduates rated their faculty's influence on their current thinking higher than did their counterparts, whereas the traditional graduates responded more positively to the statement that there is a better way to learn in the first 2 years of medical school than the way they learned.
In response to the question about the lasting effects of their first 2 years of medical school, one area in which there appeared to be differences was in the approach to lifelong learning. The NP graduates more frequently described seeking information to solve patients' problems, whereas the traditional graduates emphasized staying up-to-date with and managing the enormous body of scientific research.
The authors concluded that these two groups of graduates were more alike than different in terms of the behaviors and attitudes assessed in this study except for significant differences in the domain of humanistic medicine. They state, "We believe the evaluation makes a convincing case that a humanistic approach to medicine can be taught and learned (p.478)"
Copeland HL, Hewson MG: Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Acad Med 2000; 75:161—166
Copeland and Hewson describe the development of an instrument to assess clinical teaching effectiveness across their entire institution. In addition to completing a comprehensive literature review, the authors held meetings with all relevant stakeholders to get their input on the proposed instrument. The final form had 15 specific items, one general item, and space for written comments. Each item was rated on a five-point scale.
Over a 20-month period, more than 8,000 instruments were completed, yielding a median of eight instruments per faculty member. The average rating for all 15 items was 4.12 (Often/Very Good). The items were interrelated, with correlations ranging from 0.57 to 0.77. Factor-analysis yielded one factor that accounted for 71% of the variance in the ratings, indicating that the instrument was measuring one core concept. Different measures of reliability were all high, and the new instrument was also found to be highly usable.
Copeland and Hewson concluded that, "By providing a psychometrically sound and theory-based instrument, we can not only improve the teaching at this academic medical center but also promote the importance of clinical teaching and demonstrate the value the institution places on such efforts through appropriate responses (p. 166)." Educators at other institutions might also find this evaluation tool to be appropriate for measuring clinical teaching in their settings.