Tamblyn R, Abrahamowicz M, Brailovsky C, et al: Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA 1998; 280:989—996
This study, which was performed in Canada, explored the relationship among licensing examination scores and several aspects of primary care practice. The subjects were 614 physicians who had recently passed the family medicine licensing examination, applied for a license to practice in Quebec, and entered fee-for-service practice in Quebec. The licensing examination had three components. A written examination consisting of multiple-choice questions and short-answer management problems assessed diagnostic ability, management skill, and knowledge of disease prevention.
An oral examination that simulated a patient office visit was used to assess communication skills, and clinical skills were measured by direct, structured observation of candidates responding to 24—26 clinical problems in an OSCE (objective structured clinical examination) format.
The aspects of primary care practice that were measured were ones for which variation in practice activity had been previously identified and that could be assessed by using provincial health care databases. The four practice indicators were consultation rate, symptom-relief prescribing rate relative to disease-specific prescribing rate for elderly patients, inappropriate prescribing rate for elderly patients, and mammography screening rate in women ages 50 to 69. Physicians were assessed during the 18-month period after the earliest possible entry into practice, and during the study period they encountered more than a million patients. Baseline data were retrieved for the 18 months before the physicians entered practice.
Higher test scores were associated with higher consultation rates, higher rates of disease-specific prescribing, compared with symptom-relief prescribing, lower rates of inappropriate prescribing, and higher rates of mammography screening. The authors conclude that the study "supports the validity of licensing examination as measures of clinical competence" (p. 994). The researchers emphasize the importance of measuring different aspects of clinical competence and possibly requiring demonstration of a minimal level of proficiency in each of them. Because few studies of this type have been done because of methodological and logistical difficulties, this promises to become an oft-cited reference by those addressing the links between test performance and performance in practice.
Regehr G, MacRae H, Reznick RK, et al: Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine 1998; 73:993—997
An objective structured clinical exaination (OSCE)-format examination was administered to 53 surgery residents in the first through the sixth years of training. There were eight 15-minute stations involving bench-model simulations of various general-surgery operative procedures (e.g., tracheostomy, abdominal wall closure, small-bowel anastomosis). Performance at each station was independently rated by two surgeons, one who used a global rating scale with seven dimensions and one who used an operation-specific checklist consisting of 20—40 items, followed by the global rating scale. In addition, the quality of the final product from six of the eight stations was assessed by two judges using four rating scales.
The interstation reliabilities were 0.79 for the checklist score, 0.85 for the global score alone, and 0.89 for the checklist-global score combination. Multiple regression analyses done with year of training and final product ratings as the dependent variables indicated that the global rating scores accounted for more of the variance in year of training and in the final product scores than did the checklist scores.
Based on the evidence of higher reliability and better construct and concurrent validity, the authors recommend that "when constructing a performance-based examination, primary consideration should be given to the possibility of having expert raters use carefully constructed global rating scales to assess examinees." (p. 997)
Sternberg RJ: The concept of intelligence and its role in lifelong learning and success. American Psychologist 1997; 52:1030—1037
Daniel MH: Intelligence testing: status and trends. American Psychologist 1997; 52:1038—1045
Sternberg RJ: Intelligence and lifelong learning: what's new and how can we use it? American Psychologist 1997; 52:1134—1139
These three articles, as well as others in this issue of American Psychologist, discuss contemporary concepts of intelligence, the status of instruments used to measure intelligence, and the relationship among intelligence, or IQ, and other valued measures, such as academic and occupational achievement.
Sternberg argues that, while the concept of intelligence has evolved from that of a unitary trait (g) to multiple, though correlated, traits, it is still too narrow. IQ tests primarily measure memory and analytical ability, as do indices of school achievement, for example, so high correlations between the two are not surprising. He proposes broadening the definition of IQ to "the mental abilities necessary for adaptation to, as well as shaping and selection of, any environmental context" (p. 1030), which should help us to better understand the role of intelligence in lifelong learning.
Sternberg and Daniel both observe that in some ways intelligence testing has changed little since its inception, although several major instruments have recently been revised. Daniel describes three broad categories of measurement devices: tests of psychometric abilities, tests based on neuropsychological processing models, and dynamic assessments.
In the first category are those tests that are most familiar (e.g., the Wechsler Intelligence Scale for Children and Adults, the Stanford-Binet Intelligence Scale). Tests in this category assess cognitive abilities, such as verbal and spatial abilities and inductive reasoning, which have been identified through factor analysis of performance on diverse tasks. An issue that needs further research is the relationships among scores from these different instruments.
Tests in the second category were developed based on Luria's model of brain function and assess arousal and attention, encoding and processing of information, and planning and monitoring functions. Daniel notes that the constructs represented by scores on these tests require clarification. In dynamic assessment, the focus is on the ability to learn, rather than on the structure of abilities.
Daniel then reviews several of the uses of intelligence testing in educational settings, including classification of students, diagnosis of educational problems, and guidance of remedial efforts, and discusses ways in which intelligence testing is likely to evolve.