Ziv A, Ben-David MF, Sutnick AI, et al: Lessons learned from 6 years of international administrations of the ECFMG's SP-based clinical skills assessment. Academic Medicine 1998; 73:84—91

The Educational Commission for Foreign Medical Graduates (ECFMG) has announced that in July 1998 a clinical skills component will be added to the examination that foreign medical graduates (FMGs) must take to enter U.S. residency programs. Ziv and colleagues report on a series of studies undertaken to evaluate the feasibility of developing and administering such an assessment in a number of cultures.

Representatives of the ECFMG worked with physicians in Israel, Spain, Brazil, Russia, and Ukraine to develop site-specific examinations that drew on a pool of 21 standardized patient cases, all of which were designed to assess the clinical skills of senior medical students or first-year residents. The cases, which were translated into each country's spoken language, consisted of an encounter with a standardized patient, followed by writing a postencounter patient note.

There were 10 cases on each examination, and 4 scores were produced for each examinee: data gathering, interpersonal skills, documentation and interpretation of clinical data, and total medical skills score. A total of 683 subjects participated in the various projects, including some FMGs already in residency training in the United States.

Each of the seven administrations yielded subtest scores with reliabilities that exceeded 0.70, with one exception. At all sites, the highest mean score was for interpersonal skills, followed by history taking, physical examination, and patient notes, with similar standard deviations for all components, except physical examination scores, which were more variable.

Correlations between the three independent subtest scores were calculated within sites, and there were similar patterns across sites. The lowest average correlation was between patient notes and interpersonal skills (0.29), and the highest was between history taking and interpersonal skills (0.69).

The authors conclude that, based on the success of these projects, psychometrically sound clinical skills assessments can be conducted on an international basis through systematic collaboration with local physicians.

Harasym PH, Leong EJ, Violato C, et al: Cuing effect of "all of the above" on the reliability and validity of multiple-choice test items. Evaluation and the Health Professions 1998; 21:120—133

One of the rules of thumb for writing multiple-choice questions is to avoid use of "all of the above" as a distracter. This study systematically evaluated the effect that option had on the psychometric properties of an examination.

Two versions of a 100-item multiple-choice test were developed, and each was administered to 92 first-year nursing students as the final examination for a course in human anatomy and physiology. Tests A and B had 80 items in common, and there were also 20 experimental items on each test.

On Test A, 10 of these included the option of all of the above, and the remaining 10 were in the multiple-response format that required the examinees to indicate whether each distracter was true or false. On Test B, the formats for these 20 items were switched, that is, the multiple-response items were changed to all-of-the-above items and all-of-the-above items were reformatted as multiple-response items. These items were randomly distributed throughout the test, and all of them were scored dichotomously.

There was no significant difference in the mean scores of the 2 groups on the 80 nonexperimental items. However, on the experimental items, the multiple-response items were significantly more difficult than the items with the all-of-the-above option.

Harasym and his colleagues suggest that the all-of-the-above items were easier because of cuing effects. Examinees who recognized that at least two of the options were correct selected all of the above as the correct answer even if they were not completely sure about the correctness of the two remaining options. Because these items are easier and therefore less discriminating, they are associated with lower reliability coefficients than when the same content is tested in the multiple-response format. In addition, performance on the multiple-response items was more predictive of performance on the nonexperimental items.

Because of the reduction in validity and reliability, the authors advise using the multiple-response format to test knowledge of multiple facts instead of items with an all-of-the-above option.

Solomon DJ, Speer AJ, Rosebraugh CJ, et al: The reliability of medical student ratings of clinical teaching. Evaluation and the Health Professions 1997; 20:343—352

This study was undertaken to determine the reliability of medical student ratings of clinical teaching. The evaluation form consisted of 13 positively worded descriptions of teaching behavior that were rated on a scale of 1 (rarely) to 4 (consistently). There was also space for written comments. The questionnaire was administered in conjunction with a clerkship examination.

Over a 4-year period, 1,570 complete forms were obtained on which 147 faculty members were rated at least twice. On average, each faculty member was rated by six students per academic year. The data were analyzed using intraclass correlation coefficients.

The interrater reliability coefficient of a single rater was quite low. The coefficient for the 6 ratings typically received in an academic year was about 0.63, and it was about 0.85 for the 18 ratings received over 3 years. At the point of going up for promotion and/or tenure, the typical faculty member would have 30—40 student ratings with an interrater reliability of approximately 0.92.

Hence, the authors conclude that the number of student ratings that have usually accumulated at the time of evaluation for promotion and/or tenure would provide "meaningful information about a faculty member's teaching performance" (p. 352), although the authors also caution that data from a small number of raters is likely to contain a substantial amount of measurement error.

Raymond MR: Establishing weights for test plans for licensure and certification examinations. Applied Measurement in Education 1996; 9:237—256

Raymond compares two different methods of assigning weights to content categories on examinations in four different medical imaging specialties.

First, task inventory surveys were conducted to identify the tasks that practitioners in a given specialty must perform. Information was gathered from multiple sources, including published research and job descriptions, to develop the surveys that were administered to practitioners in the specialty of interest.

Next, expert panels specified the knowledge, skills, and attitudes required to successfully perform these tasks and developed test content outlines.

The panelists were then asked to make holistic judgments about the percentage of items that should be assigned to each of the major content categories. Weights were also derived from a statistical model that incorporated ratings of task frequency and task criticality.

The weights derived from these two different methods were compared, and it was found that for all four tests there was little agreement between the two sets of weights, except for categories that focused on specific imaging procedures.

Raymond discusses several factors that could have contributed to these findings. He suggests that "a prudent approach for assigning weights to test plans would be to rely on expert panels as a primary source of information and then use statistical weights to affirm or challenge those judgments." (p. 254)