Residency training directors dread various moments—for instance, when the Residency Review Committee (RRC) reviews their program, or Match Day. Another such moment of anxiety is when residents answer the Accreditation Council for Graduate Medical Education (ACGME) Resident Survey. This survey asks residents about their compliance with duty hours and their opinions of faculty, the feedback they receive, the educational content of the teaching, their program’s resources, and, finally, their overall experience. These are seemingly fair, important questions. Why would anyone dread seeing the answers? What would be wrong with a survey asking this information about the program? Why would anybody think that this is not a good and useful survey? In my opinion, at least four cardinal problems with the ACGME Resident Survey may trigger training director anxiety and dread.

The first problem is the validity and comprehensibility of the survey questions. Fahy and colleagues (1) suggested that the responses obtained on the ACGME Survey might inaccurately reflect the magnitude of noncompliance found in certain areas, and they proposed that this discrepancy might be due to the limited range of responses available on the survey. Sticca and colleagues (2) reported that 14% of residents admitted to not answering the questions truthfully, and 37% of residents felt that the survey did not provide an accurate evaluation of their work-hours. These authors felt that a tool in which 1 in 7 responders admits to answering questions falsely and 1 in 5 responders had difficulty interpreting the questions may not be a valid evaluation tool. In contrast, Holt and colleagues (3) found that the ACGME Survey demonstrated a high degree of internal reliability and thus felt that this survey was a reliable, valid, and useful tool for evaluating residency programs. Holt and colleagues (3) also reported that programs having resident-identified duty-hour issues were more likely than those without such issues to have received duty-hour citations from residency review committees (sic). These are contradictory results.

Some might question the first two studies (1, 2) because they were conducted by surgeons who have been unhappy with the implementation of the 80 duty-hour limit. Others may question the third study (3) because it was conducted by the main stakeholder and creator of this survey: the ACGME itself. Unfortunately, these studies, as well as many critics of this survey, focus on just one part of the survey—duty-hours—which is probably the most objective part of the ACGME Resident Survey.

I have a harder time with the interpretation of some questions in the other parts of the survey. For instance: “How sufficient is the supervision you receive from faculty and staff in your program?” (Possible answers: Extremely, Very, Somewhat, Slightly, Not At All); “Thinking about the faculty and staff in your program overall, how interested are they in your residency education?” (same answers); or “Thinking about the faculty and staff in your program overall, how effective are they in creating an environment of scholarship and inquiry?” (same answers) are all very subjective questions. Their interpretation could vary profoundly from resident to resident.

Similarly, the question “In your opinion, how often do your rotations and other major assignments provide an appropriate balance between your residency education and other clinical demands?” sounds ambiguous. What constitutes an appropriate balance is a matter of opinion, and this question raises further questions: What is considered education? Is seeing patients under attending supervision “clinical demand” or “education”? Is classroom teaching the only form of education?

Or consider another question—“How often do you work in interdisciplinary teams to care for patients?” What does the answer “Somewhat/Sometimes” mean? Are we expecting that residents work always in interdisciplinary teams? If they see patients in individual therapy (i.e., no interdisciplinary team), does it mean “noncompliance”? I could discuss more questions, but it seems clear that validity and easy comprehension/interpretation are not very strong features of the ACGME Resident Survey.

The second problem is that the ACGME marks a certain percentage of answers as noncompliant responses: the infamous “shaded boxes.” What does the term noncompliant mean here, and with what are programs noncompliant? In the era of evidence-based medicine (EBM), the leading proponent of using EBM in education (ACGME) does not provide an explanation of what noncompliance really means. I am even more troubled because it is not clear why and how the thresholds for noncompliance were determined. The requirement of certain percentages of “correct” or “desired” answers (i.e., 70%–90%) is not explained. The inherent subjectivity of the survey and this threshold is supposedly balanced by benchmarking; but how is this benchmarking done, with whom is this information shared, and is this a valid approach? The Residency Review Committee (RRC) supposedly uses this survey as one of many sources of data to address program compliance with its standards, which is fine, but why use predetermined percentages and shaded boxes? And how does the RRC (and program reviewers) use this survey and the shaded areas?

The Psychiatry RRC also uses two standard deviations from the national mean in three survey questions as a threshold for looking at programs outside of the usual accreditation cycle. Why two standard deviations? How was this level determined? How can answers to badly-formulated questions be used to determine anything? Are we trying to get an honest opinion about the program, or are we striving to get some ephemeral, unclear, unscientifically predetermined, and unrealistic numbers?

The third problem is that the residents evaluate the program at one point in time, which may not reflect their year-long experiences. One or more residents may experience an unsatisfactory or just not-so-laudatory evaluation by the faculty or an unusually hectic day or night call, and suddenly, all previous, possibly positive experience is forgotten. Then the Resident Survey arrives, and the chance for deserved or undeserved “payback” is here… Several residents in our training program actually raised this possible scenario with me privately.

The last problem I see is related to the third one. The existence of a survey that uses unclear, predetermined values for marking programs as noncompliant will put a severe brake on honest, frank evaluation of residents. Faculty and staff members will gradually realize how “threatening” the shaded areas on the ACGME Resident Survey are for them and their program. Actually, the conclusion of the study by Holt and colleagues (3) partially supports this justified or unjustified fear-prone view: programs with resident-identified duty-hours issues received more citations related to duty hours. Faculty and staff members who are afraid of negative “payback” evaluations by residents may be inclined to avoid giving honest feedback and evaluations of residents. The non-cognitive qualities or faults of our trainees thus would not be addressed and rectified. During my long teaching career, I have encountered faculty members who had something critical to say about their trainees’ performance, yet their written evaluations did not reflect this criticism. A lack of good, honest feedback to trainees is not what we strive for.

I am aware that the third and fourth problems are not specific to just this survey and are probably true for the evaluation of most educational activities. However, the effect and consequences of the ACGME Survey seem much larger and more serious than evaluation of any other educational activity or function, a fact probably well known to trainees.

The ACGME Survey was created to address some external, non-ACGME threats to regulating duty hours and the desire to move to a longer accreditation site-visit cycle with more continuous monitoring of programs. Some changes have attempted to improve the reliability and validity over the last few years. Nevertheless, the ACGME Resident Survey still seems to be a problematic tool, of questionable validity and clarity. The way it is currently used to evaluate individual programs is neither clear nor useful. It needs to be changed, replaced, or abolished. A frank discussion of this survey and its use that includes all stakeholders (e.g., residency training director organizations, ACGME), needs to take place. It is time to end the unspoken tyranny of an imposed evaluation of questionable validity and usefulness and to start debating how to improve the evaluation of training programs and make this evaluation a reflection of reality.