Acad Psychiatry 29:443-447, December 2005
doi: 10.1176/appi.ap.29.5.443
© 2005 Academic Psychiatry
Do Data Obtained From Admissions Interviews and Resident Evaluations Predict Later Personal and Practice Problems?
Steven L. Dubovsky, M.D.,
Michael Gendel, M.D.,
Amelia N. Dubovsky, B.A.,
Joseph Rosse, Ph.D.,
Robert Levin, Ph.D. and
Robert House, M.D.
Received December 14, 2004; revised March 7, 2005; accepted March 29, 2005. Dr. Steven Dubovsky is Professor and Chair, Department of Psychiatry, University at Buffalo, Buffalo, New York and Adjoint Professor of Psychiatry and Medicine at the University of Colorado School of Medicine, Denver, Colorado. Dr. Gendel is affiliated with the University of Colorado School of Medicine, Department of Psychiatry, Denver, Colorado. Mrs. Amelia Dubovsky is affiliated with New York University School of Medicine, Medical Student Education, New York, New York. Drs. Rosse and Levin are affiliated with the University of Colorado, Psychology, Boulder, Colorado. Dr. House is affiliated with the Department of Psychiatry, Denver Health Medical Center and the University of Colorado School of Medicine, Department of Psychiatry, Denver, Colorado. Address correspondence to Dr. Steven L. Dubovsky, Erie County Medical Center, 462 Grider St. Room 1182, Buffalo, NY 14215; dubovsky{at}buffalo.edu (E-mail). Copyright © 2005 Academic Psychiatry.
|
ABSTRACT
|
OBJECTIVE: The authors assessed whether current methods of evaluating residency applicants and residents identify psychiatrists who later develop evidence of impairment. METHOD: Residency admissions and performance data for all physicians who were enrolled in a psychiatry residency between 1965 and 1994 and who were referred to an impaired physician program up to 35 years later were matched for age and gender with a nonreferred physician from the same class. RESULTS: There were no significant differences between groups in admission interview assessments, performance ratings, or narrative observations by faculty during residency. CONCLUSIONS: Standard approaches do not identify physicians at risk of later impairment.
|
INTRODUCTION
|
Physician impairment is defined as "the inability to practice medicine adequately by reason of physical or mental illness, including alcoholism or drug dependency" (1). Using traditional definitions of impairment, the lifetime prevalence of illnesses that could impair physicians’ practices is around 8%–15% (2–4). Psychiatric illnesses, primarily depression, comprise 18% of total disability payments to physicians (5), and the equivalent of a medical school class commits suicide each year (6).
Along with depression, substance related disorders are among the most common causes of physician impairment (2, 3, 7, 8). The substance that physicians abuse most frequently is alcohol. Physicians may be more likely than other professionals to misuse prescription drugs (9), the rate of prescription drug dependence in physicians being 10–30 times that of nonphysicians (3) (probably related to ease of access). Recognizing such problems is especially important because the treatment of substance-dependent physicians can have a very favorable outcome (2, 10, 11).
Conditions that cause physician impairment can be identified in medical students and residents (12). Rates of major depression diagnosed by structured interview in house staff have been found to range from 21% to 40% (13). Department Chairs, residency program directors and chief residents responding to a survey at an academic medical center (13) felt that 12% of residents at their institution had "significant emotional problems" that impaired performance. In a questionnaire survey of residents in various specialties in a Canadian program (14), 13% of respondents had received psychiatric or psychological treatment in the past for psychiatric illness and 14% were receiving ongoing psychiatric care. Residents who were in treatment were not experiencing more or different stressors than those who were not in treatment, but they were more likely to prescribe tranquilizers for themselves. Of 341 first-year medical students in four medical schools located in different regions of the United States, 43% reported that they were frequent or heavy drinkers according to a standard rating protocol (15). In other studies, 5% of residents reported daily use of alcohol (16), and 11.5% of male residents and 7.3% of female residents reported using marijuana in the previous month (17).
Because psychiatric and substance use disorders can be precursors of later physician impairment (12) and impairment may not be evident for years after a psychiatric disorder develops, it would be desirable to identify risk factors early so that interventions could be instituted that could prevent later adverse outcomes. However, few attempts have been made to systematically identify students or residents who are prone to impairment (12). One recent comparison of all 68 graduates of the University of California, San Francisco School of Medicine (UCSF), who were disciplined by the California Medical Board from 1990 to 2000 with 196 comparison subjects matched by graduation year and specialty found that 38% of disciplined physicians versus 19% of comparison subjects (OR 2.15, p=0.02) had some notation of unprofessional behavior during medical school without any differences between groups in factors such as undergraduate and medical school grades and United States Medical Licensing Examination (USMLE) and Medical College Admission Test (MCAT) scores; however, no mention was made of possible illness or impairment (18).
We addressed the question whether any factor noted during the selection and training of residents might predict which graduates of the program would be referred to an impaired physicians program. To reduce confusion that might result from combining data from residencies in different specialties with different methods of evaluating residents, we utilized a large database from a single psychiatric residency. A psychiatry residency may be an appropriate place to start because illnesses that could lead to impairment may be somewhat more frequent among psychiatrists (19, 20). More importantly, psychiatric faculty may have more experience identifying psychiatric and substance related disorders, so if these faculty members could not identify potential causes of impairment, other specialists might not be expected to do any better. Our hypothesis was that significant differences would be identified between psychiatric residents who were referred to an impaired physicians program at some point after graduation and those who were not referred.
|
Method
|
This study was determined to be exempt by the Combined Institutional Review Board of the University of Colorado Health Sciences Center. Selection data (e.g., medical school grades, MCAT scores, dean’s letter, letters of recommendation, interview evaluations) and evaluations of residency performance were extracted from records of all physicians who entered a single large adult psychiatry residency (average number of residents 52–84) between 1965 and 1994. The resident selection process included 3–6 interviews by senior faculty who rated applicants on a numerical scale using the same anchor points used in evaluations of performance during the residency. Numerical ratings ranged from 1 (unacceptable under any circumstance) through 2 (marginal, with possible mitigating factors), 3–4 (acceptable) to 5 (exceptional, important contributions expected) and 6 (outstanding academic potential) and were supplemented by detailed narrative comments. The wording of anchor points changed somewhat over the years, but their substance did not. Although interviews were not standardized, correlations between numerical ratings of individual residents by different evaluators indicated a moderate to high degree of interrater reliability (on a randomly selected group of 5 residents from each class, r=0.78).
On each clinical rotation during their residencies, residents had been evaluated by 2–4 faculty supervisors who provided narrative comments and numerical ratings. On some rotations, residents were evaluated by the same faculty members who had interviewed them for admission to the residency. As was true of admissions interviews, performance ratings represented a single global assessment, while narrative comments were more detailed and specific.
A single investigator (A.N.D.) with no association with the residency program who adhered to a strict confidentiality agreement selected from this cohort all residents who at any time between 1965 and 1999 were referred to the Colorado Physicians Health Program (CPHP—an organization designated to work with physicians who may suffer from illness or behavioral problems). The same investigator then determined how many of these physicians had also been sanctioned for unprofessional or incompetent behavior by the Colorado Board of Medical Examiners (BME). For each graduate who was referred to CPHP, a comparison resident who was not referred to CPHP was matched for residency class, gender and age.
After all potentially identifying information was removed, all interview and residency numerical ratings and evaluation comments about each physician were reviewed in random order and rated by a group of senior clinicians and educators who did not participate in resident selection or teaching. Using consensus discussions, the group rated each narrative for mention of specific problems during residency along the dimensions of lack of competence, behavioral problems, performance deficits, maladaptive personality traits, substance use and Axis I diagnoses other than substance use disorders. Each comment was rated on a scale of 0–3, where 0 = no concern, 1 = observation of questionable significance, 2 = definite concern, and 3 = clear diagnosis or extensive data documenting a problem or deficit.
Data were analyzed using SPSS v. 11.5. Because residency evaluations and interview scores were normally distributed and variation of numerical ratings by different faculty of individual residents was low, mean interview and residency scores were compared by two-tailed paired t tests. One-way ANOVA was used to compare multiple means (e.g., mean residency evaluation grade and evaluation grade in each year of residency). Since mean performance ratings for each resident did not vary significantly across the 4 years of the residency for referred physicians (one-way ANOVA, p=0.09) and comparison subjects (one-way ANOVA, p=0.07), and since there were no significant differences between referred physicians and comparison subjects in year-by-year comparisons of mean performance scores, mean performance scores for the entire residency were compared. Concerns about competence, behavior, performance, personality, substance use and other Axis I disorders were compared using two-sided Fisher’s exact test.
|
Results
|
A total of 54 psychiatrists were identified who had been referred to CPHP (referred subjects), 26 of whom also received BME sanction during the study period. Many of these individuals had more than one problem, and several had more than one kind of outcome (e.g., practice monitoring through the BME and monitoring of personal treatment by CPHP). Problems requiring referral (and number of physicians) included substance related disorders, mood and/or anxiety disorders (17), substandard care not related to illness (10), boundary violations (6), antisocial or illegal behavior (5), medical illness that could impair practice (5), personality disorders (4), marital and other psychosocial stresses (3), disorganized or bizarre behavior (2), acute risk of suicide (1), and patient complaint (1).
Only five of the referred subjects left residency training before completing it (these residents subsequently completed residency elsewhere and were referred to CPHP as practitioners). A substantial group of subjects (23) recovered with or without monitored treatment. The licenses of 18 physicians were sanctioned by the BME through probation, practice monitoring and/or letters of admonition. Licenses of six other physicians were revoked, two physicians’ licenses were suspended and later reinstated and five physicians received long-term monitoring by CPHP without reporting to the BME. One BME case was under litigation at the time of the study.
Mean admission interview scores for referred subjects (4.54 [SD=0.6]) and comparison subjects (4.58 [SD=0.6]) were not significantly different (t=0.4088, df=53, p=0.6844). Mean residency scores for referred physicians (4.57 [SD=0.8]) and comparison subjects (4.73 [SD=0.6]) were also statistically similar (t=1.130, df = 53, p=0.2636). There were no differences between referred subjects and comparison subjects in any other selection information.
There were no differences between referred physicians and comparison subjects in number or content of comments in admissions interviews or residency evaluations about competence (2-sided p=0.0799), behavior (p=0.4643), performance (p=0.2788), personality traits (p=0.0657), or Axis I disorder not involving substance use (p=0.5385). The only actual mention of substance use was during the admission interview of one referred physician who reported having been treated for a substance use disorder prior to applying to the residency and was already under the supervision of the BME. There was also no difference between groups in consistency of observations related to potential impairment. Concerns were raised by faculty about the same subject in at least one admissions interview and at least one residency evaluation regarding competence in two referred subjects and two comparison subjects, about performance problems in six referred subjects and two comparison subjects, and about personality issues in five referred subjects and five comparison subjects. The same faculty member never noted the same problem in a given resident in both admissions and residency evaluations and the same concern was never identified in both an admissions interview and a residency evaluation for behavior problems, substance use or axis I diagnosis such as major depression.
|
Discussion
|
Neither a selection process involving open ended interviews nor observations during postgraduate education by experienced psychiatrists successfully differentiated between residents who later did or did not exhibit problems that led to referral to CPHP with or without sanction by the BME. This finding is obviously limited by the retrospective method, the global method of scoring resident evaluations, and involvement of multiple evaluators. However, in the study of disciplined physicians mentioned earlier (18), nonstandardized medical school records completed by multiple faculty members were judged to contain sufficient information to retrospectively rate concerns about unprofessional behavior according to a 5-point scale that included "no adverse comments," comments from a single instructor about immaturity, negative comments in one course, comments about unprofessional behavior in two or more courses and severe comments, even if they were present on only one occasion.
Another limitation of the present study is that the only form of practice impairment that was measured involved referral to regulatory agencies, which undoubtedly requires a higher threshold than less blatant or prodromal forms of impairment. Psychiatrists who became known to the BME without having also been referred to CPHP and who might have a different pattern of impairment were not studied. We had no information about referral of psychiatrists from this residency to similar agencies in other states or about referrals of other specialists to CPHP. However, many graduates of this psychiatry residency have remained in the state, and we had extensive data about those psychiatrists who were evaluated by CPHP.
Our data cover three decades during which the orientation of the profession toward residency education evolved substantially, and the faculty themselves changed. However, there was no difference between earlier and later cohorts in the frequency of identification of resident problems. On the other hand, this analysis involves one of the largest groups of residents studied thus far, with a longer follow up than has previously been reported. Retrospective evaluation of data that were not obtained in a research protocol does not definitively disconfirm the study hypothesis, but to the extent that the lack of correlation between any routinely evaluated factor and subsequent impairment is at least intriguing, several possible explanations are worth further discussion.
One consideration is that admissions interviews and residency performance evaluations are not structured to identify potential sources of impairment. No matter what their clinical expertise, the orientation of faculty members is appropriately toward teaching and identifying problems in learning rather than toward diagnosing illness. Even in an era in which the focus of clinical teaching was on understanding the patient by exploring the personal psychology of the resident, evaluation for significant health and behavioral problems was not a routine component of the evaluation of applicants or of residents.
In the McNamara and Marquiles study mentioned earlier (21), 30% of training program directors had had little formal education about physician impairment and just one third of programs provided such education for residents. Lack of orientation toward identifying potential impairment may explain why in a survey sent to all members of the Society for Academic Emergency Medicine, 1% of residents were identified by program directors as having an alcohol problem, compared with 12.6% of the same group who were found on an alcoholism screening instrument (CAGE) to have possible alcohol problems (21). Without adequate expertise or education in the identification of problem behaviors and symptoms in trainees, any expectation of earlier recognition and intervention is not likely to be realized.
Even if they are asked directly, applicants often fail to mention psychiatric illnesses, substance use problems, or even criminal records (22). Indeed, they may do their best to conceal such issues out of the understandable expectations that the information is not likely to be revealed, that their application would be jeopardized if it did, and by the feeling that such information is private. In our subjects, the only applicant who revealed a history of substance abuse was required to do so by the BME.
Data about potential impairment that might be available from other sources also often are not shared with those responsible for residency education. For example, subjects in a survey of physicians in Alcoholics Anonymous (19) reported that they knew of other untreated, impaired students and faculty but they had not shared this information with the training program. When physicians are terminated from a residency program and then transfer to another program or change specialties, reasons for the termination often are not conveyed to the new training program (22). Fear of legal action and political pressure may be among the reasons why information about illness or behavioral problems in trainees is not revealed.
Even if prodromal features of substance related disorders and other conditions that could later impair practice were present, early identification in residents may be difficult because psychological and physical functioning are usually affected long before occupational impairment occurs (8). The problematic use of substances by physicians in training frequently begins well before medical school (8), but performance, at least early in medical school, may be unaffected. In one study alcoholic medical students were found to score higher on Step I of the USMLE examination than students who did not drink excessively (8). From this vantage point, the observation of work performance is not likely to be helpful in predicting who will develop practice impairment. The remitting and relapsing nature of many substance use and mood disorders, as well as the impossibility of predicting future events that might tip the balance in favor of impairment, may also make recognition difficult until impairment is more pervasive.
If faculty were educated about the prevalence and early manifestations of conditions that can result in practice impairment, it might be possible to identify at least some residents at risk at a point at which intervention might be simpler and more effective. However, an increasing emphasis on privacy of medical information and a litigious environment for medical education have made the evaluation of anything other than observable performance more complex.
|
Conclusions
|
The Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to "identify impaired physicians, and intervene appropriately" (23). In view of the failure of experienced psychiatric faculty to predict later physician illness or professional behavior problems based on extensive information obtained during application to and performance in a psychiatric residency in which parallel process and introspection are used as teaching techniques, it does not seem likely that current methods of resident evaluation are likely to fulfill this charge. Considering the lag between developing a psychiatric or substance use disorder and that leads to later practice impairment, a prospective study of potential correlates of later impairment would have to be too long to be practical. A multi-site study involving graduates of residencies in more than one specialty might have more power to detect subtle indicators of risk, but it is not clear that such indicators would be useful clinically. Since peers may have more information about early indicators of potential impairment, studying methods of encouraging residents to report evidence of relevant disorders, perhaps to a peer intervention group, may prove more fruitful. Any future research must include a protocol for data collection consistent with current standards of confidentiality and appropriate educator-student boundaries.
|
REFERENCES
|
- Reuben DB, Noble S: House officer responses to impaired physicians. JAMA 1990; 263:958–960[Abstract/Free Full Text]
- Skipper GE: Treating the chemically dependent health professional. J Addict Dis 1997; 16:67–73[Medline]
- Flaherty JA, Richman JA: Substance use and addiction among medical students, residents and physicians. Psychiatr Clin North Am 1993; 16:189–197[Medline]
- Sansone RA, Wiederman MW, Sansone LA: Physician mental health and substance abuse. Arch Fam Med 1999; 8:448–451[Abstract/Free Full Text]
- Wall BW, Appelbaum KL: Disabled doctors: the insurance industry seeks a second opinion. Bull Am Acad Psychiatry Law 1998; 26:7–19
- Roy A: Suicide in doctors. Psychiatr Clin North Am 1985; 8:377–387[Medline]
- Ratzan RM: Broken wings, mandalas, impaired physicians, and medical examining boards. J Emergency Med 1997; 15:885–888[CrossRef][Medline]
- Collins GB: New hope for impaired physicians: helping the physician while protecting patients. Cleveland Clinic J Med 1998; 65:101–106[Medline]
- Brooke D: Impairment in the medical and legal professions. J Psychosomatic Res 1997; 43:27–34[CrossRef][Medline]
- Strang J, Wilks M, Wells B, et al: Missed problems and missed opportunities for addicted doctors: we need a special service for doctors addicted to drugs or alcohol. Br Med J 1998; 316:405–406[Free Full Text]
- Pelton C, Lang DA, Nye GS, et al: Physician diversion programs experience with successful graduates. J Psychoactive Drugs 1993; 25:159–164[Medline]
- Shapiro J, Prislin MD, Larsen KM, et al: Working with the resident in difficulty. Fam Med 1987; 19:368–375[Medline]
- Urbach JR, Levenson JL, Harbison JW: Perceptions of housestaff stress and dysfunction within the academic medical center. Psychiatr Qu 1989; 60:283–295
- Hurwitz TA, Beiser M, Nichol H, et al: Impaired interns and residents. Can J Psychiatry 1987; 32:165–169[Medline]
- Forney MS, Ripley WK, Forney PD: A profile and prediction study of problem drinking among first-year medical students. Int J Addictions 1988; 23:767–779[Medline]
- Hughes PH, Conard S, Baldwin DC, et al: Resident physician substance use in the United States. JAMA 1991; 265:2073
- Westermeyer J: Substance use rates among medical students and resident physicians. JAMA 1991; 265
- Papadakis MA, Hodgson CS, Teherani A, et al: Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Acad Med 2004; 79:244–249[CrossRef][Medline]
- Bissell L, Skonna JK: One hundred alcoholic women in medicine: an interview study. JAMA 1987; 257:2939–2944[Abstract/Free Full Text]
- Hughes PH, Baldwin DC, Sheehan DV, et al: Resident physician substance use, by specialty. Am J Psychiatry 1992; 149:1348–1354[Abstract/Free Full Text]
- McNamara RM, Marquiles JL: Chemical dependency in emergency medicine residency programs: perspective of the program directors. Ann Emerg Med 1994; 23:1072–1076[Medline]
- Anonymous. The problem resident: learning from our mistakes. Fam Med 1999; 31:729–731[Medline]
- American Medical Association: Directory of Graduate Medical Education Programs. Chicago, AMA, 1992
This article has been cited by other articles:
|
|
|
|
 
C. Horn, B. Plazas Snyder, J. H. Coverdale, A. K. Louie, and L. W. Roberts
Educational Research Questions and Study Design
Acad Psychiatry,
May 1, 2009;
33(3):
261 - 267.
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2005
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
