Acad Psychiatry 30:279-280, August 2006
doi: 10.1176/appi.ap.30.4.279
© 2006 Academic Psychiatry
Assessing the ACGME Competencies in Psychiatry Training Programs
Stewart Mennin, Ph.D.
Received December 13, 2005, and accepted December 16, 2005. Dr. Mennin is professor emeritus,Department of Cell Biology and Physiology, University of New Mexico School of Medicine, Albuquerque, New Mexico. Address correspondence to Dr. Mennin, Avenida Jacutinga, 579 Apto 41, Sao Paulo, Brazil CEP 04515–030; smennin{at}gmail.com (E-mail). Copyright © 2005 Academic Psychiatry.
The Accreditation Council for Graduate Medical Education (ACGME) changed the landscape for medical schools when it defined and mandated the assessment of residents in six competency areas. Although a shift in educational perspective had been in progress for some time, the ACGME’s decision disturbed the status quo tolerably and facilitated the reexamination of current education practices. It has now become necessary for graduate programs to reexamine their approaches to teaching, learning and assessment.
The Residency Review Committee requirements for general psychiatry training in which each program is required to demonstrate that it has "an effective plan for assessing resident performance throughout the program and for using assessment results to improve resident performance" is the focus of contemporary dialogue. An effective approach has been developed by the American Association of Medical Colleges’ (AAMC) Group on Educational Affairs, which sponsored "Conquering the Competencies Challenges—the SOAP Approach" as part of a larger project, entitled CACHE: Competencies Across the Continuum of Health Education. It provides self-guided modules designed to enhance residency program directors’ abilities to address and assess effectively and efficiently the six core ACGME competencies. It is available from the AAMC at http://www.aamc.org/members/gea/cubesoapstart.htm together with a workbook guide at http://www.aamc.org/members/gea/cubeworkbook.pdf.
"Assessing the ACGME Competencies in Psychiatry Training Programs" (1) is a welcome and timely contribution to the literature that can help residency directors address the challenges of assessing competence. The article frames competence in a larger context and outlines essential principles and concepts of assessment. It also provides a brief and clear description of selected assessment tools. Most teachers in medical schools are not fluent in or knowledgeable about contemporary practices of teaching, learning, and assessment. Without a coherent working knowledge of formative and summative assessment, standard setting and benchmarks, competency and objectives, validity and reliability, feedback, blueprinting and assessment planning, and norm and criterion referenced assessment, the vital relationship between assessment and learning is reduced to an assemblage of tools and techniques disconnected from significant learning experiences.
Key questions to be considered include: What is to be assessed and for what purpose? How will the different domains be sampled? If there are measurements from multiple sources, how will they be combined? Consideration of an effective assessment plan leads to the conclusion that assessment is more about educational design than measurement and logistical problems (2). An effective plan likely will show that multiple methods of assessment need to be combined carefully in order to achieve comprehensive assessment of competency.
We have gotten very good at taking things apart, measuring the pieces, and making reliable inferences about them. In many medical schools, planners of curricula and graduate programs begin discussions about assessment with a focus on individual assessment tools and methods without first having agreed upon competencies, a blueprint for an overall assessment program plan or a clear understanding of the congruence of teaching, learning, and assessment. This often results in an increase in fragmentation which can lead to a decrease in humanism and educational coherence in undergraduate and graduate curricula. More and more we are learning that the relationships between things and not the things themselves are the key to understanding competence. Each institution and residency program will benefit from an in-depth dialogue about what is or is not a competency. There will need to be consideration for the implications of the new ACGME standards for teaching, learning, and assessment.
Psychiatry residents practice and learn in multiple dynamic and complex environments and work with different teams. It is difficult to measure individual competencies by themselves without consideration of their interconnections with other competencies. Competency is expressed differently in different contexts. For example, competency in interpersonal and communication skills may emphasize a particular set of tasks as it is expressed in the context of patient care and a somewhat different set of tasks when viewed in the context of management of the work environment or professional self-development (3).
Many of the important things in life are difficult to measure, and medical education is no exception. There is still a strong bias among faculty that favors more "objective" assessment tools, such as check lists, as opposed to more subjective ones, such as global ratings.
There is a widespread misconception about the relationship between subjectivity and reliability. People believe that subjective measures are inevitably unreliable, and that objective measures are more reliable by definition. This is not true. Objective measures can be unreliable and subjective measures can yield reproducible results (4):
Suppose that we compose 10 pieces of music ourselves and that we select 10 pieces of music by Mozart. We then submit them to a panel of 10 experts who will judge the musical artistry of all 20 pieces. Finally, we ask the experts to award a prize for the best composer. In all likelihood the panel will give the prize to Mozart (probably unanimously). The outcome of this test would be unaffected by a different selection from both our own and Mozart’s music, or by a different panel of experts. The decision is "generalizable" to all sources of variance (and, nevertheless, highly subjective). Essential in this matter is that careful sampling has taken place, in the form of samples of our music, of Mozart’s music, and of the panelists. In such a situation the underlying concept of reproducibility applies and leads to reproducible outcomes (2).
Advances in our understanding of how physicians learn, combined with new approaches to assessment, have helped to move the focus of assessment from measurement to education. This challenge is taking place during a time of increasing pressure on U.S. medical schools to earn more of their operating budget from patient care and research. Education and assessment of learners takes time and can be expensive. There is a palpable risk that the time required to respond to the ACGME requirements will be perceived, in the current economic climate, as a conflict with patient care and research activities. We cannot afford to make this mistake. If competency is a habit (5), we must increase our awareness of it, explore it, discover our connections to it, and use it as part of the learning and the assessment of psychiatry residents. "Assessing the ACGME Competencies in Psychiatry Training Programs" points us in the right direction in a clear and accessible way. Educational improvements are at risk in a climate of increased fiscal demands on medical school faculty.
|
REFERENCES
|
- Swick S: Assessing the ACGME competencies in psychiatry traning programs. Acad Psychiatry 2006; 30:330–351[Abstract/Free Full Text]
- Schuwirth LWT, Van der Vleuten CPM: Changing education, changing assessment, changing research? Med Educ 2004; 38:805–812[CrossRef][Medline]
- Hays RB, Davies HA, Beard JD, et al: Selecting performance assessment methods for experienced physicians. Med Educ 2002; 36:910–917[CrossRef][Medline]
- Van der Vleuten CPM, Norman GR, De Graaf E: Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ 1991; 25:110–118[Medline]
- Epstein RM, Hundert EM: Defining and assessing professional competence. J Am Med Assoc 2002; 287:226–235[Abstract/Free Full Text]
Get information about faster international access.
Privacy Policy
Copyright © 2006
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
REFERENCES
