Advanced Search

Regular Article | March 01, 2004

Assessment of Competency in Child and Adolescent Psychiatry Training

John Sargent, M.D.; Sandra Sexson, M.D.; Steven Cuffe, M.D.; Martin Drell, M.D.; Timothy Dugan, M.D.; Peter Ferren, M.D., M.P.H; Wun Jung Kim, M.D., M.P.H; Dorothy Stubbe, M.D.; Bonnie Zima, M.D., M.P.H; Trish Brown, M.A.

Academic Psychiatry 2004;28:18-26. 10.1176/appi.ap.28.1.18

View Article Information

Dr. Sargent is Professor of Psychiatry and Pediatrics at the Baylor College of Medicine, Houston Texas. Dr. Sargent, Dr. Sexson, Dr. Cuffe, Dr. Drell, Dr. Dugan, Dr. Ferren, Dr. Kim, Dr. Stubbe, Dr. Zima, and Dr. Brown are all members of the Work Group on Training and Education of the American Academy of Child and Adolescent Psychiatry. Address correspondence to Dr. Sargent, Baylor College of Medicine, One Baylor Plaza, #350, Houston, TX 77030; asargent@bcm.tmc.edu (E-mail).

Article

Tables

References

text A A A

Abstract

Objective: Residency training programs in all areas of medicine are required to identify core competencies expected of all graduates and develop methods to assess and ensure attainment of these competencies. To assist with this process for residency programs in child and adolescent psychiatry, the Work Group on Training and Education of the American Academy of Child and Adolescent Psychiatry has developed several principles of the assessment process and compiled a variety of assessment methodologies for use in assessing competency. The principles of assessment include 1) residents should share responsibility for assessment; 2) assessment should be an open, ongoing and predictable process; 3) a wide range of evaluators should be utilized in the process; 4) residents should demonstrate competency in a variety of formats; 5) the goal is for 100% of residents to achieve core competencies. Methods: Sample methods of assessment are provided in the report with special attention to how the method could be used in child and adolescent psychiatry. Conclusion: A multi-method, multi-evaluator for process of assessment is recommended.

Figures in this Article

Residency training programs in the United States are entrusted with the responsibility to educate and graduate competent physicians. This responsibility is receiving increasing national attention both within both residency training regulatory bodies and residency training programs (1). The Work Group on Training and Education of the American Academy of Child and Adolescent Psychiatry has responded by proposing a set of sample core competencies for residents in child and adolescent psychiatry training that are consistent with the goals of the Accreditation Council for Graduate Medical Education (2). The multifaceted nature of these competency expectations adds significant complexity to the always challenging task of assessing the knowledge, skill, and attitude attainment of trainees. The purpose of this paper is to set forth sample guidelines for assessment of residents’ knowledge, skills, and attitudes in each core competency.

Similar to previous discussions of this issue (3), we believe assessment methods should be:

Residency training is a unique adult learning experience in which the trainee is a student, an apprentice, and an employee all at the same time. A resident is asked to balance the demands of pursuing an education while providing clinical services. Assessment of educational achievement and clinical performance thus requires a collaborative dialogue between the resident and the faculty, which is mediated by the residency training director (4). The following recommendations for assessment are designed to foster the necessary agreement among faculty and residents about what to learn and the commitment to ensure that learning takes place.

The Resident Should Share Responsibility for Identification and Remediation of Deficiencies in Knowledge, Skills, and Attitudes With the Training Director and Faculty

Residents must establish a trajectory of self-learning and the desire to continually assess their abilities in residency that will persist over the resident’s career. Traditionally, training directors and faculty have assumed the primary responsibility for monitoring the progress of trainees. However, this approach does not require residents to actively demonstrate to themselves their mastery of competencies and reflect on areas of relative weakness. While training directors have an ethical and administrative obligation to ensure that graduates of residency training programs are competent when they complete their training, residents are responsible for developing their knowledge, skills, and attitudes throughout their careers to ensure the public trust and the safe, effective treatment of children and their families.

The Assessment of Residents Should Be an Ongoing, Open, and Predictable Process

Collaborative assessment in which the resident is an active participant should occur in an atmosphere of openness (3). Together, the resident and training director should document mastery of the core competencies and meet regularly to discuss and compare impressions of the resident’s progress toward achieving these competencies. In addition, faculty evaluators should be expected to review and discuss feedback they submit to the training director with the resident. To facilitate this process, the training director should assist faculty in developing skills that are necessary in order to concisely identify and constructively communicate areas of strength and weakness. Identification of strengths helps the resident appreciate the skills, knowledge, and attitudes that they will build on through training and practice. Recognition of weaknesses helps the resident understand areas on which to focus and build skills. This process should occur regardless of the overall competency attainment of any individual resident. Sometimes the faculty is uncomfortable giving negative feedback. Faculty members may need assistance in mastering this ability and will need the support of their training director to ensure that constructive criticism is appreciated by the resident.

A Wide Range of Evaluators Should Be Utilized to Ensure the Most Accurate View of Resident Competency

Efforts should be made to assess competency through direct observation of clinical performance by traditional as well as nontraditional evaluators. In each training program there are certain individuals who are easily identifiable as appropriate evaluators of residents’ performance, such as ward and service attending physicians, supervisors, and didactic seminar leaders. However, traditional evaluators often must base their assessments on a resident’s self-report of a clinical encounter or other second-hand means. Direct observation of the resident’s clinical work, through live observation of interviews or video taped review of clinical encounters should be included in any resident’s assessment process. Additional assessments from other individuals whose roles in a resident’s education are significant should also be obtained. These individuals may be supervising attending physicians during on-call situations, research mentors, nonphysician members of care teams (i.e., psychologists, social workers, nurses, and teachers) as well as patients and their families.

Residents Should Be Encouraged to Demonstrate Competency Through a Variety of Formats

Individuals learn knowledge, skills, and attitudes through a variety of educational methods such as assigned reading, lectures, clinical demonstrations, discussion, interactive demonstrations, direct clinical experience, and feedback on their performance. There are a variety of formats through which residents can demonstrate competency, including written multiple-choice or essay examination, oral examination, written or oral presentation, didactic discussion participation, and actual clinical performance. Although residents should have a basic ability to communicate learning through all of these formats, multiple assessment formats should be provided so that an accurate evaluation of competency can be based on either a composite of several assessments or the resident’s best performance among these formats. This will ensure that residents are able to demonstrate competency through both universal assessment measures and through formats that best reflect their abilities.

The Goal Is for 100% of Residents To Achieve the Core Competencies

Core competencies within any program are written to be comprehensive but achievable and represent a flexible minimum standard for child and adolescent psychiatrists graduating from training programs. Nonetheless, mastery of the core competencies requires an active partnership between resident and training director to ensure success. The assessment process is crafted not as a method of stratifying residents into a hierarchy of performance capabilities, but rather as a method of enabling all residents to emerge from training as competent child and adolescent psychiatrists.

Assessment Methods

Abstract | Assessment Methods | Conclusions | References

The following tables describe sample methods of assessment and their use in child and adolescent psychiatry residency training. A multi-method, multi-informant method is likely to be most effective for training programs. Each method will need to be modified to fit the unique culture of each training program (5). While some are readily quantifiable, others are more subjective. Some depend upon the specifics of rotations and training sites, and others depend upon the individual interests, goals, and qualities of each resident. A number of assessment methods (Table 1) reflect a formal evaluation process. Such methods are oriented toward directly evaluating residents’ competency in a reproducible fashion. Some examination formats (e.g., objective structured clinical examinations, simulated patients, computer based simulations) are methods that will benefit from pilot project development efforts, efforts of several programs working together, and collaborations between programs and national professional organizations. Other methods (Table 2) require a program to encourage independent resident activity and assess the competencies demonstrated. The final group of assessment methods (Table 3) makes use of a check list format and expands the range of evaluations to many of those who work with a particular resident whose evaluation can assist in the formation of a comprehensive portrait of resident competency.

Conclusions

Abstract | Assessment Methods | Conclusions | References

Training programs will need to develop their own compendium of assessment methods in order to achieve the most practical, comprehensive, and equitable process for their trainees and faculty. Each training program must evaluate their own resources and utilize methods that work. Most programs will appreciate that the use of several methods with assessments by many differing evaluators will be most effective.

Regardless of the methods of assessment utilized by a training program, each program will need to address the following key issues to ensure a successful assessment experience for each trainee:

TABLE 1. Sample Methods of Assessment: Formal Examinations or Evaluation Processes

View Large | Download Slide(.ppt)
Add to My POL

TABLE 2. Sample Methods of Assessment That Encourage Independent Resident Activity

View Large | Download Slide(.ppt)
Add to My POL

TABLE 3. Sample Methods of Assessment: Using Other Sources to Provide Evaluations

View Large | Download Slide(.ppt)
Add to My POL

References

Abstract | Assessment Methods | Conclusions | References

Accreditation Council for Graduate Medical Education: ACGME Outcome Project. Available at:
http://www.acgme.org

Sexson S, Sargent J, Zima B, Beresin E, Cuffe S, Drell M, Dugan T, Fox G, Kim WJ, Matthews K, Sylvester C, Pope K: Sample core competencies in child and adolescent psychiatry training. Academic Psychiatry 2001; 25:201—213[PubMed][CrossRef]

Epstein RM, Hundert EM: Defining and assessing professional competence. JAMA 2002; 287:226—235[PubMed][CrossRef]

Dauphinee WD: Assessing clinical performance: where do we stand and what might we expect? JAMA 1995; 274:741—743[PubMed][CrossRef]

Frankford DM, Konrad TR: Responsive medical professionalism: integrating education, practice, and community in a market-driven era. Acad Med 1998; 73:138—145[PubMed][CrossRef]

Norcini JJ, Swanson DB, Grosso LJ, Webster GD: Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence. Med Educ 1985; 19:238—247[PubMed][CrossRef]

Yang JC, Laube DW: Improvement of reliability of an oral examination by a structured evaluation instrument. J Med Educ 1983; 58:864—872[PubMed]

MacRae HM, Cohen R, Regehr G, Reznick R, Burnstein M: A new assessment tool: the patient assessment and management examination. Surgery 1997; 122: 335—343

Blane CE, Calhoun JG: Objectively evaluating student case presentation. Invest Radiol 1985; 20:121—123[PubMed][CrossRef]

Joorabchi B: Objective structured clinical examination in a pediatric residency program. AJDC 1991; 145:757—762[PubMed]

Kaiser S, Bauer JJ: Checklist self-evaluation in a standardized patient exercise. Am J Surg 1995; 169:418—420[PubMed][CrossRef]

Matsell DG, Wolfish NM, Hsu E: Reliability and validity of the objective structured clinical examination in paediatrics. Med Educ 1991; 25:293—299[PubMed][CrossRef]

Petrusa ER, Blackwell TA, Ainsworth MA: Reliability and validity of an objective structured clinical examination for assessing the clinical performance of residents. Arch Intern Med 1990; 150:573—577[PubMed][CrossRef]

Singer PA, Robb A, Cohen R, Norman G, Turnbull J: Performance-based assessment of clinical ethics using an objective structured clinical examination. Acad Med 1996; 71:495—498[PubMed][CrossRef]

Barrows HS: An overview of the uses of standardized patients for teaching and evaluating clinical skills: AAMC. Acad Med 1993; 68:443—451[PubMed][CrossRef]

Gomez JM, Prieto L, Pujol R, Arbizu T, Vilar L, Pi F, Borrell F, Roma J, Martinez-Carretero JM: Clinical skills assessment with standardized patients. Med Educ 1997; 31:94—98[PubMed][CrossRef]

Keynan A, Friedman M, Benbassat J: Reliability of global rating scales in the assessment of clinical competence of medical students. Med Educ 1987; 21:477—481[PubMed][CrossRef]

Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP: Use of peer ratings to evaluate physician performance. JAMA 1993; 269:1655—1660[PubMed][CrossRef]

Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians. Acad Med 1997; 72 (suppl 1):582—584

Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ: How accurate are faculty evaluations of clinical competence? J Gen Intern Med 1989; 4:202—208[PubMed][CrossRef]

Cohen R, Rothman AI, Poldre P, Ross J: Validity and generalizability of global ratings in an objective structured clinical examination. Acad Med 1991; 66:545—548 [PubMed][CrossRef]