Acad Psychiatry 30:498-504, November-December 2006
doi: 10.1176/appi.ap.30.6.498
© 2006 Academic Psychiatry
Online Resources for Assessment and Evaluation
Sheldon Benjamin, M.D.,
Lisa I. Robbins, M.D. and
Simon Kung, M.D.
Received February 15, 2006; revised May 15, 2006; accepted June 16, 2006. Dr. Benjamin is affiliated with the Department of Psychiatry, University of Massachusetts Medical School, Worcester, Massachusetts. Dr. Robbins is in private practice in Livingston, New Jersey. Dr. Kung is affiliated with the Department of Psychiatry and Psychology, Mayo Clinic College of Medicine, Rochester, Minnesota. Address correspondence to Dr. Benjamin, 55 Lake Avenue North, Worcester, MA 01655; sheldon.benjamin{at}umassmed.edu (e-mail).
|
ABSTRACT
|
OBJECTIVE: The Accreditation Council for Graduate Medical Education and the Liaison Committee on Medical Education have mandated the transition from written global evaluation methods to competency-based assessments in resident and medical student training. Assessment of competency requires analysis of performance data from numerous sources. This article reviews characteristics of Web-based evaluation and assessment systems and recommends areas for further development. METHOD: The authors review functions common to a variety of online evaluation and curriculum management systems with attention to their adaptation for competency documentation and assessment. Details of online global assessments, examination methods, electronic portfolios, procedure/case logs, and survey systems are provided along with a list of Internet resources. RESULTS: Online evaluation and assessment systems not only provide data on trainee competence but can provide valuable feedback on faculty teaching, curriculum quality, and the learning environment. Suggestions for future development, legal ambiguities that could become obstacles, and the need to clarify the responsibility for funding this work are discussed. Web-based performance assessment and evaluation are seen in the larger context of moving toward competency assessment of practicing physicians. CONCLUSIONS: Online assessment systems offer advantages over paper systems in allowing robust data analysis, reporting, and flexibility. A national online procedure/case log tracking system would facilitate gathering data that could provide one type of experiential benchmark for determining competency.
|
INTRODUCTION
|
The Accreditation Council on Graduate Medical Education- and Liaison Committee on Medical Education-mandated transition from older written methods for faculty assessment of trainee skills to newer competency-based assessments has required the development or adaptation of assessment tools that are relatively new to the medical profession. The core competencies for residency training were introduced along with a menu of various competency assessment methods known as the "competency assessment toolbox" (Appendix 1) (1), only two of which—global ratings and written exams—were previously used by most training programs. The Accreditation Council on Graduate Medical Education (ACGME) itself is beginning a gradual transition to online reporting by training programs. Training directors have been required to specify the assessment tools being used for each core competency in the ACGME’s online Accreditation Data System (ADS) since 2005. With the explosion of new assessment methods, training directors have been faced with a new dilemma: how to synthesize the numerous types of data being collected into a cohesive method of competency determination that would also allow for future verification of competency.
The maturation of online solutions coincident with this transition has provided medical faculty with an expanded armamentarium of possible tools. What began with a few medical schools and residency programs developing in-house computer-assisted evaluation strategies quickly blossomed into a plethora of vendors of online assessment systems competing in the medical training marketplace. Although the field of medical competency assessment is still in its infancy, these systems have been rapidly adopted.
Online assessment systems were initially designed to automate the process of collecting standard written evaluation forms and provide an efficient means of accomplishing various administrative tasks inherent in running training programs. In addition to the task of learner assessment, these systems provide a mechanism for addressing other training needs, including quality assessment of faculty, curriculum, teaching environment, and the provision of feedback to trainees, faculty, and administration.
Although many programs simply converted their paper evaluation systems to online versions, technology can be harnessed to assess competency in ways that are just being realized. At the medical school level, online assessment systems may be implemented independently or as part of global online education solutions, providing resources for curriculum management across all 4 years. Implementation of a single online assessment system, however customizable for each program, provides an institution the ability to compare performance across programs, spot quality issues in a manner that allows early intervention, and prepare system-wide reports for various supervising bodies.
This article will explore some of the types of computer-assisted and online assessment tools currently available, and suggest areas in need of further development.
|
Overview of Online Assessment Systems
|
A number of residency programs and medical schools utilize evaluation and curriculum management systems (CMS) developed in-house. Some of these efforts have been compared by the Computer Resources in Medical Education group (2). Many schools have implemented any of a number of available Web portal systems to facilitate campus communication, administration, and access to CMS. Appendix 2 lists several commercial providers of online evaluation systems for students and residents. CMS systems for medical schools (Appendix 2) (3) include examination and assessment modules as well as mechanisms for course material development and distribution, and faculty-learner and interlearner communication. Both types of systems are moving into electronic portfolio development. In addition, the ACGME has developed a free Web-based evaluation system for urology residencies that provides a number of tools for online documentation of competency (4). This optional program indicates ACGME interest in exploring the provision of online assessment tools and may foreshadow future access to Web-based ACGME evaluation instruments for other specialties.
Administrative components included in different systems are listed in Appendix 3. Implementation may vary widely among systems, with no single solution appropriate for all programs.
Concerns shared by all system users include data security, permanence, and portability. Evaluative data must remain confidential and protected. Programs typically allow administrators to determine who should be permitted to view evaluations and whether the viewer will be able to see individual versus aggregate data. A flexible permission structure should allow coordinators of various aspects of the educational program to view evaluative data about their area. Data entry should take place over a password-protected secure interface (e.g., using Secure Sockets Layer technology) and should be maintained on secure servers with adequate backup, redundancy, and loss protection. Given the inability of providers to guarantee their own future, it is critical for programs to receive backup copies of data in a common, easily read format (e.g., comma-delimited values). Programs must then initiate their own redundant systems to insure that data backups are not lost and are filed in a manner that facilitates later access (e.g., placing a CD with all evaluative data in the trainees' file on graduation). The ACGME requires that programs using online assessment systems keep a printed copy of the final summative training evaluation in each trainee’s file. Some programs derive from the accumulated electronic data a final report documenting competencies attained. Paper training files are typically thinned to their essentials at a predetermined interval following graduation. Although there can be little doubt that electronic training files will become the standard, a standard for maintenance of these files is yet to emerge.
There are major advantages for institutions to agree on a single online assessment system for the graduate medical education (GME) community. A single system required of all GME programs allows an institution to verify compliance with ACGME standards, conduct system-wide quality surveys, and system-wide work-hour monitoring, and develop periodic assessment initiatives across programs. An institution-wide program saves money and can provide a layer of in-house support to users. It also allows for a collaborative environment in which programs learn from one another.
|
Assessment Tools
|
Web-Based Evaluations
Web-based evaluation systems have enabled training programs to derive more useful data from global evaluations than did paper forms. Advance scheduling, automated reminders, and compliance tracking improves the compliance rate by both faculty and trainees while preserving administrative staff time for other tasks. The assurance of anonymity encourages trainee feedback about the curriculum, faculty, or training environment. Computerized evaluation forms are easily modified if necessary. Questions from different evaluation forms may be grouped together for analysis. Examples include pulling together all global assessments related to a given core competency or type of training as a single score for a summative report. Web-based systems may allow keyword designations for any evaluation question, didactic session, rotation, experience, or supervision type, allowing programs to both derive reports documenting numerous approaches to the assessment of a given competency or required experience and utilize a given event toward more than one competency assessment. Individual and group statistics are typically calculated comparing an individual’s performance with that of a peer group. Automated notification systems for below threshold performance provide training directors, clerkship directors, advisers, or individual site directors timely notification of situations requiring intervention or remediation. Reporting tools allow trainees and faculty to track their own performance over time. Individual competency-based evaluation summaries may be prepared for faculty meetings and periodic reviews. Summaries by training year or for an entire program allow directors to spot trends that facilitate early course correction. 360-degree evaluations are easily accomplished by providing limited system access to all mandated evaluators.
In addition to uses for trainee and curriculum evaluation, Web-based systems lend themselves to the establishment of a culture of feedback within a department. Annual feedback summaries to faculty may be one factor considered in faculty promotion decisions or faculty performance evaluations. These same systems may be adapted to provide hospitals with competency assessment systems for their physician staff.
Surveys
Some online systems are sufficiently flexible to allow the collection of on-demand survey data within a program. Surveys can be used to look at specific aspects of a given training experience not anticipated by the routine curriculum evaluation process, for further examination of detected trends, or even for curriculum research if appropriate Institutional Review Board approval is obtained.
Examinations
Online evaluation systems allow examination scores, "mock oral" examination feedback, and procedure training feedback to be immediately posted for review by learners. CMS systems allow the creation of secure online examinations, replacing paper examinations, and final course grade tabulation (3). Objective structured clinical examinations (OSCEs), employing standardized patients who grade and/or provide feedback to students, are used to assess student clinical skills and prepare them for the USMLE (United States Medical Licensing Examination) Step 2 Clinical Skills examination. Web-based evaluation systems facilitate OSCE scoring by allowing the standardized patient and/or the preceptor to input data for immediate review. Each OSCE case has a question checklist entered into the host program. Questions may be true/false to evaluate knowledge, or Likert scale to determine interviewing skills. Students enter the exam room and interview the standardized patient for a predetermined, fixed time. After the interaction, the student leaves the exam room and the standardized patient immediately completes the online checklist. If a large proportion of students have not asked a particular question on the checklist, the curriculum is reviewed to verify whether the missing skill is being taught; the OSCE video record may be reviewed to make certain the standardized patient was conveying the information correctly; and a review is held to determine why the students had difficulty seeing the importance of the question. Interview technique may be evaluated using Likert scale questions. Some institutions also utilize post-encounter examinations in which students answer questions relevant to the case just seen. The post-encounter response data are combined with the other evaluations of the learner to determine the final grade. Review of the OSCE video may be combined with the online data for added feedback to the learner. OSCEs are beginning to be used in the assessment of resident evaluation skills as well (5).
Another computer-assisted evaluation tool, the live audience response system, is not necessarily an online assessment tool but may be used as part of an online curriculum management system and is becoming increasingly common in medical education. These systems feature handheld devices that allow class participants to respond in real time to questions posed during a didactic presentation. Responses are instantly tallied and displayed for the group to see, usually via a software presentation tool, such as Microsoft PowerPoint. In addition to their use in assessing learner knowledge, audience response systems increase participant attention, allow presentations to be tailored to audience need, allow measurement of presentation efficacy by comparing pooled pre- and post-test data, and can be used for creative approaches, such as team-based learning games. Some systems are designed for ease of interface with CMS applications, allowing seamless integration of quizzes or examinations into the online curriculum environment. Audience response systems are listed in Appendix 2.
Portfolios
Training portfolio systems are emerging primary tools for the assessment of competency. Training portfolios may contain many forms of data, such as case write-ups, literature assessments, case/procedure logs, examples of clinical documentation, psychotherapy session transcripts, practice-based learning demonstrations, or other clinical or academic artifacts that foster reflective learning to encourage growth over time. Electronic portfolios, being developed by many of the online providers listed in this article, may include specific data mined from online evaluation and case log systems, or any digitized uploaded content (6).
Case/Procedure Logs
Several Web-based evaluation systems or CMS systems include procedure logs that may be used to track trainee exposure to required clinical experiences. These systems may include tracking by diagnosis, demographic variables, responsible supervisor, training site, or treatment type, and may be used to fulfill LCME and ACGME-mandated clinical experience tracking. PDA input options are often available. Dr. Benjamin worked with an online vendor to develop a customized psychiatry case log system that allows tracking and reporting by the above variables while allowing individual trainees to compare their case experience to the mean of their PGY level, seeking out specific experiences on their own if they note a significant variance from the mean. Procedure logs may also be used to demonstrate required skills for specific rotations. Experiential logs facilitate reflective goal-setting during periodic training reviews. The ACGME currently provides procedure tracking to a number of different specialties but has not yet offered this service to psychiatry programs, in part because the requirements for psychiatry training have thus far not defined minimum experience required in terms of case volume. ACGME implementation of a national psychiatry clinical log would allow the field to begin working toward data-driven definitions of experiential competency criteria.
|
Areas in Need of Further Development
|
Much progress has been made in the area of online assessments. Ten years ago, most medical schools and residencies still relied on paper evaluation forms. Administrative assistants had to reformat and collate the data by hand. Today, many institutions are making the transition to online systems, whether developed in-house or by an outside vendor, and reports that analyze different variables can be automatically generated. Many medical schools and residencies have purchased online assessment tools for all departments and programs to use, promoting standardization within an institution as well as saving costs. There are intrinsic risks with purchasing third-party products, such as support for a product being discontinued, or data portability issues in the event of a company merging or going out of business. However, as long as the purchaser plans for data backup and retention, the benefits typically outweigh the risks. With increasingly sophisticated software and wider integration of diverse data elements, more advanced features can be created, allowing clerkship or program directors to view assessment data for many purposes and in different views. National standards for data exchange, collaborative efforts, and open source sharing would further advance this field. The following is a list of features that will hopefully be more common within the next 5 years.
- The availability of Web-based assessment systems allows rapid analysis of data that could automatically generate specific interventions if performance thresholds are crossed. Examples of interventions and remediation strategies keyed to online performance thresholds should be developed.
- The provision of meaningful performance feedback during scheduled periodic review sessions has become more and more complex with the proliferation of competency assessment tools. Macros that automatically mine and seek trends from multiple types of data would facilitate the job of providing substantive feedback.
- Attendance documentation systems should be able to generate reports of missed topics to determine whether make-up work is indicated rather than simply calculating the percentage of seminars attended for compliance with ACGME attendance requirements.
- Assessment systems should be capable of automated transcript generation summarizing both experiences completed and competencies attained in a fashion that would allow use in professional credentialing. Ultimately, the determination of an acceptable national standard could one day replace the specialty certification examinations.
- Electronic portfolio systems should eventually standardize content types while allowing maximal flexibility of specific content if portfolios are to be an enduring measure of competency documentation beyond the training years. Methods of documenting progress and translating portfolio contents into standards of competency that would allow comparison of trainees graduating from different programs would also be useful.
- Data from case/performance logs should be compared nationally to develop minimal standards for experience-based competency assessment. If the ACGME were to implement a log system for psychiatry, even an optional one as has been done in other specialties, this data gathering would be facilitated. The tying of case log data to clinical or educational outcome measures would improve the validity of case log systems in determining competence. Live feedback with comparison to benchmarks or means must be available to trainees on a real-time basis for log systems to be useful in guiding training.
- The development by a national organization, such as the ACGME, of flexible, open-access, online evaluation, case-log, and portfolio systems would not only help move the psychiatry training community toward interprogram competency assessment validity, but would create documentation that would be easily portable in the event of interprogram transfers.
- Legal issues must be resolved in order to move forward. Data necessary to be maintained by training institutions for future competency documentation will need to be determined. If competency documentation requires the accumulation of discoverable data about remediable performance during training, the costs may be too high. Questions such as who owns the data, who may access it, how long it must be kept, and in what format will need to be addressed. Safeguards will be needed for trainee and faculty confidentiality. Institutions will need to adopt policies governing transfer of data as storage formats and software become obsolete.
- Assessment and case log data as well as outcome measures will need to be mined automatically from the electronic medical record (EMR) with the movement of health care delivery to EMR systems (7).
|
Getting Started and Being Successful (5 Tips)
|
- Carefully assess your institution’s readiness for using online assessment tools, both from the learner and faculty perspective, then move forward. Waiting for the perfect system to be developed will deprive your program of the competency to adapt quickly to improved online systems as they emerge.
- Utilize in-house expertise in system selection and/or design, being mindful of the institution’s strategic plan. Look for commonalities of purpose between undergraduate and graduate medical education programs to maximize purchasing power and support options.
- Network with others outside your institution who have experience with online assessment tools for their suggestions. Organization listservs can be an excellent source of advice.
- There are major advantages to selecting a single institution-wide program for GME online assessment. Attempt to develop synergistic systems when possible (e.g., audience response systems that interact with the CMS).
- Do not select a vendor on cost alone. Instead, consider flexibility, responsiveness, record of innovation, knowledge of the training environment, and quality of technical support available. If developing a system in-house, avoid dependence on a single individual for development and/or support.
|
Conclusions
|
The explosion of online data capacity has opened up new horizons in competency assessment, but has also made the job of training much more complex. Utilizing the amount of data now available takes time. Funding for the administrative time needed to derive meaningful information from available data and the faculty time needed to interpret it and intervene will need to be identified. The demand for better competency assessment methods is occurring at a time of maximal stress on our ability to deliver basic health care. The transition to data-based competency determination can only occur as more efficient and affordable methods of automating tasks are developed. On the other hand we have tools that should allow us to continually improve the training experience by providing feedback in a more timely fashion than ever before possible.
No single vendor provides a solution that addresses all of the functionality described in this article with equal flexibility and efficiency. Vendors continually improve their software and may differ in quality of technical support and responsiveness to individual program needs. Purchasers of this technology may wish to utilize the information presented as part of their selection process.
|
REFERENCES
|
- Accreditation Council on Graduation Medical Education and American Board of Medical Specialties, Toolbox Of Assessment Methods, version 1.1, September 2000
- Computer Resources in Medical Education (Survey Results: Online Evaluation Systems). Available at http://missinglink.ucsf.edu/crime/projects/survey_online_eval.htm. Accessed on February 20, 2001
- Chan C, Robbins L: E-Learning Systems: Promises and Pitfalls. Acad Psychiatry 2006; 30:491–497[Abstract/Free Full Text]
- Available at http://www.acgme.org/acWebsite/resEvalSystem/reval_about.asp. Accessed October 13, 2006
- Srinivasan M, Keenan C, Yager J: Visualizing the future: technology competency and development in clinical medicine, and implications for medical education. Acad Psychiatry 2006; 30:480–490[Abstract/Free Full Text]
- Wilkes MS, Howell L: Technology as an instrument to improve quality, accountability and reflection in academic medicine. Acad Psychiatry 2006; 30:456–464[Abstract/Free Full Text]
- Keenan C, Nguyen H, Srivinisan M: Electronic medical records and their impact on resident and medical student education. Acad Psychiatry 2006; 30:522–527[Abstract/Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
K G Hoffman, M D Griggs, C A Kerber, M Wakefield, E Garrett, C Kersten, M C Hosokawa, and L A Headrick
An educational improvement project to track patient encounters: toward a more complete understanding of third-year medical students' experiences
Qual. Saf. Health Care,
August 1, 2009;
18(4):
278 - 282.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
S. L. Dubovsky, M. H. Gendel, A. N. Dubovsky, R. Levin, J. Rosse, and R. House
Can Admissions Interviews Predict Performance in Residency?
Acad Psychiatry,
November 1, 2008;
32(6):
498 - 503.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
S. Martino and T. Odle
New Instructional Technology
Radiol. Technol.,
September 1, 2008;
80(1):
67 - 74.
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2006
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
