The Accreditation Council on Graduate Medical Education- and Liaison Committee on Medical Education-mandated transition from older written methods for faculty assessment of trainee skills to newer competency-based assessments has required the development or adaptation of assessment tools that are relatively new to the medical profession. The core competencies for residency training were introduced along with a menu of various competency assessment methods known as the “competency assessment toolbox” (Appendix 1) (1), only two of which—global ratings and written exams—were previously used by most training programs. The Accreditation Council on Graduate Medical Education (ACGME) itself is beginning a gradual transition to online reporting by training programs. Training directors have been required to specify the assessment tools being used for each core competency in the ACGME’s online Accreditation Data System (ADS) since 2005. With the explosion of new assessment methods, training directors have been faced with a new dilemma: how to synthesize the numerous types of data being collected into a cohesive method of competency determination that would also allow for future verification of competency.
The maturation of online solutions coincident with this transition has provided medical faculty with an expanded armamentarium of possible tools. What began with a few medical schools and residency programs developing in-house computer-assisted evaluation strategies quickly blossomed into a plethora of vendors of online assessment systems competing in the medical training marketplace. Although the field of medical competency assessment is still in its infancy, these systems have been rapidly adopted.
Online assessment systems were initially designed to automate the process of collecting standard written evaluation forms and provide an efficient means of accomplishing various administrative tasks inherent in running training programs. In addition to the task of learner assessment, these systems provide a mechanism for addressing other training needs, including quality assessment of faculty, curriculum, teaching environment, and the provision of feedback to trainees, faculty, and administration.
Although many programs simply converted their paper evaluation systems to online versions, technology can be harnessed to assess competency in ways that are just being realized. At the medical school level, online assessment systems may be implemented independently or as part of global online education solutions, providing resources for curriculum management across all 4 years. Implementation of a single online assessment system, however customizable for each program, provides an institution the ability to compare performance across programs, spot quality issues in a manner that allows early intervention, and prepare system-wide reports for various supervising bodies.
This article will explore some of the types of computer-assisted and online assessment tools currently available, and suggest areas in need of further development.
A number of residency programs and medical schools utilize evaluation and curriculum management systems (CMS) developed in-house. Some of these efforts have been compared by the Computer Resources in Medical Education group (2). Many schools have implemented any of a number of available Web portal systems to facilitate campus communication, administration, and access to CMS. Appendix 2 lists several commercial providers of online evaluation systems for students and residents. CMS systems for medical schools (Appendix 2) (3) include examination and assessment modules as well as mechanisms for course material development and distribution, and faculty-learner and interlearner communication. Both types of systems are moving into electronic portfolio development. In addition, the ACGME has developed a free Web-based evaluation system for urology residencies that provides a number of tools for online documentation of competency (4). This optional program indicates ACGME interest in exploring the provision of online assessment tools and may foreshadow future access to Web-based ACGME evaluation instruments for other specialties.
Administrative components included in different systems are listed in Appendix 3. Implementation may vary widely among systems, with no single solution appropriate for all programs.
Concerns shared by all system users include data security, permanence, and portability. Evaluative data must remain confidential and protected. Programs typically allow administrators to determine who should be permitted to view evaluations and whether the viewer will be able to see individual versus aggregate data. A flexible permission structure should allow coordinators of various aspects of the educational program to view evaluative data about their area. Data entry should take place over a password-protected secure interface (e.g., using Secure Sockets Layer technology) and should be maintained on secure servers with adequate backup, redundancy, and loss protection. Given the inability of providers to guarantee their own future, it is critical for programs to receive backup copies of data in a common, easily read format (e.g., comma-delimited values). Programs must then initiate their own redundant systems to insure that data backups are not lost and are filed in a manner that facilitates later access (e.g., placing a CD with all evaluative data in the trainees' file on graduation). The ACGME requires that programs using online assessment systems keep a printed copy of the final summative training evaluation in each trainee’s file. Some programs derive from the accumulated electronic data a final report documenting competencies attained. Paper training files are typically thinned to their essentials at a predetermined interval following graduation. Although there can be little doubt that electronic training files will become the standard, a standard for maintenance of these files is yet to emerge.
There are major advantages for institutions to agree on a single online assessment system for the graduate medical education (GME) community. A single system required of all GME programs allows an institution to verify compliance with ACGME standards, conduct system-wide quality surveys, and system-wide work-hour monitoring, and develop periodic assessment initiatives across programs. An institution-wide program saves money and can provide a layer of in-house support to users. It also allows for a collaborative environment in which programs learn from one another.
Web-based evaluation systems have enabled training programs to derive more useful data from global evaluations than did paper forms. Advance scheduling, automated reminders, and compliance tracking improves the compliance rate by both faculty and trainees while preserving administrative staff time for other tasks. The assurance of anonymity encourages trainee feedback about the curriculum, faculty, or training environment. Computerized evaluation forms are easily modified if necessary. Questions from different evaluation forms may be grouped together for analysis. Examples include pulling together all global assessments related to a given core competency or type of training as a single score for a summative report. Web-based systems may allow keyword designations for any evaluation question, didactic session, rotation, experience, or supervision type, allowing programs to both derive reports documenting numerous approaches to the assessment of a given competency or required experience and utilize a given event toward more than one competency assessment. Individual and group statistics are typically calculated comparing an individual’s performance with that of a peer group. Automated notification systems for below threshold performance provide training directors, clerkship directors, advisers, or individual site directors timely notification of situations requiring intervention or remediation. Reporting tools allow trainees and faculty to track their own performance over time. Individual competency-based evaluation summaries may be prepared for faculty meetings and periodic reviews. Summaries by training year or for an entire program allow directors to spot trends that facilitate early course correction. 360-degree evaluations are easily accomplished by providing limited system access to all mandated evaluators.
In addition to uses for trainee and curriculum evaluation, Web-based systems lend themselves to the establishment of a culture of feedback within a department. Annual feedback summaries to faculty may be one factor considered in faculty promotion decisions or faculty performance evaluations. These same systems may be adapted to provide hospitals with competency assessment systems for their physician staff.
Some online systems are sufficiently flexible to allow the collection of on-demand survey data within a program. Surveys can be used to look at specific aspects of a given training experience not anticipated by the routine curriculum evaluation process, for further examination of detected trends, or even for curriculum research if appropriate Institutional Review Board approval is obtained.
Online evaluation systems allow examination scores, “mock oral” examination feedback, and procedure training feedback to be immediately posted for review by learners. CMS systems allow the creation of secure online examinations, replacing paper examinations, and final course grade tabulation (3). Objective structured clinical examinations (OSCEs), employing standardized patients who grade and/or provide feedback to students, are used to assess student clinical skills and prepare them for the USMLE (United States Medical Licensing Examination) Step 2 Clinical Skills examination. Web-based evaluation systems facilitate OSCE scoring by allowing the standardized patient and/or the preceptor to input data for immediate review. Each OSCE case has a question checklist entered into the host program. Questions may be true/false to evaluate knowledge, or Likert scale to determine interviewing skills. Students enter the exam room and interview the standardized patient for a predetermined, fixed time. After the interaction, the student leaves the exam room and the standardized patient immediately completes the online checklist. If a large proportion of students have not asked a particular question on the checklist, the curriculum is reviewed to verify whether the missing skill is being taught; the OSCE video record may be reviewed to make certain the standardized patient was conveying the information correctly; and a review is held to determine why the students had difficulty seeing the importance of the question. Interview technique may be evaluated using Likert scale questions. Some institutions also utilize post-encounter examinations in which students answer questions relevant to the case just seen. The post-encounter response data are combined with the other evaluations of the learner to determine the final grade. Review of the OSCE video may be combined with the online data for added feedback to the learner. OSCEs are beginning to be used in the assessment of resident evaluation skills as well (5).
Another computer-assisted evaluation tool, the live audience response system, is not necessarily an online assessment tool but may be used as part of an online curriculum management system and is becoming increasingly common in medical education. These systems feature handheld devices that allow class participants to respond in real time to questions posed during a didactic presentation. Responses are instantly tallied and displayed for the group to see, usually via a software presentation tool, such as Microsoft PowerPoint. In addition to their use in assessing learner knowledge, audience response systems increase participant attention, allow presentations to be tailored to audience need, allow measurement of presentation efficacy by comparing pooled pre- and post-test data, and can be used for creative approaches, such as team-based learning games. Some systems are designed for ease of interface with CMS applications, allowing seamless integration of quizzes or examinations into the online curriculum environment. Audience response systems are listed in Appendix 2.
Training portfolio systems are emerging primary tools for the assessment of competency. Training portfolios may contain many forms of data, such as case write-ups, literature assessments, case/procedure logs, examples of clinical documentation, psychotherapy session transcripts, practice-based learning demonstrations, or other clinical or academic artifacts that foster reflective learning to encourage growth over time. Electronic portfolios, being developed by many of the online providers listed in this article, may include specific data mined from online evaluation and case log systems, or any digitized uploaded content (6).
Several Web-based evaluation systems or CMS systems include procedure logs that may be used to track trainee exposure to required clinical experiences. These systems may include tracking by diagnosis, demographic variables, responsible supervisor, training site, or treatment type, and may be used to fulfill LCME and ACGME-mandated clinical experience tracking. PDA input options are often available. Dr. Benjamin worked with an online vendor to develop a customized psychiatry case log system that allows tracking and reporting by the above variables while allowing individual trainees to compare their case experience to the mean of their PGY level, seeking out specific experiences on their own if they note a significant variance from the mean. Procedure logs may also be used to demonstrate required skills for specific rotations. Experiential logs facilitate reflective goal-setting during periodic training reviews. The ACGME currently provides procedure tracking to a number of different specialties but has not yet offered this service to psychiatry programs, in part because the requirements for psychiatry training have thus far not defined minimum experience required in terms of case volume. ACGME implementation of a national psychiatry clinical log would allow the field to begin working toward data-driven definitions of experiential competency criteria.
Much progress has been made in the area of online assessments. Ten years ago, most medical schools and residencies still relied on paper evaluation forms. Administrative assistants had to reformat and collate the data by hand. Today, many institutions are making the transition to online systems, whether developed in-house or by an outside vendor, and reports that analyze different variables can be automatically generated. Many medical schools and residencies have purchased online assessment tools for all departments and programs to use, promoting standardization within an institution as well as saving costs. There are intrinsic risks with purchasing third-party products, such as support for a product being discontinued, or data portability issues in the event of a company merging or going out of business. However, as long as the purchaser plans for data backup and retention, the benefits typically outweigh the risks. With increasingly sophisticated software and wider integration of diverse data elements, more advanced features can be created, allowing clerkship or program directors to view assessment data for many purposes and in different views. National standards for data exchange, collaborative efforts, and open source sharing would further advance this field. The following is a list of features that will hopefully be more common within the next 5 years.
The availability of Web-based assessment systems allows rapid analysis of data that could automatically generate specific interventions if performance thresholds are crossed. Examples of interventions and remediation strategies keyed to online performance thresholds should be developed.
The provision of meaningful performance feedback during scheduled periodic review sessions has become more and more complex with the proliferation of competency assessment tools. Macros that automatically mine and seek trends from multiple types of data would facilitate the job of providing substantive feedback.
Attendance documentation systems should be able to generate reports of missed topics to determine whether make-up work is indicated rather than simply calculating the percentage of seminars attended for compliance with ACGME attendance requirements.
Assessment systems should be capable of automated transcript generation summarizing both experiences completed and competencies attained in a fashion that would allow use in professional credentialing. Ultimately, the determination of an acceptable national standard could one day replace the specialty certification examinations.
Electronic portfolio systems should eventually standardize content types while allowing maximal flexibility of specific content if portfolios are to be an enduring measure of competency documentation beyond the training years. Methods of documenting progress and translating portfolio contents into standards of competency that would allow comparison of trainees graduating from different programs would also be useful.
Data from case/performance logs should be compared nationally to develop minimal standards for experience-based competency assessment. If the ACGME were to implement a log system for psychiatry, even an optional one as has been done in other specialties, this data gathering would be facilitated. The tying of case log data to clinical or educational outcome measures would improve the validity of case log systems in determining competence. Live feedback with comparison to benchmarks or means must be available to trainees on a real-time basis for log systems to be useful in guiding training.
The development by a national organization, such as the ACGME, of flexible, open-access, online evaluation, case-log, and portfolio systems would not only help move the psychiatry training community toward interprogram competency assessment validity, but would create documentation that would be easily portable in the event of interprogram transfers.
Legal issues must be resolved in order to move forward. Data necessary to be maintained by training institutions for future competency documentation will need to be determined. If competency documentation requires the accumulation of discoverable data about remediable performance during training, the costs may be too high. Questions such as who owns the data, who may access it, how long it must be kept, and in what format will need to be addressed. Safeguards will be needed for trainee and faculty confidentiality. Institutions will need to adopt policies governing transfer of data as storage formats and software become obsolete.
Assessment and case log data as well as outcome measures will need to be mined automatically from the electronic medical record (EMR) with the movement of health care delivery to EMR systems (7).
The explosion of online data capacity has opened up new horizons in competency assessment, but has also made the job of training much more complex. Utilizing the amount of data now available takes time. Funding for the administrative time needed to derive meaningful information from available data and the faculty time needed to interpret it and intervene will need to be identified. The demand for better competency assessment methods is occurring at a time of maximal stress on our ability to deliver basic health care. The transition to data-based competency determination can only occur as more efficient and affordable methods of automating tasks are developed. On the other hand we have tools that should allow us to continually improve the training experience by providing feedback in a more timely fashion than ever before possible.
No single vendor provides a solution that addresses all of the functionality described in this article with equal flexibility and efficiency. Vendors continually improve their software and may differ in quality of technical support and responsiveness to individual program needs. Purchasers of this technology may wish to utilize the information presented as part of their selection process.