Advanced Search

Original Articles | September 01, 2004

Use of a Computerized Evaluation System in a Psychiatry Clerkship

Aurora J. Bennett, M.D.; Lesley M. Arnold, M.D.

Academic Psychiatry 2004;28:197-203. 10.1176/appi.ap.28.3.197

View Article Information

Dr. Bennett is Psychiatry Clerkship Director, Department of Psychiatry, University of Cincinnati College of Medicine, Cincinnati, Ohio. Dr. Arnold is Director of Medical Student Education, Department of Psychiatry, University of Cincinnati College of Medicine, Cincinnati, Ohio. Address correspondence to Dr. Bennett, University Psychiatric Services, 222 Piedmont Ave., Medical Arts Building, Suite 8500, Cincinnati, OH 45219; aurora.bennett@psychiatry.uc.edu (E-mail).

text A A A

Abstract

Objective: Paper-based evaluations of medical students on clerkships are inefficient and time consuming to compile and analyze. To improve the evaluation process, the authors developed a web-based program. Methods: A secure web-based system was designed to collect evaluations of students and their feedback about the clerkship. Acceptance, compliance, and impact of the web-based system on the clerkship were assessed. Results: The web-based program was universally accepted by medical students who have been 100% compliant with prompt completion of their evaluations of the rotation. Most of the preceptors have utilized the program, resulting in a more timely receipt of their evaluations of students. The web-based program computes numerical scores, compiles comments, and formats reports. The clerkship director has unlimited access to all of the data, which allows for efficient and timely communication with preceptors, medical students, and the dean's office. Conclusions: During the clerkship, the web-based program for evaluation and feedback has improved the efficiency of the evaluation process and the timely communication with students and preceptors.

Figures in this Article

Psychiatry clerkship directors are responsible for providing a full-time clinical experience for medical students that typically includes multiple sites and faculty preceptors. The 6-week psychiatry clerkship at the University of Cincinnati uses approximately 30 inpatient and outpatient sites and 60 faculty members within the University and throughout the community to serve the training needs of approximately 160 students each year. Although psychiatric residents participate in the teaching of the medical students, they do not perform a separate evaluation of the students. Rather, residents provide input about a student's performance to the faculty preceptor who is responsible for completing the evaluation. Because the Association of American Medical Colleges (AAMC) mandates that clerkships develop comparable educational experiences across sites (1), clerkship directors must perform an ongoing assessment of the quality, quantity, and diversity of clinical experiences offered by each of the teaching sites. Therefore, timely access to students' numerical and narrative evaluations about their clerkship experiences is crucial to maintaining an optimal educational experience. Furthermore, ready access to preceptors' evaluations of students facilitates the communication of grades to the students and to the dean's office.

The psychiatry clerkship at the University of Cincinnati has relied on paper-based evaluations with a numerical rating scale and the option of writing additional comments. There are several limitations inherent in paper evaluations, including the time required to track, collect, and collate large amounts of data and delays in compiling data for student grades and feedback to preceptors.

Nationally, the majority of clerkships were continuing to utilize paper evaluations with a minority of programs utilizing commercially available products (http://www.webct.com) (2) or beginning to design customized programs for their specific needs (3—5).

To address the limitations of paper-based evaluations, the department of psychiatry at the University of Cincinnati developed a customized web-based system of evaluation called the Psychiatric Core Clerkship Evaluation System (PCCES), a system designed solely for the evaluations of the clerkship sites and students (2001 Copyright). The psychiatry clerkship is the first clerkship to develop a web-based evaluation system at the University of Cincinnati College of Medicine. Following implementation of the PCCES, we assessed system acceptance, compliance, and impact on the clerkship.

METHOD

The PCCES was designed in collaboration with the department of psychiatry's Information Technology (IT) staff and a local technology company, Global Techmasters, Inc. (global_techmasters@yahoo.com).

Technology and Programs

The PCCES is a collaborative system that is composed of both a web-based user interface, a locally installed administrative backdoor, and a database engine on the back, in which data is stored. The system is connectively housed within MS Windows IIS v.5 and MS SQL 2000 servers that are located logically in a restricted Microsoft 2000 domain. Both servers are running on secure, 128bit encrypted, 100mb network access, MS Windows 2000 member servers. Both physical servers are designed with mirrored system drives and backup via a Sony AIT drive nightly on a 5-day "complete" backup schedule. Servers are located in a secured, electrically filtered/power fault resistant and environmentally controlled room. Backup media is stored in a fire/impact resistant safe. Users require access to Internet Explorer 4.0 or higher to access the Internet for data entry and transmission.

Design of the Evaluation Forms

Preceptors and students access secured evaluation forms via the Internet. The web-based forms are identical to the previously used paper evaluations. All of the questions and narrative spaces must be completed for the system to allow the evaluator to submit the form to the central database. Furthermore, the program provides the clerkship director and clerkship coordinator access to the names of students and preceptors who have not completed evaluations.

Preceptors evaluate students in nine areas that are under the general categories of fund of knowledge, clinical skills, professionalism, and self-education (Figure 1). Students evaluate several areas of the rotation, including numbers of hours worked, level of active participation in clinical care, quality of interactions with preceptors, and knowledge gained (Figure 2). There is a separate evaluation form for students to evaluate the on-call experiences and the clinical case conferences.

Access of the Evaluation Forms

Preceptors and students can complete the evaluations from any computer with Internet access via the psychiatry department's website or by saving the URL to access the system directly. The front page asks users whether they are preceptors or students and then directs them to a page requiring their confidential username and password. The clerkship coordinator assigns these codes and is the only person with access to these identifiers. Each evaluation form is specific to the user. The user is presented with a drop-down menu listing only those sites, preceptors, or students who have been assigned to the user during a specific rotation. After users complete the evaluations, they click the "Submit" button, and the data is sent to the department's server for storage, computation, and collation. The preceptors and students no longer have access to the data after it is submitted.

Access of the PCCES Data

Access to the data submitted by the preceptors and students is limited to the administrative coordinator, the clerkship director, and the director of the psychiatry department's IT division. In contrast to the blank evaluation forms, the completed PCCES data can only be accessed from campus computers.

Student evaluations submitted by the preceptors are available in two formats: individual evaluation forms completed by the preceptors and a report that lists all of the students for a particular rotation, along with each of their clinical scores, oral examination scores, subject examination scores, and final computed score (Table 1).

Feedback from students is available in several formats. Numerical ratings are sorted by sites and preceptors (Table 2), which allows for the assessment of individual preceptors working at the same site. These reports can be generated for previous rotations, providing an important historical perspective on sites and preceptors. Cumulative averages of numerical ratings are also available to provide a general overview of scores over a longer period of time (e.g., average ratings for inpatient units during the past 2 years). Separate reports emailed to preceptors on a semiannual basis provide data about the three main clerkship rotations that include sites on an inpatient unit, an outpatient center, and substance abuse treatment program (Table 3). For example, inpatient preceptors receive their individual scores; the average score for inpatient, outpatient, and substance abuse rotations; and the average scores for the other inpatient sites to be used for comparison. No other faculty names appear on their reports to protect anonymity. This report also provides them with the anonymous student comments submitted during a delineated time period, usually the previous 6 months.

Cost of the PCCES

The PCCES was implemented in 2001 for the psychiatry clerkship at a cost of $5,300.00. We have spent an additional $1,150.00 during the past 2 years implementing changes in the database to reflect changes that have occurred in the clerkship (e.g., addition of an oral examination). Smaller changes such as names of preceptors or sites are easily conducted by the administrative coordinator, but changes to the format of the reports or the formula for calculating grades have required consultation with Global Techmasters.

RESULTS

PCCES Acceptance and Compliance

Student feedback about the use of the PCCES has been very positive. The 58 students who rotated through psychiatry during the previous four rotations responded to an anonymous survey asking whether they preferred paper or web-based evaluations. Seventy six percent of students preferred the web-based program, while 21% thought the two formats were comparable, and 3% favored paper evaluations due to access difficulties. The majority of the students have found that the PCCES is readily available and easy to use. Before implementation of the PCCES, 80% of students were compliant with timely return of evaluations. With the PCCES, students who have not completed the evaluations are easily identified and can be reminded to complete the evaluations prior to leaving the clerkship. With this system, students have been 100% compliant with prompt completion of evaluations.

Forty-five (75%) of the 60 preceptors have been compliant with the PCCES. Of those who consistently utilize the PCCES, 77% rate it as superior to a paper-based format due to the ease of use. Six preceptors (10%) do not have readily available access to the Internet and nine (15%) have had fire-wall problems or prefer paper evaluations. The most frequent complaints from those preferring paper evaluations have been about technical difficulties that limit access to the program due to fire-wall problems. These faculty members fax their paper evaluations. The data is then entered into the PCCES by the clerkship coordinator. Prior to instituting the web-based program, preceptors returned paper-based evaluations in a timely manner only 50% of the time, with several evaluations received up to 2 weeks past the acceptable deadline set by the Dean's office. Preceptors using the PCCES return evaluations on time 90% of the time. The preceptors have unanimously felt positive about the detailed reports, provided to them semiannually, regarding the students' evaluations of their teaching efforts.

Impact of PCCES on the Clerkship

With the PCCES, the psychiatry clerkship more frequently submits grades within the mandated 6-week period following the end of a clerkship cycle. The PCCES has also reduced the administrative coordinator's time from approximately 8 hours per 6-week rotation for printing, mailing, faxing, collecting, and collating paper evaluations to about 1 hour per rotation for sending email reminders to complete online evaluations and enter data from faxed evaluations. Furthermore, the web-based report calculates the final grade without the need for the coordinator to separately enter grades into a spreadsheet, which also has reduced clerical time. This greater efficiency has made it feasible to allot more of the coordinator's time toward other desirable activities for the clerkship.

The PCCES provides the preceptors with semiannual feedback about their teaching methods through a report with numerical and narrative information. This information allows the faculty to make timely interventions to improve the learning experience at their sites. The faculty members also find this report to be useful for reappointment and promotion purposes.

The PCCES has improved the clerkship's ability to quickly respond to student concerns. For example, the clerkship director has immediate access to ongoing information about a student's performance. If a student expresses concern about grades, the clerkship director can quickly access the evaluations, identify areas of concern, and help the student focus on the areas that need improvement before the end of the clerkship. The PCCES also allows the clerkship director to respond quickly to student feedback about sites, which has enhanced the learning experience for students, as illustrated by the following case example:

Following a 6-week rotation, a student gave negative feedback about excessive workload at one inpatient site, which detracted from the learning experience for the student. The clerkship director was able to immediately access both historical and comparative data on this inpatient site and discovered that the number of hours and number of patients assigned to the students had increased over the previous 6 months and had become substantially different than other inpatient sites. The clerkship director contacted the inpatient attending, who agreed to adjust the workload of the students to be in line with other inpatient sites. The clerkship director was able to make this intervention before the next 6-week rotation. The learning experience on the inpatient site was subsequently rated highly by the students.

DISCUSSION

The PCCES provides efficient and readily accessible data regarding the most important aspects of our clerkship curriculum. Students have readily accepted the PCCES and have been 100% compliant with timely completion of evaluations. The majority of the faculty preceptors have also been compliant with the PCCES.

Technological problems related to college-wide server problems or changes in fire-wall arrangements have limited access to some off-site faculty and to others within the medical complex who are in buildings other than the centralized location of the department. This has caused mutual frustration for these faculty preceptors and the clerkship administration. Many have been willing to resolve these problems in consultation with our IT staff, despite it having been a time-consuming endeavor for some. A minority of faculty have reverted to paper evaluations whenever technical problems arise.

Gaining an historical perspective of the problems and strengths of clinical sites was previously costly and inefficient and therefore not readily available. Numerical ratings and narrative comments were not utilized to their fullest potential but now are immediately accessible to the clerkship director for specific preceptors, or sites are in chronological order, allowing for identification of important trends in the quality of the clinical experiences. The quantity of narrative comments has not increased significantly but their utilization has. Necessary changes now occur in a timely manner due to early identification of problems such as excessive or insufficient workload, lack of a diverse clinical exposure, and less than optimal access to teaching from faculty and resident preceptors. Rapid response to student feedback has played a role in improving the quality of learning experiences. The PCCES has also provided a faster and more efficient evaluation of student performance. This allows the clerkship director to have immediate access to students' individual evaluations and examination scores to enhance discussions with students who raise concerns about their grades.

Future plans include updating the system to a completely web-based format that would not require utilization of the local departmental server, which would eliminate the fire-wall problems. The majority of changes that will continually be required as the clerkship evolves could be completed internally without additional costs. The clerkship director and the administrative coordinator could readily access the data from an off-site computer or PDA, allowing access when away from the campus. Students and faculty would have the similar flexibility of PDA access to the evaluation forms.

ACKNOWLEDGMENTS

The authors thank Mary Ann Schmidt, Clerkship Coordinator, for her contribution of data and for facilitating the utilization of the PCCES by faculty and students, and Gary Peck, IT Director for their department, for providing the technical description of the program and facilitating the design of the PCCES.

TABLE 1. Sample Student Scores

View Large | Download Slide(.ppt)
Add to My POL

TABLE 2. Sample Site and Preceptor Report

View Large | Download Slide(.ppt)
Add to My POL

TABLE 3. Sample Semiannual Inpatient Faculty Report

View Large | Download Slide(.ppt)
Add to My POL

FIGURE 1.: Medical Student Performance Evaluation Form

View Large | Download Slide(.ppt)
Add to My POL

FIGURE 2.: Clerkship Evaluation Form

References

Association of American Medical Colleges: AAMC Databook. Washington, DC, AAMC, 2002

Kochar MS, Simpson DE, Brown D: Graduate medical education at the Medical College of Wisconsin: new initiatives to respond to the changing residency training environment. WMJ 2003; 102:38—42

Rosenberg ME, Watson K, Paul J, Miller W, Harris I, Valdivia TD: Development and implementation of a web-based evaluation system for an internal medicine residency program. Acad Med 2001; 76:92—95[PubMed][CrossRef]

Cunha JD, Larson CE, Maddaus MA, Landis GH: An internet-based evaluation system for a surgical residency program. J Am Coll Surg 2003; 196:905—910[PubMed][CrossRef]

Schell SR, Lind DS: An internet-based tool for evaluating third-year medical student performance. Am J Surg 2003; 185:211—215 [PubMed][CrossRef]

Download all saved figures as a .zip file.

View Large
Download Slide(.ppt)

Add to My POL

FIGURE 1.

Medical Student Performance Evaluation Form

View Large
Download Slide(.ppt)

Add to My POL

FIGURE 2.

Clerkship Evaluation Form

TABLE 1. Sample Student Scores

View Large | Download Slide(.ppt)
Add to My POL

TABLE 2. Sample Site and Preceptor Report

View Large | Download Slide(.ppt)
Add to My POL

TABLE 3. Sample Semiannual Inpatient Faculty Report

View Large | Download Slide(.ppt)
Add to My POL

Association of American Medical Colleges: AAMC Databook. Washington, DC, AAMC, 2002

Kochar MS, Simpson DE, Brown D: Graduate medical education at the Medical College of Wisconsin: new initiatives to respond to the changing residency training environment. WMJ 2003; 102:38—42

Cunha JD, Larson CE, Maddaus MA, Landis GH: An internet-based evaluation system for a surgical residency program. J Am Coll Surg 2003; 196:905—910[PubMed][CrossRef]

Schell SR, Lind DS: An internet-based tool for evaluating third-year medical student performance. Am J Surg 2003; 185:211—215 [PubMed][CrossRef]

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.

Submit a Comments

Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of JBJS editorial staff.

* = Required Field

Comment Author(s)*(if multiple authors, separate names by comma)

Example: John Doe

Affiliation & Institution*

Comment Title*

Comment*

Cancel

Print
PDF
E-mail

Recipient(s) will receive an email with a link (good for 72 hours) to 'Use of a Computerized Evaluation System in a Psychiatry Clerkship' and do not need to have Academic Psychiatry account to access the content.
Your Name:*
Example: John Doe

Email Address:* CC Me:
Enter your valid email address. Example: jdoe@example.com

Recipient's Email Address:*
Separate multiple email address with semi-colons (up to 5).

Subject:* 's Academic Psychiatry: 'Use of a Computerized Evaluation System in a Psychiatry Clerkship'
Subject for your email.

Message:
(Optional, message will truncate at 1000 characters)

Processing your request... Please Wait...

Copyright © in the material you requested is held by The American Psychiatric Association (unless otherwise noted). This email ability is provided as a courtesy, and by using it you agree that you are requesting the material solely for personal, non-commercial use, and that it is subject to The American Psychiatric Association's Terms of Use. The information provided in order to email this topic will not be used to send unsolicited email, nor will it be furnished to third parties. Please refer to The American Psychiatric Association's Privacy Policy for further information.

Copyright © American Psychiatric Association. All rights reserved.
Share
Get Citation

Aurora J. Bennett, Lesley M. Arnold; Use of a Computerized Evaluation System in a Psychiatry Clerkship. Academic Psychiatry. 2004 Sep;28(3):197-203.

Download citation file:

RIS (Zotero)

EndNote

BibTex

Medlars

ProCite

RefWorks

Reference Manager

Copyright © American Psychiatric Association. All rights reserved.
Reprints