In recent years the Liaison Committee on Medical Education (LCME) has required that all medical schools base their curricula on either the six competency domains described and required by the Accreditation Council for Graduate Medical Education (ACGME) or those determined by the individual medical school. The LCME also requires that schools determine specific learning objectives for each clerkship and demonstrate that students are achieving those targets (1). There has been discussion in the literature regarding the use and potential value of competency-based education in medical education (2, 3). Numerous medical schools have attempted to document what students see, do, learn, and experience while on their clerkships using web-based systems (4–6), computerized medical records systems (7), paper and computer combinations (8–10), and paper encounter records (11, 12).
There have been multiple efforts to assess competencies for evaluation purposes in medical schools (13–16) and in residency training programs (17–19), but few to document student experiences with competencies other than knowledge and clinical skills. Data have been reported for clerkships in internal medicine, family medicine, pediatrics (6, 8–12), between sites in a single clerkship (20), and for the entire clinical curriculum (4). In one comprehensive effort, students on a psychiatry clerkship documented encounters and were given symptom targets to meet (21). We report what three classes of medical students experienced during their psychiatry clerkships, in all six competency domains, and how this information helped modify and improve our clerkship.
Dartmouth Medical Encounter Documentation System (DMEDS)
In 2003, Dartmouth Medical School adopted the six competency domains recommended by the ACGME. Dartmouth had previously developed a card-based and a PDA-based system to enable students to document their learning experiences during outpatient primary care clerkships (10). More recently a web-based system was developed for students to document their experiences across all required clerkships in all six competency domains. Details of this system, the Dartmouth Medical Encounter Documentation System (DMEDS), are described elsewhere (22).
The DMEDS requires students to report types and numbers of patients seen by diagnosis or clinical condition. Students record specific clinical skills they are performing and their degree of independence in performing those skills. Finally, they record their activities in the “newer” four competencies. The DMEDS became the required documentation system for all students in clerkships in July 2004. We report data for the psychiatry clerkship using DMEDS for the academic years 2004–2007.
The third-year psychiatry clerkship lasts 7 weeks. For the major portion of their rotation, students are assigned to a service at Dartmouth-Hitchcock Medical Center; the Veterans Administration Medical Center (VAMC) in White River Junction, Vt.; or the New Hampshire State Hospital in Concord.
Dartmouth-Hitchcock Medical Center students are assigned to either the inpatient service for 6 weeks and the consultation service for 1 week or to the inpatient service for 2 weeks and the consultation service for 5 weeks. A few students are assigned to the consultation service for half the rotation and spend the other half at the Children’s Psychiatric Unit at New Hampshire State Hospital. Students at the VAMC spend 5 weeks on the inpatient service and 2 weeks on the consultation service. Full-time New Hampshire State Hospital students are assigned exclusively to an inpatient unit.
All students receive experience in an outpatient setting, two substance abuse treatment settings, a day at the Children’s Psychiatric Unit at New Hampshire State Hospital, and at least one ECT session. Didactic sessions occur one afternoon a week.
Each clerkship director sets criteria for which patients the students are to record. Initially, students were required to document only patients for whom they cared directly. More recently, students have been instructed to document patients if the student sees the patient intensively and participates in discussions regarding the patient’s care.
The goal is that each student documents encounters with at least 25 discrete patients. A primary diagnosis and other data are required. A secondary or tertiary diagnosis may be entered, if it represents an active problem for the patient.
The clerkship director selects specific disorders and clinical skills as “learning targets”—the number of patients with a disorder a student is expected to see to develop a basic level of competency in diagnosing and treating that condition. At the end of the year, if students are consistently seeing fewer than the targeted number for a specific disorder, the director either adjusts the learning target or modifies the clerkship experience, so that most students can meet the target. Similar targets are set for clinical and counseling skills. Students are to report their experiences with the four other competency domains when they experience unusual challenges or learning opportunities, but no specific learning targets are set.
We examined parameters that provided some sense of how well the clerkship was taught. We analyzed the students’ documentation of their learning experiences in each competency domain: knowledge, assessed by the diagnoses and conditions of patients that were documented; clinical skills, with an emphasis on types of patient counseling; advanced communications skills; professionalism; personal and continuous learning; and practicing in a complex health care system.
General Clerkship Parameters
For the reporting period, all 173 students on the psychiatry clerkship reported a total of 4,676 patient encounters (Table 1). The average number of patients seen was 27.2 per student. Instruction by faculty or residents occurred with 82% of encounters. Students reported receiving feedback during 70% of encounters. Patients declined contact with the student in only 0.5% of instances. Average patient encounters per student increased from 21.2 to 35.0 during the second year and then decreased to 25.4 during the third year (Table 1).
Diagnoses or Conditions Seen
Of all diagnoses recorded, 85% to 89% were psychiatric, with students documenting involvement with an average of 48.8 psychiatric disorders during their clerkship (Table 1).
Learning targets were initially set for 18 diagnoses for a total of 41 patients to be seen in required categories. By the third year, these learning targets were modified to a total of 33 patients to be seen in 17 diagnostic categories (Table 2). The students met learning targets for the diagnoses of anxiety disorder, bipolar affective disorder, depression, borderline personality disorder, posttraumatic stress disorder, psychosis, schizophrenia, and substance abuse (alcohol). They also listed other disorders without targeted goals, but none appeared frequently enough to become a target.
The ability of students to see patients with the most important diagnoses (those with specified learning targets) was analyzed by clerkship sites. Students assigned primarily to the inpatient wards consistently met more diagnoses than those on other services. Students assigned primarily to the consultation service at Dartmouth-Hitchcock Medical Center met only the target for depression in 2004–2006, but in 2006–2007 they met the same targets as students assigned to the inpatient services. Students assigned to the split consultation-liaison service-children’s unit met no learning targets for psychiatric disorders for any of the 3 years, although they came close.
Targets were set for counseling skills in a number of specific areas, with the expectation that students would spend time discussing these issues with patients during the rotation. The only category for which the targets were met was “family issues,” though there was an increase in numbers from the first to the second year.
Students reported their progress in developing competency through helpful learning situations in advanced communication skills, professionalism, personal and continuous learning/improvement, and practicing in a complex health care system. By examining the “Competency Reports” of each student, we were able to determine the types of interactions engaged in by each student. Activity in at least one of the four competency areas was reported by 60.4% of students, and being challenged in these domains was reported in 0.3% to 13.4% of all encounters. Adding depth and meaning to the statistics are student free-text reports concerning the detailed nature of these encounters (see examples in Table 3).
Advanced Communications Skills.
In the domain of advanced communications skills, students were asked to address how frequently they were confronted with challenges in five specific areas. In 12.6% of documented encounters (n=568), students reported being challenged by discussing “difficult or sensitive issues” with their patients. Overcoming “other communication barriers” (9.3% of encounters, n=427) and working to solicit “patient preferences” (7.0%, n=303) were noted with some frequency. Because of the ethnic makeup in northern New Hampshire, students rarely faced the challenges of the “need to use an interpreter” (only 0.3% of encounters, n=12) or “encountering cultural differences” that posed a challenge (0.7%, n=31) (Table 3).
Students were more likely to document challenges in the professionalism domain than in the other three. Most commonly, they were “challenged to assess their own strengths and weaknesses” (13.4% of encounters, n=533), “put aside their personal biases” while delivering patient care (12.5%, n=474), or “place patients’ interests first” (12.1%, n=453). Being challenged to “adhere to professional ethical standards” (11.7%, n=433) and “maintain confidentiality” (11.1%, n=402) were also commonly documented (Table 3).
Personal and Continuous Learning.
In the domain of personal and continuous learning, students documented activity in less than 7.5% of all encounters. “Measured or improved my own patient care” (7.1%, n=292) and “Read/assessed latest literature” (6.2%, n=277) were documented most often (Table 3).
Practicing in a Complex Health Care System.
Of the four choices in the area of practicing in a complex health care system, the activity most reported (12.1% of encounters, n=534) was “Worked directly with other services or teams.” The other three encounters were documented less frequently (3.2%–4.8%) (Table 3).
Initially it seemed that the average of 27 documented patients per student during a 7-week rotation was low. In fact, students were seeing four new patients a week. This does not vary significantly from other reports of student activity during psychiatry clerkships (13). Combined with the finding that students were meeting targets for the disorders most commonly seen in psychiatry, this suggests they were getting adequate exposure to both volume and types of patients one would expect for a third-year clerkship (Table 2).
Of interest is the difference in the average number of patients documented each year (21.2 in 2004–2005, 35.0 in 2005–2006, and 25.4 in 2006–2007). A possible explanation for the difference between the first 2 years is that as the clerkship director became more familiar with the Dartmouth Medical Encounter Documentation System (DMEDS) and was provided with the data from the first year, he became more motivated to ensure that students were recording their encounters. During subsequent years, students were required to bring copies of their DMEDS data to mid-clerkship and final meetings with the director. At least a portion of the difference likely occurred as a result of changing reporting criteria during the second year.
Of interest is that the numbers declined in the third year to close to the level of the first year. This is difficult to explain. The director continued to emphasize the importance of the DMEDS reporting and how the data might be used by students and faculty. Discussion with students revealed significant concern about the time required to enter data when they actually complied with what is asked in all areas. Possibly this “fatigue factor” resulted in students being less assiduous in recording their data. Some students reported seeing the DMEDS as “not user friendly.” One would wonder if a “student culture” attitude might have been developing—seeing the DMEDS as having more value for the faculty than for themselves.
For the majority of students, this clerkship is their only formal experience in clinical psychiatry. Therefore it is important that they are exposed to patients with as many different diagnoses as possible in order to recognize and treat, refer, and respond appropriately later in their careers. The DMEDS facilitates the ability to determine the numbers and variety of the diagnoses of patients seen on the psychiatry clerkship. The data show that the large majority of students are seeing the most common disorders: anxiety, major depression, and substance abuse (23, 24), and, arguably, those with which they should be most familiar. The Association of Directors of Medical Student Education in Psychiatry (ADMSEP) suggests that students on the psychiatry clerkship be involved in evaluation and management for at least one patient with the following diagnoses: major mood, anxiety, personality, substance abuse, cognitive, and somatoform disorders and psychosis (25). When delirium and dementia were combined as a “cognitive disorder,” the students fulfilled all ADMSEP requirements except for somatoform disorder. Students did not record adequate numbers of patients with somatoform, adjustment, eating, and childhood disorders. Outpatient exposure and on-call time in the emergency department may help address this in the future.
The variation from year to year on the consultation service at Dartmouth-Hitchcock Medical Center is puzzling. Encouragement of attention to the DMEDS by the clerkship director may explain the increase between the first and second year of data collection, but not the change from the second year to the third, because the director’s behavior was essentially unchanged. The “fatigue factor” may be a partial explanation.
Despite the fact that psychiatric patients tend to have significant coexisting medical problems, which often require management and interact with their psychiatric conditions (e.g., diabetes, hypertension, chronic pain), relatively few nonpsychiatric diagnoses were recorded (Table 1). This may be due to the clerkship director’s lack of emphasis on this area. Psychiatric patients also frequently have secondary diagnoses co-occurring with their primary diagnosis (e.g., substance abuse, PTSD, personality disorders). The DMEDS allows students to record only three diagnoses for any one patient. When comorbid psychiatric diagnoses were recorded, the medical diagnoses may have been pushed aside. Students mentioned the three-diagnosis limit as the reason for the lack of medical diagnoses, even though on average they entered only two diagnoses per patient. Since this was the psychiatry clerkship, students possibly felt they should be primarily recording psychiatric diagnoses. Examination of the DMEDS data from other clerkships revealed a parallel lack of recording of psychiatric diagnoses. This varied from 8.5% on family medicine to 0.4% on surgery, supporting the idea that students report diagnoses to fit the perceived clerkship expectation. This suggests that attention to medical problems may be lacking in the psychiatry clerkship but seems less likely than the above-mentioned habit of recording diagnoses to fit the clerkship. It also suggests a need for the clerkship director to emphasize the importance of medical diagnoses in psychiatric patients and for students to acknowledge their importance.
In clinical skills, the goals for counseling skills were never met (Table 3). In four important areas where it was expected that students would be involved in counseling (depression/suicide, disease issues, life stressors, and medications), students recorded between 2.8 and 3.4 patient encounters per clerkship, though the initial targets for all four areas were initially set higher (4–8 for each). It appeared that students were less likely to practice or document counseling skills interactions than they were to record patient diagnoses. In a clerkship which emphasizes development of such skills, this requires explanation. The targets may have been set too high. This seems unlikely, considering that the problems with the highest targets for counseling (depression/suicide, medications, family issues, life stressors, and safety) are the issues occurring most often on services where students were assigned. The clerkship director was certain, from observation as a ward attending, that students were discussing these issues with patients, offering more than adequate opportunity for students to meet the targets. It may be that “counseling” was not the correct term for what the students perceived they were doing. “Discussion,” “support,” or even “therapy” may have been terms more likely to prompt the data being sought. Also the reporting of such skills was emphasized less by the clerkship director than was the recording of diagnoses and may have been perceived as less important for documentation. Finally, the student-reported attitude toward the DMEDS might once again have played a role. By the time the students got to recording clinical skills near the bottom of the documentation form, they were less likely to take time to record as thoroughly.
When the four other competencies were addressed, a majority of students (60.4% over 3 years) recorded useful learning challenges. The data indicated that those students were actively involved in practicing advanced communication skills, reported both quantitatively and qualitatively (by student free-text comments) (Table 3). They were involved in discussions of sensitive areas, interviewing patients with significant communication difficulties, and considering patient preferences in treatment decisions, quite possibly to a greater degree than reported.
In professionalism, students reported assessing their own reactions in terms of putting the patients’ interest ahead of their own, becoming aware of their own biases and how these may affect interactions with patients, and issues of confidentiality and ethics (Table 3).
In the personal and continuous learning domain, students accessed new resources, especially web-based, up-to-date literature on the disorders affecting their patients. They reported that what they were learning was improving patient care and that their efforts were helping their treatment team care for the patients (Table 3).
Finally, in practicing in a complex health care system, students documented that they were learning the difficulties of managing their patients within a confusing health care system. They learned to work with a team, became more aware of the cost of medical care and health economics in terms of access to health care, and became acquainted with the roles and contributions of health care providers other than physicians (Table 3).
A problem in the four last competencies was the inconsistency of student reporting. Only 60.4% of students responded at least once in these areas (Table 3). Examination of individual reports revealed significant variation between students. Some documented significant numbers of interactions, while others reported only one or two. It is likely that some students are more interested in examining what they are learning and experiencing than others. Although documentation of patient diagnoses and clinical skills was required and had specified targets, there was no obligatory reporting for these four competencies. This may explain why some students did not record their challenges in these areas. It also may be again that by the time they reached these areas, near the bottom of the form, their interest and motivation had flagged and these received little attention.
Ultimately we have learned that students value what they perceive that the faculty values. Had we required and made clear that each student was expected to record pertinent medical diagnoses and challenges in each of the four competency areas and that their faculty preceptors would review their documentation weekly, it would likely have gotten the same attention as the rest of the DMEDS documentation.
An interesting phenomenon that may have contributed to the results was the cascade of interactions between the DMEDS data, its effect on the behavior of the clerkship director, and the resulting effect that his changed behavior had on the reporting process. As the potential value of the data became more apparent to the director, his interest in the importance of recording was communicated to the students. They seemingly complied, with better documentation of encounters. The director then used student data to modify aspects of the clerkship, making the overall experience better and more comprehensive for the students.
Changes as a Result of the Project
Reviewing the Dartmouth Medical Encounter Documentation System (DMEDS) data has resulted in changes to the clerkship. We modified how the DMEDS is introduced, explained, and monitored to attain more consistent reporting. We adjusted learning targets for diagnoses and skills to a more realistic level. We modified assignments at clinical sites; more students now have both consult and inpatient experiences and all have exposure to night call in the emergency department. The combination consultation-liaison/child rotation was modified because the experience was demonstrated not to be educationally equivalent to other clerkship rotations. The didactic curriculum was changed to cover new topics. In the future, particularly in light of the insight and self-examination shown in the students’ free-text comments, we are likely to require that students document at least two challenges in each of the four areas that have been partially ignored. Finally, though not a result of this project alone, the DMEDS has been restructured to increase ease of use and speed of data entry, reducing the time it takes for students to enter their required data with a focus on making the process as user friendly as possible.
The DMEDS provides valuable data for direct use by students in what they are seeing, practicing, and learning. It encourages students to take ownership of their clerkship experience and to take more responsibility for ensuring that they search out the learning opportunities and patient diagnoses they have been missing. It serves as an excellent resource for a clerkship director in assessing what the student experience is on the clerkship. Reviewing the DMEDS data increases the awareness of deficiencies in the clerkship, stimulating change and opportunities for improvement, as well as encouraging students to self-assess what they are learning and how they are changing. By learning from student data how to improve our clerkship and by modifying the DMEDS computer-student interface, we have improved the benefit-to-time ratio for students and made the DMEDS more useful to the clerkship director as well.