In the early 1990s, the University of Melbourne revised the medical student course to introduce contemporary approaches to teaching and learning, including a higher profile for teaching population health and social medicine, earlier clinical attachments, and more small-group and self-directed learning. Pediatrics is taught in the 9-week child and adolescent health program in the fifth or sixth year of the medical course and is coordinated by the Department of Pediatrics, based at the Royal Children’s Hospital, Melbourne. Other subject-based evaluations of innovations in this program have been reported elsewhere (1, 2).
The new medical course initially situated student learning about child psychiatry within adult psychiatry teaching; however, there was little emphasis on the child-parent relationship, and anecdotally we know that the students had variable clinical experience with children. The students’ learning was also isolated from other supervised clinical learning experiences with medically unwell children and adolescents. A decision was made to relocate child psychiatry teaching to the pediatrics rotation to introduce a variety of interactive learning activities and to have one consolidated teaching day. Although the costs were not formally estimated, this approach may be more resource-intensive but is effective for both teaching and learning.
We are not alone in seeking the best arrangements for child psychiatry teaching. A survey of Australian medical schools (3) showed a large range in the number of hours dedicated to the teaching (between 5 and 80), suggesting a lack of agreement about the importance of the subject. Recently, Sawyer et al. (4) proposed a collaborative approach to the pedagogy of setting learning objectives and methods across Australian medical schools.
Our curriculum development also engages us in determining a more appropriate pedagogy to stimulate interest in a career in this field (5, 6).
This article describes an approach that is, of course, not entirely new; in 1988 Dobson (7) described the use of role play, video, and other methods. We now describe a systematic evaluation of the use of these learning methods in the teaching program.
The Need for Effective Teaching for the Health of Australian Children and Adolescents
Attention to the quality of the teaching and learning provision is timely, given that the latest national survey identified that 14% of children and adolescents have impairing mental health problems and that general practitioners and pediatricians are the health professionals most commonly consulted for help (8). Mental health is now regarded as one of the “new morbidities” in Australian pediatrics (9).
The New Melbourne Teaching Day Intervention
To address the deficits in the students’ experience, a one-day teaching program was designed in 2006 for the child and adolescent health course that is delivered annually during each of four 9-week terms for approximately 75 students per term. For the child psychiatry teaching, each group was divided into small learning groups (∼15 students) for the day. One academic covered four key topics. Four structured small group discussions followed each of four mini-lectures on the categorical and dimensional description of child psychiatric disorders requiring multiple informants, age, gender, and IQ to be considered; key high-prevalence and low-prevalence child psychiatric disorders; current important biological and psychosocial risk and resilience factors for onset, progression, medication and psychological treatment, and monitoring of treatment responsiveness of child psychiatric disorders; and key principles of biological and psychological treatment for child psychiatric disorders.
Videotaped case assessments then involved students in 1 hour of task-based observation of history taking/problem formulations; then, in two separate 1-hour sessions of role plays, students considered the experiences of the parent of a “disturbed” child. In contrast to the previous lecture-based teaching, this program is child-focused, varied, and interactive. This design provided opportunities for students to engage with the theoretical component and personally engage with the (videotaped) clinical practice component. We intended to help students engage with some of the realities that arise during extended child psychiatry consultations (10, 11), provide time for the teacher to bring to life the values of the profession (12), and expose students to stimulating ideas and positive influences in the field (13).
We were guided by the notion of evaluation as “inquiry,” producing knowledge based on systematic inquiry to assist decision-making about a program (14). We sought to gain an understanding of the students’ incoming level of knowledge, their experience of the teaching, and changes in their knowledge that were identifiable at the end of the program. Consistent with guidelines for evaluation of medical education interventions (15), the evaluation was planned and prepared for implementation alongside the introduction of the teaching day. Also, the education evaluator was external to the teaching and the assessment of the students. Finally, our approach to data collection and data analysis was exploratory rather than confirmatory in intention.
Although a comparison with evaluations of previous teaching in the course would be valuable, no similar evaluations had been conducted. Unfortunately, only broad course survey data are available, not information at the level of subject or rotation.
The evaluation did not require approval from the University Human Research Ethics Committee (16), because it was considered a normal education practice of current education instructional strategies (17).
We used a pre/post method to assess students’ knowledge change. We estimated that a large volume of textual responses from two of the four small groups in one term would produce an adequate sample for analysis. The students were asked to complete an anonymous (codenamed) questionnaire covering the main topics in the teaching day.
We also collected two levels of written feedback from the student group on rotation (n=75) to inform us of their perceptions of the program’s worth immediately following the teaching day and later, in a different format, at the end of their pediatrics term. The data collection was administered by the education evaluator separately from the teacher and the course coordinators.
The pre- and postprogram knowledge questionnaires were assessed separately by the academic child psychiatrist (AV) and the education evaluator (JG). The data were entered into a spreadsheet and assessed against the model answers provided by the academic child psychiatrist to the educationalist.
The students’ pre- and postprogram questionnaires were matched using codenames. The assessors separately analyzed the groups’ responses, identifying both correctness and quality in the students’ knowledge of key topic areas. Descriptive statistical analysis was conducted. We entered students’ additional comments in the feedback forms into a spreadsheet and analyzed manually to identify themes. Conclusions were arrived at together after the education evaluator described the interim results.
Knowledge Assessment: Quantity
Our main goal was to establish whether students were leaving the teaching day with improved knowledge in child psychiatry that would assist them to respond to and learn from observations of children and families in their ward attachments. From two of the small teaching groups (n=29), we collected the full text pre/postprogram answers to questions covering knowledge in the nine key topics listed in the subsequent text.
The correctness of the students’ answers to the nine questions was assessed against model answers prepared by the academic child psychiatrist. For each question, there were between four and eight items in the model answers. The students’ responses were grouped into the number of items/factors each student was able to generate in response to each question. The results are presented in Table 1, showing the numbers of students who provided half or more than half of the number of items in each model answer. A capacity to produce half or more than half was selected as an indicator of a satisfactory number of responses compared with the model answers.
Table 1 illustrates a noticeable change in knowledge on each of the nine questions. Upon entry to the teaching day (preprogram), none of the students was able to construct any response to four of the nine questions, but this changed at the end of the day (postprogram): knowledge of medication treatment (from 0 to 18 students); biopsychosocial formulation (from 0 to 13), key factors of childhood (from 0 to 27), and the influences of genes (from 0 to 24).
Knowledge Assessment: Quality
In addition to the quantity of change in knowledge described earlier, our analysis of the content of answers to the nine questions gave us further information about the success of the 1-day teaching program. The relatively poorer results for the first two questions are noticeable: these results could reflect primacy and recency effects because this content was taught in the middle of the teaching day and received less emphasis than the other topics or it may reflect that the content was the focus of lectures rather than part of the experiential learning activities. However, the content of the answers tells us that the students who did answer had learned that multiple informant reports are a crucial component in determining the presence of a child psychiatric disorder.
Very few students were able to correctly describe even one of the key principles of medication and psychological treatment prior to the teaching; when they did attempt an answer, most referred to medication, not psychological treatment. At the end of the day, most of the students were able to separately and correctly describe medication and psychological treatments and, importantly, almost always included therapeutic alliance and empathic engagement in their lists.
The small number of students who were able to describe any of the key components of a biopsychosocial formulation at the start of the day usually wrote “family history.” In the postprogram questionnaire, most were able to correctly name at least one key factor, most commonly family history, coping style, genetics, or self-identity.
Similarly, of the small number upon intake who could describe even one of the key psychological, social, and biological factors of childhood, they named family. After the teaching, however, many students were able to give 4/4 factors in the model answer. If they missed one, it was most often the biological factor (18). In terms of the expected answers for psychological factors, almost all of the students correctly gave egocentricity in their answer.
The students’ knowledge about adolescence improved, particularly being able to name more than puberty as a key factor.
In describing mechanisms by which genes exert their influence, the students were not only able to give more comprehensive answers later, but they were also able to structure their answer better than they could on arrival. Most were able to give a model answer, but where they did not, they commonly missed temperament and gene-environment correlation.
Half of the students showed some knowledge of prevalence prior to the program, and after the teaching almost all of the students were able to give a model answer on prevalence. Prior to the teaching, attention-deficit/hyperactivity disorder and depression were listed but usually wrongly classified as low prevalence disorders. Also, students who listed autism and/or eating disorders were unable to classify them in terms of prevalence. These understandings were corrected during the course and were reflected in the results. Finally, most students demonstrated an increased, more structured, and more detailed knowledge of the key aspects of treatments used in child psychiatry.
End-of-Day Student Evaluation of Topics/Sessions
The question posed to students at the end of the day was, “How clinically useful was this component of the teaching day?” and a 4-point scale of usefulness was provided for them to rate each topic/session. Students were very clear about the clinical usefulness of all the teaching and learning activities: between 79% and 96% of the 125 students surveyed (over half of the whole student group for the year) rated the sessions “very useful” or “useful.”
End-of-Term Child and Adolescent Health Course Surveys
Over three terms, 161 students completed the end-of-term surveys. On average, 40% agreed or strongly agreed that the child psychiatry teaching day was a “worthwhile use of [my] time,” and an average of 37% agreed somewhat that it was worthwhile. The students are making this judgment after 9 weeks of excellent clinical teaching and learning in the pediatrics and adolescent health course.
These results confirmed the value of the teaching day. The feedback from students indicates that over 80% considered the teaching day useful, and the language used in their comments is noticeably positive. The ratings and comments allowed us to shorten the teaching day and to design a specific observation task to help students learn from the videotapes.
Most important to us, however, is the noticeable improvement in the group’s knowledge of child psychiatry areas that we measured. The students arrived with, at best, little knowledge on many of the topics, and on most topics they finished the day with improved ability to construct answers to the nine questions.
This analysis of the pre- and postprogram answers provided direction for teaching adjustments to be made. A definitive knowledge of the key components that contribute to determining a child psychiatric disorder and the key biological factors relevant for child psychiatry are two topics that require more or different attention (19). All other questions showed at least satisfactory levels of knowledge, suggesting that the current approach is successful in producing immediate knowledge change.
The teaching ability and commitment of the individual academic are essential to the positive learning outcomes, but cannot be captured in writing. Two particular limitations of this evaluation are the small sample size and the need for further study of the retention and integration of the knowledge gained over longer periods.
At the time of submission, the authors reported no competing interests.