Acad Psychiatry 31:443-446, November-December
doi: 10.1176/appi.ap.31.6.443
© 2007 Academic Psychiatry
The Structured Clinically Relevant Interview for Psychiatrists in Training (SCRIPT): A New Standardized Assessment Tool for Recruitment in the UK
Rahul Rao, M.D., F.R.C.Psych.
Received Nov. 4, 2006; revisions received Jan. 26 and Apr. 17, 2007; accepted May 2, 2007. Address correspondence to Dr. Rao, King’s College, London, Department of Psychiatry, North Southwark Community Team, The Gatehouse, 1 Ann Moss Way, Rotherhithe London SE16 2TS, United Kingdom; tony.rao{at}kcl.ac.uk (e-mail)
|
ABSTRACT
|
OBJECTIVE: The multifaceted nature of training and the diverse backgrounds of potential Senior House Officers (Postgraduate Residents) require a novel approach to the selection of trainees wishing to pursue a career in psychiatry. The author reports the properties of a semi-structured interview (the SCRIPT) for assessing doctors short-listed for a large Senior House Officer Training Scheme in psychiatry in South East England.METHODS: Data from 3 recruitment periods between 2005 and 2006 was examined to assess both interrater reliability and variation in scoring between interviewers. All questions were operationally defined and were modeled on General Medical Council (UK) guidelines for Good Medical Practice.Result: For 3 consecutive recruitment periods, interrater reliability (measured by Cronbach alpha) retained a high level of significance (p<0.001). Differences between the maximum and minimum mean scores between panels differed by only 6% of the total possible score and differences between mean scores at the same (p values between 0.8 and 0.9) and different (p values between 0.2 and 0.4) interviews showed no significant differences.CONCLUSIONS: The development of a valid and reliable method for selecting Senior House Officers in Psychiatry shows promise in the recruitment of "tomorrow’s psychiatrists." Prospective data on the positive predictive value of individual scores in career development awaits further exploration. Given the birth of a new ‘Run Through Grade,’ the interview may have its place in the assessment process within Modernising Medical Careers in the United Kingdom.
Key Words: Evaluation • Careers in Psychiatry • Residents: Recruitment
|
INTRODUCTION
|
Traditionally, psychiatry has been viewed as a "shortage specialty," with considerable difficulties experienced by some United Kingdom Training Schemes in both the recruitment and retention of doctors undergoing Basic Specialist Training (1). This observation has been mirrored by similar difficulties experienced in the United States (2) and this may have been influenced, in part, by aspects of the undergraduate curriculum. In particular, the interaction between acquisition of knowledge, awareness of the therapeutic potential of psychiatric interventions, and direct patient contact all show a positive association with a positive change in attitude to psychiatry among medical students (2). However, such attitudinal change may not be sustained over the course of the undergraduate curriculum (3). To some degree, this recruitment problem has lessened over recent years, with considerable competition for some training schemes among graduates from outside the United Kingdom. For example, in 2005, the number of applications to the scheme described in the current study was 550; over 90% of these were from graduates outside the European Union. Although new legislation means that this is likely to change, there is still likely to be cultural diversity among potential applicants in the future, requiring closer scrutiny in the validity and reliability of assessment procedures during the recruitment process.
The Objective Structured Clinical Examination (OSCE) is now an established method of assessment at a postgraduate level in psychiatry and is also adopted by some Senior House Officer Training Schemes within the United Kingdom (the equivalent of years PGY1-4 postgraduate residency programs in the United States). However, on its own, this method of assessment may not provide a balanced overview of aptitude in assessing the suitability of doctors for training in Psychiatry. This is because the OSCE does not tap into the wider breadth of attributes required by the General Medical Council for "good medical practice" (4).
The main aims of the current study were to assess the interrater reliability of a semi-structured interview across three recruitment panels/time frames and to examine differences in average scores for the same periods of recruitment.
|
Method
|
The Guy’s, King’s, and St Thomas/South Thames (East) rotation is one of the largest Training Schemes in England, spanning seven "Trusts" across London and South East England and comprising 130 Senior House Officers (SHOs). Each Trust forms a subscheme in terms of postgraduate training and service organization. In contrast to the United States, where there is an established procedure involving electronic registration and application (ERAS), applications to Postgraduate Training Schemes in the U.K. have not traditionally been centralized, so that applications are made to individual Training Schemes via paper/electronic submission. In our own Training Scheme, applicants are screened at three different levels to assess suitability for training. The first of these, termed "long-listing," assesses essential criteria such as registration with the medical practitioner licensing body (General Medical Council), potential to benefit from training, and evidence of a genuine interest in psychiatry. Each application form is then scored by a panel of three psychiatrists to assign a score for other criteria; this forms the basis of short-listing for the interview stage. In a similar way to recruitment in the United States, points are awarded for other degrees/qualifications, prizes/other accolades, publications, teaching experience, IT and audit experience, evidence of leadership qualities, and outside interests. Short-listing criteria are made available to all applicants.
Since 2005, interviews have been standardized to reflect both face and construct validity, with each question being scripted and marked using predefined criteria that cover areas relevant to psychiatry within the General Medical Council’s Good Medical Practice document (5). The interview is supplemented by two clinical vignettes that assess knowledge, skills, and attitudes in simulated "real-life" clinical situations. These vignettes are designed to cut across all levels of experience and minimize possible disadvantage to doctors who have not trained in the United Kingdom. Prior to each interview session, a briefing is carried out for interviewers (by the author) in order to explain the scoring procedure.
Data were collected for three sets of SHO interviews for rotations beginning in February 2005, August 2005, and February 2006. All data were analyzed using SPSS/PC 12.0. Interrater reliability was assessed using Cronbach alpha for groups of raters. Any value greater than 0.6, was considered "acceptable" in showing consistency between raters. In order to compare average scores across time periods, an initial boxplot was carried out to examine the range of scores rated by each interviewer to assess whether mean or median value should be used. Comparison between average values was then made. The maximum score possible for each interviewee was 20 marks. All statistical analyses reported 2-tailed p values.
|
Results
|
The interview domain exploring individual areas of good medical practice examined aspects of patient care such as keeping up to date with medical practice, research, communication with patients, and maintaining good clinical care. Interviewees with limited understanding of the question and/or who demonstrated poor linguistic ability were rated poorly. Conversely, those who were fluent and used their own ideas creatively were rated higher. For example, one interviewee, in response to a question about the usefulness of information technology in health care, gave a narrative of the personal skills that had been acquired in proficiency with software application programs. The highest ranked interviewees gave the clinical advantages of IT (e.g., access to patient information out of hours, where risk and clinical details were uncertain) and limitations such as data protection and confidentiality. Similarly, for the clinical vignettes, fluent interviewees who adopted a systematic and structured approach and who gave valid reasons for their clinical decision making were rated higher than those lacking these qualities. In response to a clinical scenario where competing demands on time were presented for an inpatient with low risk and someone presenting to casualty with high risk, a poorly performing interviewee failed to identify the difference between deliberate self-harm and possible delirium, as well as adopting an unstructured and poorly justified approach that may have led to an unsafe clinical outcome. A highly rated interviewee was able to make a preliminary diagnosis on the basis of limited clinical information, prioritized clinical risk, and identified appropriate interventions such as excluding physical disorders and clarifying aspects of the history.
Table 1 details interview scores for the three interview periods. Reliability coefficients are shown for each group of raters. For two of the rotations (August 2005 and February 2006), there were three raters per panel; for the remaining rotation, four raters sat on each panel. All interrater reliability values were greater than 0.75, with all but one of the six groups of raters having a lower-bound 95% confidence interval of greater than 0.7. All values of Cronbach alpha were significant at the p<0.001 level. Boxplots for each score range for each rater showed evidence of an approximately normal distribution, in spite of the relatively small total number of observations. Mean scores were therefore calculated for each rater. The mean scores given by groups of raters showed little variation between panels at both similar and different points in time (Figure 1). Using paired t tests to comparing panels 1 and 2 at the three different time points, all mean differences were nonsignificant (February 2006: t= –0.5, p = 0.8; August 2005: t= –0.9, p=0.9; February 2005: t= –0.3, p = 0.8). Similar findings were observed from one way Analysis of Variance (ANOVA) to compare Panel 1 (F=2.5, p = 0.2) and Panel 2 (F=0.9, p = 0.4) mean scores between interviewers across the two different time points. The difference between the maximum and minimum mean scores between any two panels differed by only 6% of the total possible score.
View larger version (37K):
[in this window]
[in a new window]
|
FIGURE 1. Scores Given by Raters Across Three Rotationsa
a For each panel, total number of observations (count)=114 (3 interviewers per panel), except for February 2005 (N=116, 4 interviewers per panel).
b Mean scores: Panel 1=11.7 (SD=2.9); Panel 2=12.9 (SD=3.5)
c Mean scores: Panel 1=11.8 (SD=3.1); Panel 2=12.5 (SD=3.7)
d Mean scores: Panel 1=12.1 (SD=3.8); Panel 2=12.6 (SD=3.0)
|
|
Discussion
|
The use of an interview covering a range of standardized dimensions that assess knowledge, skills, and attitudes appears to be feasible. The consistency of such an approach is also shown by the high interrater reliability and similar rating outcomes between groups of raters in this study. It is perhaps also surprising that the range of scores for each rater showed an approximately normal distribution, giving weight to the validity of the procedure in distinguishing between trainees with a wide variation in attributes.
All questions in the interview were open-ended and addressed a clinically relevant aspect of training. Although not evident from the results, it is important to note that for each rotation, the proportion of overseas doctors who were interviewed was similar to that originally applying for the scheme.
Although core skills necessary to achieve competency in psychiatry may change very little, the infrastructure of the Health Service is changing rapidly. Today’s psychiatrist is required to be familiar with information and communication technology and needs to adapt with changing ways of working such as is evident within multidisciplinary teams. In other words, doctors should be "fit for purpose" to meet the changing demands on them at both an educational and a service level. Such facets are included within GMC guidelines for Good Medical Practice.
Another aspect of assessment involves putting interviewees in real-life situations where they are required to use skills such as problem-solving and applying clinical "nouse" to common clinical encounters. Such an approach has its roots in Problem Based Learning, a concept that is now nearly 20 years old (6) and has successfully been applied to teaching and learning in psychiatry residencies in the United States (7); it also taps into a plethora of qualities that may also aid the interviewer in making a decision to the effect of "would I want this doctor to look after my patients?"
Given that the practice of psychiatry involves the development of psychotherapeutic skills which may not be measurable, a possible criticism of the current study is that it is too simplistic in its approach to the selection of SHOs for training in psychiatry.
No method of recruitment and selection is free from pitfalls, but the current study reports a process that is able to assess qualities in a hierarchical fashion, the acme of which is clinical common sense. It is perhaps up to the teacher or clinical supervisor to ensure that such an approach is nurtured at an early stage of undergraduate training (8).
|
Conclusions
|
There appears to be considerable scope for replicating the SCRIPT system currently used for selecting SHOs for the above training scheme. The changing face of medicine may mean that the approach will need to be revisited frequently over the coming years, so that "tomorrow’s psychiatrist" can be "fit for purpose."
The predictive value of individual scores as assessed by a variety of outcomes such as performance at clinical examinations and within work-based settings will require further exploration as trainees progress through the training scheme. The reliability of the assessment tool also requires further replication across other training schemes in psychiatry.
|
REFERENCES
|
- Brockington I, Mumford D: Recruitment into psychiatry. Br J Psychiatry 2002; 180:307–312[Abstract/Free Full Text]
- Rao NR: Psychiatric workforce: past legacies, current dilemmas, and future prospects. Acad Psychiatry 2005; 27:238–240
- Singh S, Baxter H, Standen P, et al: Changing the attitudes of ‘tomorrow’s doctors’ towards mental illness and psychiatry: a comparison of two teaching methods. Med Education 2001; 32:115–120
- Baxter H, Singh S, Standen P, et al: The attitudes of ‘tomorrow’s doctors’ towards mental illness and psychiatry: changes during the final undergraduate year. Med Education 2001; 35:381–383[CrossRef]
- Good Medical Practice. London, The General Medical Council, 2001
- Walsh W: The McMaster programme of medical education, Hamilton, Ontario, Canada: developing problem-solving abilities, in Personnel for Health Care: Case Studies of Educational Programmes. Edited by Katz FM, Fulop T. Geneva, World Health Organization, 1978
- Schultz-Ross R, Kline AE: Using problem-based learning to teach forensic psychiatry. Acad Psychiatry 1999; 23:27–41
- Rao R: Dignity and impudence: how should medical students acquire and practise clinical skills for use with older people? Med Education 2003; 37:190–191[CrossRef]
This article has been cited by other articles:
|
|
|
|
 
P. Whelan, M. Meerten, R. Rao, P. Jarrett, A. Muthukumaraswamy, and D. Bhugra
Stress, lies and red tape: the views, success rates and stress levels of the MTAS cohort
J R Soc Med,
June 1, 2008;
101(6):
313 - 318.
[Abstract]
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2007
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
ABSTRACT
INTRODUCTION
