The national emphasis on attainment of competencies in the education of healthcare professionals has created a need for methods to assess competency in specific skills (1, 2). Competency in suicide risk-assessment and management is expected of mental health professionals, including psychiatrists, psychologists, social workers, psychiatric nurse-practitioners, and others (3, 4). According to the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Psychiatry and Neurology (ABPN) (4–6), assessment and management of risk for suicide is a core competency for psychiatrists. Moreover, calls have been made for increased training of primary-care providers in recognition and management of suicide risk, in light of research showing that approximately half of American suicide victims had contact with a primary-care provider during the month before their death (7). Although many programs have been developed to teach clinicians about working with suicidal patients (8–11), our literature review identified no published methodology for assessing the competence of individual clinicians in evaluation and management of risk for suicide.
This report describes development and evaluation of the reliability, validity, and acceptability of a method for assessing competency in suicide risk-assessment and management. We identified pertinent domains for a competency-assessment instrument for suicide risk-assessment (the CAI–S) on the basis of a literature review and focus groups with faculty from multiple sites in a large academic psychiatry department. We applied the CAI–S in the context of an objective structured clinical examination (OSCE) (12, 13), in which psychiatry residents and clinical psychology interns performed suicide risk-assessments of a standardized patient while observed by faculty members. The faculty rated learners' performance and provided feedback according to the structure of the CAI–S. We addressed the following questions about the CAI–S: What is its internal consistency, reliability, and interrater reliability? Do senior learners perform better than junior learners? Do learners who have more clinical experience with suicidal patients perform better than trainees with less experience? How satisfied are faculty and learners with this method of assessing competency in suicide risk-assessment and management?
+
Development of the Competency Assessment Instrument for Suicide Risk (CAI–S)
We developed the CAI–S based on review of the literature on the standard of care in suicide risk-assessment and management (14–18), adaptation of criteria for evaluating the quality of suicide risk-assessment developed in a previous study of the effects of training on risk-assessment (11), and review of the literature on measurement of competencies in medical education (6, 12, 19–23). To further enhance the content validity of the CAI–S, we incorporated feedback on drafts of the measure from focus groups composed of clinical faculty at each of three sites of a large academic psychiatry department; these were 1) a county hospital, 2) a Veterans Administration hospital, and 3) a university hospital. The focus groups discussed factors that they felt were important for trainees to master in performing a competent suicide risk-assessment, including both content-related information (e.g., asking about suicidal ideation, knowledge about local civil commitment laws) and process-related issues (e.g., establishing rapport with the patient; reviewing the medical record; and obtaining information from collateral sources, such as family members or other clinicians). The CAI–S includes a checklist of 30 items concerning components of the risk-assessment process, each of which is rated on a 4-point scale, from 1: Task Not Done, to 4: Advanced. The 30 items include interviewing and data-collection (sources of information: 6 items; types of information: 6 items); case-formulation and presentation (6 items); treatment-planning (11 items); and documentation (1 item). Also, the CAI–S includes a rating of the overall quality of the risk-assessment for suicide, on a scale ranging from 1: Unacceptable, to 8: Advanced. The items in the CAI–S are summarized in Table 1. (The full CAI–S is available from the authors upon request.)
+
Application of the CAI–S in an Objective Structured Clinical Examination (OSCE)
Participants, who are referred to here as “learners,” were 31 trainees who attended a 5-hour workshop on risk-assessment in July 2008; 26 were psychiatry residents (12 in the first postgraduate year, 14 in the second postgraduate year), and 5 were clinical psychology interns. The workshop included a pretest, lectures on risk-assessment for suicide and violence, an OSCE, and a course evaluation. This report concerns the components pertaining to suicide risk-assessment. Learners heard a lecture based generally on APA Practice Guidelines for the Assessment and Treatment of Patients with Suicidal Behavior (14) and received relevant materials. Another lecture covered medical-legal aspects of risk-assessment and documentation. The lectures recommended gathering information about risk and protective factors, rationally weighing the significance of those factors to estimate the level of suicide risk, developing and implementing a plan of intervention to reduce the risk, and documenting the process. (Additional information about the content of the lectures in this model of workshop training in suicide risk-assessment is provided elsewhere (11)).
Thirty-one faculty (26 psychiatrists and 5 psychologists) were trained in use of the CAI–S in an OSCE setting. In this training, faculty observed mock OSCE sessions in which an individual interviewed a standardized patient, and presented the suicide risk-assessment findings to a mock examiner. Faculty observers rated the quality of the suicide risk-assessment with the CAI–S, discussed their ratings, and calibrated rating differences.
+
Standardized Patients (SPs)
We trained 31 advanced trainees (27 psychiatry residents in their third or fourth postgraduate year, and 4 postdoctoral clinical psychology fellows) to be SPs in the OSCE. SPs were trained to follow a script based on a clinical vignette. The script described a young adult patient who presented to an emergency department with various risk factors for suicide and violence. The script included information about the SP's chief complaint, history of the present illness, psychiatric and medical history, family history, social history, and mental status presentation. (Additional details about the training of SPs and faculty raters are available from the authors upon request.)
+
Objective Structured Clinical Examination (OSCE)
Each OSCE team included a learner, an SP, and a faculty observer/rater. The OSCE included a 15-minute interview of the SP; 15 minutes for the learner to write a progress note concerning the patient; a 10-minute oral presentation by the learner, including a summary of the assessment and plan regarding the patient's risk of suicide; completion of the CAI–S by the faculty observer/rater; and a 25-minute discussion, during which the faculty observer gave the learner feedback based on the structure of the CAI–S. At the conclusion of the workshop, learners and faculty completed course evaluations.
+
Interrater Reliability Study
To evaluate the interrater reliability of the CAI–S, in July 2009, we conducted a second study in which six faculty-observers (four psychiatrists and two psychologists) rated videos of three mock OSCEs. In each video, a different learner interviewed an SP and wrote a progress note containing the assessment and plan concerning the patient's risk of suicide.
The project was conducted in the Department of Psychiatry at the University of California, San Francisco, and was approved by UCSF Committee on Human Research. Participants were informed that participation in the OSCE was voluntary, and that the data would be coded in a way that removed identifiers and would not become part of individual learners' training files.
We calculated Cronbach's α to characterize the internal consistency reliability of the CAI–S. We used t-tests and correlation analyses for continuous variables, and chi-square analyses for categorical variables, to determine whether learners who were more senior and had more experience with suicidal patients performed better on the CAI–S than those who were more junior and had less experience. The subsidiary study of interrater reliability used the intraclass correlation coefficient (ICC1). We used SPSS Version 15.0 for data analysis.
+
Baseline Level of Training and Experience
The learners reported a mean (standard deviation [SD]: of 5.5 (9.0) hours of previous formal training in assessing and managing suicide risk, and 1.7 (2.0) years of previous experience providing mental health services. The number of suicidal patients encountered before the workshop included one learner with no patients, nine learners with 1–5 patients, four learners with 6–10 patients, three learners with 11–20 patients, two learners with 21–50 patients, and five learners with 51–100 patients (seven learners did not answer this item).
The internal-consistency reliability of the CAI–S was high (α=0.94), supporting the conclusion that the items on the instrument measure a common domain.
In the subsidiary study of interrater reliability, the intraclass correlation coefficient (ICC) for the 30-item CAI–S checklist was 0.94. The ICC for the rating of the overall quality of the suicide risk-assessment was 0.95.
For purposes of data analysis, we categorized the learners as Senior (second-year psychiatry residents, who had 6 months of supervised inpatient psychiatry experience that included frequent suicide risk-assessments) and junior (first-year psychiatry residents and predoctoral psychology interns). Senior learners performed better than Junior learners on the CAI–S. Mean (SD) ratings of the overall quality of the suicide risk-assessment were significantly higher for Senior learners (mean: 5.9 [0.9]), than Junior learners (mean: 4.8 [1.2]; t [29]= –2.67; p=0.01). Similarly, mean scores of the 30-item checklist of components of suicide risk-assessment were significantly higher for Senior learners (mean: 89.6 [11.1]), than Junior learners (mean: 77.3 [13.0], t [29] = –2.79; p<0.01).
More previous experience with suicidal patients was associated with better performance on the CAI–S, measured both by the rating of overall quality of the suicide risk-assessment (r=0.43; p=0.03) and the 30-item checklist (r=0.43; p=0.03). Similarly, more hours of previous training in suicide risk-assessment and management were associated with higher ratings of the overall quality of the suicide risk-assessment on the CAI–S (r=0.41; p=0.04), although this positive correlation did not reach statistical significance on the 30-item checklist (r=0.28; NS).
Summative judgments of whether learners had minimal competency in suicide risk-assessment and management were calculated by dichotomizing the ratings of overall quality of the risk-assessment as either Competent (rated as 5: Competent, to 8: Advanced) or Not Competent (rated as 1: Unacceptable, to 4: Working Toward Competency). Suicide risk-assessments by Senior learners were significantly more likely to be rated as Competent than risk-assessments by Junior learners (χ2 [1]=5.28; p<0.03); 100% of risk-assessments by Senior learners (14/14) were rated as Competent, versus 59% (10/17) of risk-assessments by Junior learners.
On course evaluations, both learners and faculty rated the CAI–S as helpful for assessing competency in suicide risk-assessment. When learners were asked whether the CAI–S would be helpful for getting feedback from supervisors in real patient encounters with potentially suicidal patients, the mean rating was 5.9 (1.0) on a 7-point scale, ranging from 1 (Not At All Helpful) to 7 (Extremely Helpful). Similarly, mean (SD) ratings by faculty on the same 7-point scale about whether the CAI–S was helpful for rating competency in working with potentially suicidal patients was 5.5 (1.1).
The results of this study support the promise of a new method for assessing the competency of individual clinicians in suicide risk-assessment and management. The CAI–S showed good internal-consistency reliability and interrater reliability. Content validity was addressed by developing the measure based on the literature on suicide risk-assessment, and by consultation from faculty focus-groups at various sites of a large academic psychiatry department. Support for concurrent validity includes the fact that senior learners in psychiatry and psychology performed better than junior learners on the CAI–S in the setting of an OSCE. Learners with more experience with suicidal patients performed better than learners with less experience. Evidence of acceptability of the method includes the finding that learners and faculty found it a helpful structure for learning and feedback about competency in suicide risk-assessment.
Limitations include the fact that the study was conducted in one psychiatry department and included only a modest number of participants. Also, given the cost of having faculty serve as raters in an OSCE, future research could evaluate the viability of having SPs rate the CAI–S. Another limitation is that although the OSCE provides a standardized context for application of the CAI–S, SPs may not show the range of problems that is comparable with actual patients encountered by learners on clinical rotations. Future research could address the applicability of the measure in clinical supervision.
Historically, evaluation of the competency of individual clinicians in skills such as suicide risk-assessment was left to summative evaluations such as examinations for board certification or licensure. Given current trends toward expectations that residency and other clinical training programs will document attainment of specific competencies (2, 6), we anticipate increased demand for objective methods to assess such skills. Methods such as the one described in this report have potential to provide a framework for formatively assessing trainees in building skills in risk-assessment, measuring acquisition of these skills, and providing feedback to them in development of competency in suicide risk-assessment and management. Furthermore, use of the CAI–S in an OSCE format may assist training programs in summative assessments related to this competency; for example, using it as part of the Clinical Skills Verification Examination (24).
Preliminary results were presented at the Annual Convention of the American Psychiatric Association, May 16–21, 2009, San Francisco, CA.
This research was supported in part by Grant T32 MH-18261 from the NIMH.
At the time of submission, the authors reported no competing interests.