Development and Reliability of the Comprehensive Crisis Plan Checklist

Summarized by Erin E. Gurdineer, University at Albany, State University of New York and Amanda B. Nickerson, University at Buffalo, State University of New York

In this article published in Psychology in the Schools, Aspiranti, Pelchar, McCleary, Bain, and Foster (2011) explain the development and research to support the reliability of the Comprehensive Crisis Plan Checklist (CCPC). This measure was developed as an extension of the brief crisis plan checklist, originally developed by Saxon and Bain (1998). The CCPC is based on Caplan's (1964) theoretical model, which proposes different levels of prevention (i.e., primary, secondary, and tertiary). The authors used this framework to structure the CCPC in the areas of crisis prevention, intervention, and postvention. The checklist is organized in a table format that presents subcategories under the three main areas. Under prevention, the subcategories are general crisis, violence, accident, and suicide prevention areas. Items within these prevention subcategories include having a media policy, identifying suicide prevention programs, and having a list of alternate team members. Within the intervention category are subcategories of general crisis intervention and specific crisis intervention plans, such as those for weapons at school and natural disasters. The last category is general crisis postvention, and does not include any subcategories. Items in the postvention area assess things like crisis hotlines and follow-up grief counseling. Next to the items in the checklist, there is a column for comments that can be used if the evaluator wants to describe specific plan components or if there are questions over the clarity of the plan elements. When completing the checklist, the evaluator marks the yes or no column to denote if the item is present or not in the crisis plan. If an item were present in the plan, the evaluator would then record the page number where the item was described in the last column.

To further examine the clarity of items on this checklist, the authors obtained five school crisis plans and had researchers rate each plan. Any discrepancies or inconsistencies were clarified, and unclear or unnecessary items were revised, deleted, or split into two separate items. The authors then conducted two phases of data collection to establish interrater reliability estimates. School crisis plans were obtained and labeled A through E. Two graduate students rated each plan to allow for reliability percentages. Due to one student not reading his/her given plan, results for only four of the plans were obtained (excludes D). The results of this first phase, in terms of how consistently raters scored crisis plans, ranged from agreements of 79% to 87% across the four plans. The average interrater agreement across these plans was 81%.

The same procedure was followed for the second phase of data collection on seven school crisis plans (labeled A through G, five of which were identical to those used in the first phase). The results of this phase ranged from 73% to 95%. The mean percent for this phase was 86%. Cohen's k coefficients were also calculated during this phase, which estimates the percent of consistency in ratings that is significantly different than what would be expected to occur by chance. The k coefficients for the plans ranged from (k = .35, p < .01) to (k = .73, p < .001) across the seven plans. Overall, all of the coefficients were significantly different from zero except for G.

The results suggest that there is adequate interrater reliability for the CCPC, as well as fair to considerable agreement according to the k coefficients. This lends support for the clarity and objectiveness of this measure for evaluating school crisis plans. One limitation of this study was deciding what constitutes proper inclusion of an item (i.e., just mentioning or having to describe an item to justify checking it off). The authors suggest that raters agree ahead of time what the inclusion criteria will be to obtain more consistent ratings.

This article gives valuable information to administrators and school psychologists. This checklist establishes a beginning in empirically based evaluation of school crisis prevention and intervention. Evaluating school crisis plans can help determine needs in terms of being prepared to deal with a crisis event. Suggestions for future research in this area include incorporating information in the checklist to examine if prevention and intervention activities are supported by evidence-based research, and adapting the checklist for use in university settings.

References

Aspiranti, K. B., Pelchar, T. K., McCleary, D. F., Bain, S. K., & Foster, L. N. (2011). Development and reliability of the comprehensive crisis plan checklist. Psychology in the Schools, 48, 146-155. doi: 10.1002/pits.20533

Caplan, G. F. (1964). Principles of preventive psychiatry. New York, NY: Basic Books.

Saxon, L. C., & Bain, S. K. (1998, April). A checklist for evaluating school crisis plans. Presentation at the National Association of School Psychologists Annual Convention, Orlando, FL.