I bring you a weekly bite-sized chunk of the science behind helicopter human factors and CRM in practice, simplifying the complex and distilling a helicopter related study into a summary of less than 500 words.
TITLE:
Does Crew Resource Management training work? An update, an extension, and some critical needs.
WHAT?
This review of CRM effectiveness asked the question of whether CRM training does actually work. It examined whether CRM changes attitudes and knowledge, improves observable crew behaviour, and ultimately improves organisational safety. It updated their influential 2001 review of 58 aviation CRM evaluations and extended the analysis beyond aviation into other safety-critical industries adopting CRM.
WHERE?
University of Central Florida and U.S. Army Research Institute.
WHEN?
Published in 2006 in Human Factors, around 25 years after CRM emerged in aviation and as its principles were increasingly being transferred to other high-risk industries.
WHY?
By 2006 CRM was widely accepted within aviation and increasingly copied elsewhere, yet the authors argued that its reputation had developed faster than the evidence demonstrating its effectiveness. They therefore sought to establish what CRM could actually be shown to achieve.
HOW?
Evidence was drawn from aviation, medicine, offshore oil production and maintenance, maritime operations, and nuclear power. The researchers reviewed 28 published evaluations of CRM training using Kirkpatrick’s four-level training evaluation framework:
- Reaction – did participants value the training?
- Learning – did knowledge, skills or attitudes improve?
- Behaviour – did workplace behaviour actually change?
- Organisational impact – did outcomes such as safety improve?
FINDINGS:
CRM generally produced positive participant reactions, but evidence for improved learning and behavioural change was mixed across and within industries.
The study could not establish whether CRM produced measurable improvements in organisational safety.
The authors argued that did not demonstrate that CRM was ineffective but that available evaluation methods and data were inadequate to prove its ultimate safety effect. Better evidence required an organisational commitment to access operational data and adequate evaluation resources.
SO WHAT?
This review become a seminal CRM paper because it challenged the industry to distinguish belief in CRM from evidence of CRM effectiveness.
Twenty years later, this remains a challenge for CRM practitioners. The paper distinguishes between delivering CRM and demonstrating that CRM works. Positive end-of-course feedback is weak evidence of effectiveness and it remains difficult to determine whether learning transfers into observable operational behaviour and where it contributes to organisational outcomes.
A 2008 meta-analysis found encouraging effects of training on attitudes, knowledge and behaviour, but large-scale reviews specifically evaluating CRM effectiveness have remained uncommon.
For contemporary CRM, CBTA and EBT programmes, the study highlighted the need for training to have clearly defined behavioural objectives, realistic opportunities to practise them, reliable behavioural assessment, and mechanisms for evaluating transfer into line operations.
REFERENCE:
Salas, E., Wilson, K. A., Burke, C. S., & Wightman, D. C. (2006). Does crew resource management training work? An update, an extension, and some critical needs. Human Factors, 48(2), 392–412. https://doi.org/10.1518/001872006777724444
competentaviators.com 