Competency based training. By trying to solve one training problem are we creating another?

At the beginning of this month I tuned in to the Royal Aeronautical Society’s webinar titled, Flight Crew Competence; Assessing what and how? The webinar aimed to address the concept of Evidence Based Training and Competency Based Training (EBT/CBT) and consider the impact it has had on the experience of instructors, examiners and trainees. The panel represented a series of viewpoints from across areas ranging from flight operations, academia, learning and development, and psychology, to share insights about the history and development of the programme, and how competencies are identified, observed and trained. As you would expect, I found some areas more thought provoking than others, but by the end of it I was left with one nagging question that stood out for me. It was this:

By trying to solve one training problem by establishing CBT techniques in the industry are we creating another with the complexity and extent of the knowledge demanded of the instructor cadre to be able to train these methods effectively?

I was lucky enough to be able to put this question to the panel, and the response was an interesting one. In short, they agreed that this could indeed present a challenge to the practical success of converting CBT from theory into practice on the aviation front line.

The problem that CBT purports to solve.

How do you define a competent pilot? 

The point is, that whatever definition you come up with, it will inevitably change and evolve over time. What we deem to be a good pilot now looks quite different to what it was 50 years ago.

The origins of Evidence Based Training arose from an industry-wide consensus that changes in operating practices, technological advances, and the understanding of teaching and learning techniques have taken place. The time was right for a strategic review of the way we approach recurrent and type-rating training. The aviation industry has changed and progressed, and how it trains its crews needs to change and progress with it.

The problem with traditional methods of training and checking is firstly that they are quantitative. One example of this is that they require that you meet a quantified minimum hours of practice as a standard, which speaks nothing to whether or not an individual is actually competent. Some will require more hours and others less to become competent in any given skillset. 

Secondly, they are proscriptive. That is to say, the training required for any individual is determined by the testing criteria (which is set top-down by the regulator) rather than being determined by an individual’s progress towards being capable of doing the job they are training for. 

Thirdly, the testing criteria itself is outdated. It is mostly based on evidence of accidents relevant to airline operations alone, and principally derived from accidents to early generations of jet aircraft. The traditional philosophy is based on the belief that simply repeating pilot exposure to ‘worst case’ events in training is enough to ensure competency. Over time, as new incidents and accidents occurred they were added to the requirements, resulting in ever more crowded training programmes. This created a ‘tick box’ approach to training and one which often overlooked the more complex contributing factors behind the technical failures that might have been the trigger factors for an accident.

As an example of the dislocation between traditional mandated training and the real world, consider the unstable approach paradox. When a final approach is unstable pilots are expected to go around. Evidence shows that they usually don’t. But when they do the missed approach is almost always badly flown. In contrast, when they land from an unstable approach, 98% of the time they do so with no issue. The requirement to go around from an unstable approach therefore arguably works against safety. 

But let’s look at the root cause of the problem. Go arounds in training are usually performed with one engine inoperative from a defined minima and with no visual references. Go arounds in real life are almost always flown with a low weight aircraft, all engines operating, and are usually visually. We are not training how we are flying.

The proposed solution: Evidence Based Training and Behavioural Competencies.

Evidence Based Training has been developed on the basis that pilots should be trained to become competent in the specific demands and challenges of their particular aviation operation. This can be done by using evidence drawn from relevant accidents, incidents, training reports, and academic studies to design training programmes that meet those specific requirements.

Having identified the kind of training required – the things that you want your pilots to be competent in to carry out their role safely – the next challenge is how to develop and evaluate crew performance according to a set of competencies.  This is where things become much more complex. We are no longer able to separate training into a series of individual skills or manoeuvres that can be simply (and largely unthinkingly) ticked off by an instructor. Instead we need to determine the idea of competency within the context of an integrated set of both technical and non-technical skills.

Under a competency based system trainees have to be able to demonstrate – and instructors have to be able to assess effectively – all of the following:

1. Application of Procedures
2. Communication
3. Aircraft Flight Path Management, (automation & manual)
4. Leadership and Teamwork
5. Problem Solving and Decision Making
6. Situation Awareness
7. Workload Management

It is relatively easy for flight instructors to concentrate on the technical side of training. It’s definable, it’s measurable, and it is more easily assessed and critiqued. The same cannot be said for the non-technical skills listed above. They all interplay with each other and are not easily picked apart. The exercise of judgement in assessing the non-technical aspects is much more subjective compared with assessing the success of a rejected take off, or actions in response to an engine fire. And there are lots of non-technical categories to assess. They require the assessor to be attentive to many different aspects of performance at the same time.

The best and most objective system that has been put forward to observe and assess these kinds of complex behaviours in action is a behavioural marker system and accompanying grading matrix. A behavioural marker system is basically a taxonomy (categorised list) of pilot behaviours. If a behaviour has been observed in flight it can be judged as evidence to prove competency in that category. For example, if the behaviour “seeks and accepts assistance, when appropriate,” is observed, then it is accepted as one evidence for competency in the category of Workload Management. 

Once those behaviours have been collected, or ‘observed’ in each category, an instructor can grade competency against a matrix that requires him to assess how many have been observed, how often, with what level of success, and with what resulting impact on safety. If it sounds technical, it’s because it is. Not only are there multiple competencies to consider, the arrival at a decision on a grade to apply is also multifaceted.

The problem of instructor training –  how to train the trainers.

The methodology is not perfect, but then it doesn’t claim to be. Nor am I suggesting that it doesn’t work. It does. The problem is that for it to be able to work effectively and achieve that paradigm shift in training practices that Competency Based Training purports to do, it depends in turn upon instructors who are themselves competent in how to apply these techniques.

Instructional Competency

Instructional competency in Competency Based Training techniques depends upon:

Knowledge

1. A full understanding of the rationale behind the concept and its key principles. 
2. An in depth knowledge not only of the competencies themselves, but of the behavioural markers taxonomy behind their evaluation and assessment.
3. An understanding of the assessment criteria and grading methodology.

Skills

1. This knowledge underpins the development of root cause analysis, the key instructional skill in making a success of competency based training. Understanding how to identify the root cause of an error or an adverse cockpit event takes practice and a capacity for analysis.
2. A skilled instructor will be able to understand the KSA (Knowledge, Skills, Attitudes) demanded of trainees and how to apply them – particularly the impact of attitudes – on improving crew competency. 
3. The ability to apply the grading matrix to come to as objective a judgement as possible on an assessment outcome. The fact that a pass/fail can be based on any of the competency areas including non-technical skills makes this decision-making skill even more challenging.

Attitudes

1. Empathy is a key trait to allow an instructor to apply the above skills successfully.
2. Buy-in: For some, adapting long engrained practices to these changes will mean a paradigm shift in instructional technique. Instructors have to want to take this step, and have an open mind to its benefits. 

The EBT/CBT initiative to address an outdated and sometimes anachronistic system of pilot training has led to a paradigm shift in thinking about how flying training should be approached, and major changes in training philosophy and methodology. The weight of all this has fallen on training departments and instructors, requiring that they get to grips with new knowledge and the practical skills to put these changes into practice. But despite the breadth and depth of these new demands on the training teams and the simple fact that the success of any competency based programme will depend upon the competency of the instructor cadre themselves, very little extra attention has been paid to what we demand of our instructors, how we choose who is suitable, and what we should be doing to teach them the skills to train others effectively. 

The training on EBT/CBT that is given to instructors is sometimes not as comprehensive as it might be. It is sometimes passed on by people who themselves don’t have a firm grasp of the concept. It can leave more questions than answers, and often involves minimal experience of actually applying the techniques in practice, before sending instructors off to spearhead the change. As ever, even the Authority admits in CAP 737 that “In terms of operational flight safety, instructors hold one of the most influential positions in the industry…[but, although]…the regulation is clear on the expectation and abilities required of the instructor, it offers little in the way of guidance.”

The problem of instructor selection.

It is not just about ensuring that we train the trainers adequately either, because the instructors that go on to lead CBT programmes need to have more than just the knowledge and skills. They need the right attitudes as well. Success will depend above all on the attitude to embrace change, and the attitude to learn a new and challenging set of skills. Not everyone will see it as progress, and not everyone will buy in to the changes.

Accepting a shift in the balance of training towards non-technical skills and briefing and debriefing using facilitation isn’t for everyone. Some instructors simply aren’t comfortable with facilitation as a valid and valuable instructional technique. Facilitation depends more on the character of the instructor than other forms of instruction. They might prefer a more hierarchical and autocratic teaching style, and will continue to defend its efficacy, asserting that facilitation doesn’t get the best from their trainees and undermines their own status and authority as an instructor.

Even more won’t have the motivation to put in the effort required to learn and apply the new knowledge, preferring to remain in their comfort zone and fall back on an easier path of the familiar style that ‘has worked for all these years’. How to identify the willing from the reticent, and the motivated from those stuck in the past is one of the challenges such significant change brings. And how we go on to separate the two groups so that CBT can achieve its goals led by an instructor cadre who are willing to invest in it is a more troublesome question than it might first seem. 

The problem of systemic and cultural factors

A further issue that is a product of a long-held culture within aviation is that instructors are not necessarily chosen by the system for their instructional potential. Ideally the system would select on the basis of character – for empathy and likability as well as other more obvious flying instructor traits – and for expertise and dedication in the area of teaching and learning as a vocation. Instead the system often selects its instructors on the basis of technical flying skills. For example, the RAF has always demanded an above average ability in the air to be able to go on to instruct. This may seem like a sound policy, but selection of pilots to go on to become instructors on the basis of their technical skills alone, means that they are being chosen for factors totally unrelated to their knowledge, interest, and skills in teaching, learning, and development. It also ignores the fact that under the new paradigm, technical skills sit alongside a raft of non-technical skills that also need to be taken into account. Worse than being selected for their technical skills, many instructors reach their position by virtue of seniority, status, or reward. Many of these are not best suited to teaching this kind of training. Furthermore, the coupling of TRI/E status to seniority often means that those individuals are burdened with other responsibilities as well forcing them to divide their time and attentions between training and other management tasks.

By trying to solve one training problem are we creating another?

In EBT/CBT we have layers of complexity which themselves need to be explained sufficiently that the trainers can then pass them on to trainees with confidence and competence. A lot of thought has been put in how to solve the first problem, but in solving the first it is my sense that we have created a second, which has so far been somewhat overlooked.

Progressing training practices towards the concept of EBT/CBT is an ambitious project of technical and cultural change. Doing anything well takes time, effort, and resource. Furthermore, such significant change was always going to throw up problems of change management and opposition created from engrained cultural habits. These are not intractable problems, but they do address embedded beliefs and practices, making it easier to come up with the solutions in theory than to put them into practice. The ideal solution is a dedicated training team with a vocation for teaching and learning. These would be people who are specialists in the specific domain of flying instruction but who are also people who are able to set their instructional skills within the domain-independent context of the theory and practice of education and learning. Aspiring instructors should be prepared to specialise and dedicate their careers to the field of training, leaving others to management and operational leadership.