Towards E-VFR flight: The dawn of mixed reality in the rotary wing cockpit? 

How progress in head-mounted display technology could revolutionise critical helicopter missions.

Envision a world in which emergency aircraft and their crews can launch in response to medical and other critical missions in almost any flight conditions imaginable. E-VFR (Electronic-VFR) speaks of this future thanks to electronically augmented visual flight which gives a sufficiently enhanced view of the external world to allow crews to use visual flying techniques around the clock and in any weather conditions. Recent leaps forward in extended reality technology means the feasibility of this concept is not as far off as it may seem. Described by scientists at NASA as a “better than visual” flight regime it has the potential to offer game changing benefits to both operational capability and safety.

As mixed reality develops apace in gaming and other commercial fields, human factors scientists and others are working away behind the scenes to see how both the tech and its lessons can be applied in aviation. A proliferation of recent studies is pushing forward the commercial applications for mixed reality devices in the cockpit in a variety of guises, but perhaps the holy grail of these is the realisation of what is coming to be called ‘equivalent visual operations’, whereby a mix of head-mounted display technology and conformal symbology, supported by an aircraft’s own sensors, will come to together to eradicate the boundary between instrument and visual flight techniques. Having spent the last two months studying progress in this fascinating field, I take a look back at what the implications of all this could mean for the future of manned helicopter operations.

The opportunities offered by E-VFR are particularly relevant to rotary wing aviation. Although modern derivatives have only recently spread to civil aircraft, HMD technology has its roots in 1950-60s military helicopters. An expanded field of regard – the fundamental difference between Head-up Display (HUD) and HMD capability – is more critical to helicopter flying than fixed wing. Low level and hover flight in the obstacle environment demands a greater range of visual scanning to the sides of the aircraft, and aerial tasks such as hoisting, load-lifting, and deck landing, often depend upon lateral scan. Civil helicopter critical missions such as HEMS, police, and SAR will continue to be manned operations for the foreseeable future, and are increasingly expected to be capable of an all-weather, H24 service.

A heightened exposure to degraded visual environments (DVE) and a susceptibility to inadvertent visual to instrument flight events have historically contributed to high accident rates in the rotary wing sector. In the last decade EASA has made a push to radically improve safety statistics, part of which has been an initiative to search for technological solutions. EHEST  One of these solutions is the potential benefits offered by augmenting outside cues with conformal terrain, obstacle, and other overlays on HMDs to aid the pilot in critical phases of flight, and unsurprisingly, it has made studies on DVE the focus of much current research. Across the pond, a recent investigation by the Federal Aviation Authority identified a number of accidents and incidents where the use of an advanced vision system may have resulted in a better outcome, suggesting that the same considerations are being examined beyond EASA. Has the time has come to for helmet mounted display technology to break through onto the civilian market?

In 2016 Microsoft Hololens first appeared, swiftly followed by competitors versions such as Google Glass and Oculus Rift. The potential of these off-the-shelf holographic visors with cutting edge optics and integrated sensors was not lost on researchers in a wide range of fields from medicine to engineering. In aviation, the contribution they could make to HMD research was jumped upon, and experiments were soon carried out examining how Hololens-like technology could be integrated into the cockpit environment.  

Research trends

Human Factors researchers have been focusing these experiments in two areas. The first is in the creation of effective conformal symbology for novel display types. Conformal symbology is the presentation of artificial scenery content or flight guidance symbology that is overlaid on natural terrain or man-made structures in a way which conforms to real-world shapes and form. The German Aerospace Centre, DLR has led the way with much of the helicopter specific research, investigating elements of HMD design specifically focused on rotary wing operations and DVE. This has included studies comparing and contrasting experimental helicopter landing symbology, hover drift cueing, and surface modelling. For example, one study evaluated a series of synthetic sea surface representations to determine which provide the best visual cues to pilots, finding artificial models were more effective than natural representations. Another tested a dashed line as a hover cue for lateral drift, and height towers for vertical hover references. In one particularly interesting study, which demonstrates the flexibility and potential offered by mixed reality and sensor integrated HMD, they investigated the impact of a 3D exocentric synthetic perspective which displays a disembodied external view of the aircraft to the pilot. Testing during landing and hover tasks alongside an offshore platform showed that innovative perspectives of this kind can improve spatial awareness and flight performance, outperforming conventional views and receiving positive feedback from pilots.

Exocentric helicopter viewpoint as displayed on HMD 

The second focus of research has been on building up a body of evidence for the hypothesised benefits of HMD. This has been largely based on the premise that conformal HMD offer the pilot enhanced situational awareness and reduced cockpit workload, particularly in DVE. The difficulty for these studies is that situational awareness and workload are two concepts that are notoriously difficult to pin down, and demonstrating that novel display types have a significant effect on these is a challenge. So far, most research has been able to show little more than equivalent performance to fixed wing head up displays or traditional head down displays based on measurement of situational awareness and workload.

Instrument to visual flight – shifting the dividing line

However, there is an alternative method for demonstrating the performance advantage offered by  HMD, and this is to focus on the critically important transition between instrument and visual flight techniques. Traditional flight rules and flying techniques establish a hard dividing line between instrument flight (flying by sole reference to instruments) and visual flight (executed with visual reference to terrain). Many accidents and incidents occur at this critical point of transition from instrument to visual flight, for example, at the bottom of an instrument approach. In helicopter operations, the most challenging of these situations is a night offshore approach to a platform or vessel, where pilots have to rebuild their mental picture from one created by interpreting the instruments, to a matching one based on visual references, all whilst in a dark environment with limited or confusing visual cues. The human performance challenges of such a situation was attributed as a key factor in the 2006 accident of Dauphin G-BLUN in Morecambe Bay on approach to an offshore platform. 

The introduction of Point in Space instrument approaches introduces a new area of operational risk for this scenario, where a transition from instruments to the final visual flight segment at unprepared sites (with no prior visibility information) could contribute to a loss of control event.  The inverse of these situations is the transition from visual to instrument flight, a problem which helicopter operators are familiar with from incidents of inadvertent IMC and brown/white out approaches. In all cases, we are talking about a degraded visual environment in one of its different guises.  

On a traditional instrument approach, the change from instrument to visual scan is a point in time defined by the decision altitude. We can hypothesise that what HMD contribute to is an amplification of this time period in such a way as to reduce exposure time to poor conditions with limited visual cues. On the one hand, conformal display symbology enhances and prolongs the ability to maintain safe visual flight by improving the perception of visual references, while on the other, full regard head mounted instrument displays allow a continued instrument scan to merge  into the outside visual flight environment. Therefore we could describe the objective of conformal HMD technology to be to allow a level of human-system integration that redefines the distinction between instrument and visual flying techniques. Ultimately, the achievement of Electronic-VFR capability would eliminate the transition from instrument to visual flight altogether.

Towards E-VFR flight: how HMD and conformal imagery can redefine the dividing line between instrument and visual flight techniques.

Marrying displays with data

Future research will focus on greater sensor integration to feed the display of a wider range of conformal information. The potential scope of this for critical missions is wide. For offshore search and rescue, search patterns, sea current flows, and wind effects could be conformally marked, and for firefighting operations, water sources, deconfliction levels, and virtual entry and exit gates to firefighting areas. Integrating a variety of sensor data such as NVD, FLIR, RADAR, LIDAR, TCAS, locator beacon and homing signals, etc. offers ground-breaking improvements in all-weather and nighttime capabilities. Some of this has already been achieved in the civil market with commercial designs in the form of Airbus’s HELLAS-A LIDAR sensor for wire detection, and Leonardo’s obstacle proximity LIDAR, but these are still being presented on conventional head down displays. The cutting-edge of military technology has taken a step further in beginning to integrate conformal symbology from on board sensors into helmet mounted displays, with Elbit Systems’ BriteNite^TM  system a good example of market leading technology in the field of helmet mounted systems fed by a sensor array.

The path to Electronic VFR

Undoubtably, a combination of the maturity of current optical and display technology coupled with modern computer processing power has contributed to a period of exciting technological progress in the field of HMD in recent years. The pressing issue of helicopter loss of control incidents in DVE is giving impetus to the commercial development of helmet-mounted displays for the civilian market for the first time. However, there is first a need to successfully prove HMD as an interface for all this new technology. Whilst there are still technological hoops to jump through to progress display integration and conformal symbology sufficiently to reach the full E-VFR flight,and the Equivalent Visual Operations envisaged by NASA, an even greater challenge lies in gathering evidence for the safety and operational benefits that can take conformal HMD to the certification stage. That work is underway, but the answer does not lie in the twin metrics of situational awareness and workload alone. It also requires a greater understanding of divided attention, the dynamics of pilot scan, and mental model-building in the critical transition period between instrument and visual flight. And it puts human factors – as ever – right at the heart of advancements in 21st century helicopter operations.