What impact does AI have on the vigilance of airline pilots? Over-reliance on automated systems tends to make us less vigilant. Will the growing integration of artificial intelligence increase the number of air accidents?

Pilot call it the situation awareness or situation conscience. . In other words, maximum alertness and awareness of all the parameters of your flight. This will enable you to anticipate the next steps and react immediately and effectively to unforeseen events.

The problem with the autopilot is that it allows pilots to fly - with hundreds of passengers - without being fully aware of their aircraft's situation. 

The impact of AI on driver vigilance: a perfectly avoidable fatal accident

In his book, The Dangers of Automation in Airliners: Accidents Waiting to Happen, Jack J. Hersh describes two almost identical situations. Except that the first resulted in the deaths of over fifty people, including the pilot and his co-pilot. The second resulted in a smooth, uneventful landing.

The first situation is that of Colgan Flight 3407 in February 2009. A routine flight from Newark to Buffalo with two experienced pilots, Captain Marvin Renslow and his co-pilot Rebecca Shaw. They were flying a Bombardier DCH-8-400, also known as the Q-400, an aircraft that both pilots had mastered for years.

As Hersh explains so well, it was a chain of small snags that led to the fatal error. Renslow hadn't slept much, Shaw was suffering from a cold. It was freezing that evening, so Renslow triggered the de-icing system. 

Just before landing, Renslow reduced his speed. But, instead of sticking to the stérile cockpit - which is to talk about nothing but the landing manoeuvre - Renslow and Shaw continued to discuss their experiences of flying in icy conditions. Shaw can be heard sneezing and sniffling between sentences on the black box recording. Meanwhile, the autopilot continued its descent. 

Suddenly, the accelerator began to vibrate, and a siren sounded in the cockpit. This was the signal for a stall. 

Let me explain. To stay aloft, an aircraft has to travel at a certain speed. As soon as it no longer reaches this speed, the wings can no longer support it and it descends. This is what happened with flight 3407. 

To increase his speed, Renslow would have had to take a nose dive. Like when you're cycling down a hill. At some point, the plane would have gained enough speed for its wings to carry it again.

When an aircraft flies in icy conditions, it needs more speed to stay aloft. The ice weighs down and significantly alters the shape of the wings, and therefore their lift.

It seems that Renslow, busy with his conversation with Shaw, forgot that he had switched on the de-icing. And that they needed more speed to land smoothly.

When he heard the stall signal, he panicked. And instead of pitching down, he pulled like crazy on his control column to regain altitude. So, on the contrary, he forced the plane to straighten its nose. In doing so, it lost speed and its wings could no longer carry it. And the plane fell like an iron. An iron carrying fifty people.

The impact of AI on driver alertness: a fatal accident perfectly avoided

The same misadventure happened a few weeks later to a colleague of Shaw and Renslow. Except that she was not in the middle of a conversation with her co-pilot and was landing in manual mode. When the stall signals were triggered, she immediately understood and performed the correct manoeuvre: she pushed on the control column to regain speed. And saved her life and that of her passengers.

Renslow and his colleague had undergone the same training, both had thousands of flying hours under their belts and both knew the aircraft and the route perfectly, having flown it many times.

But the colleague was aware of her situation. 

Excessive reliance on automatic systems leads to a kind of torpor, a drowsiness of vigilance. 

Tweet

Most humans are unable to maintain a sufficient level of attention when sitting idle for hours on end. And not just on planes. Even if we don't all have responsibility for hundreds of passengers, we do have an obligation to do a good job.

One of the risks of artificial intelligence is that it numbs our capacity for attention and vigilance.

Overconfidence in automated systems: the key to the mystery

The key to this mystery lies in an experiment led by Raja Parasumaran, an expert in neuro-ergonomics, a scientific discipline that examines the brain mechanisms underlying human behaviour. He focused particularly on interactions between humans and machines.

Parasumaran defines automation as ‘a device or system that performs - partially or completely - a function that could or would have been performed by a human operator’.

To test pilots' vigilance in relation to their confidence in the automatic system, Parasumaran developed a unique and highly revealing experiment. The pilots were placed in front of three screens showing :

• the flight of an aircraft as pilots perceive it when they are at the controls
• the fuel gauge
• engine status information

What Parasumaran wanted to measure, was the impact of the degree of confidence these pilots had in their autopilot system.

He divided the pilots into three groups. The pilots then spent two hours ‘flying’ along a given trajectory. As in a real flight, they had to maintain their heading while regularly checking the fuel gauge and engine data. Two lights, one red and one green, told them whether the gauge was working normally. If a gauge was faulty, the autopilot fixed the problem and indicated by relighting the green light that everything was working normally again.

But Parasumaran and his team made a few ‘adjustments’ to the system. The computer wouldn't fix the problem with the gauge, and the lamp would stay green. Constantly. The pilots would have to read the engine data to see that a gauge wasn't working properly. They would then have to press a button to restore this function manually. To measure the impact of confidence in the system, the team introduced two variants: in one case, the system would be 86.5% reliable; in the other, it would only be 56.5% reliable, i.e. just over half the time.

The pilots who had to deal with the faulty system one time out of two remained vigilant. They pressed the button whenever necessary to repair the gauge. Those who ‘flew’ with a reliable system were very quickly oblivious to the real state of their engines. So it was confidence in the reliability of the automatic system that created the difference in vigilance between pilots with similar skills.

Systematic reliance on automatic systems has created a state of drowsiness that is conducive to accidents.

Quel est l’impact de l’IA sur la vigilance des pilotes ? Des choix stratégiques s’imposent

The systems we have mentioned in this article are automation models. They replace pilots for certain tasks. Artificial intelligence systems go further: they can take decisions in place of pilots. What is the impact of AI on the vigilance of airline pilots? Should we expect similar air accidents in the future?

AIs sometimes make aberrant decisions. Like the one in the image below, where the teacher has three arms instead of two. This is ridiculous and rather funny. But in aviation safety, a mistake, however small, can have catastrophic, even fatal, consequences.

Dario Amodei, un des fondateurs d’Anthropic, l’entreprise qui produit Claude, un des modèles d’IA générative les plus performants, insiste sur les dangers d’une IA incontrôlée. Et il réclame une réglementation en la matière pour 2025. L’intelligence artificielle dispose d’un potentiel énorme, dont nous n’apercevons encore que les prémisses. Mais, elle cache aussi, dans le cœur de ses modèles, des surprises qui pourraient se révéler moins réjouissantes qu’une image anatomiquement fausse.

Did you enjoy this article? Subscribe to our newsletter. It's free, and you won't miss any of our publications!