Future Proofing: Can risk analysis prevent more catastrophes?

Can analysis of data from past accidents be used to predict and even prevent more catastrophes? A US task force believes being proactive can pay

On its 10-year anniversary, the US Commercial Aviation Safety Team (CAST) has come tantalisingly close to its goal of reducing the commercial aviation fatality risk 80% by 2007. As of September 2006, the accident rate stood at 0.23 fatal accidents per million departures, based on a rolling three-year average - a reduction of 65% from the starting point in 1998.

To further reduce the risks, or to keep the rate steady in the face of traffic that could double or triple in the next 10-20 years, the government-industry team is investigating novel ways to predict problems in advance - a proactive strategy, unlike the reactive process.

Preventative measures

"With a system that's growing rapidly more complex by the day, we can no longer rely primarily on forensic studies of accidents to determine our next steps," said US Federal Aviation Administration administrator Marion Blakey at a recent International Civil Aviation Organisation safety conference. "It's a straightforward approach. The accident occurs, we analyse it, we take steps to prevent its recurrence."

That approach has been the bedrock of CAST since its inception. The team helped make safety gains by analysing accidents and determining what interventions would provide the biggest return on safety for the minimum investment. Of 47 potential enhancements identified over the years, 31 have been implemented and 16 are now under way. Participants include the FAA, NASA, major engine and airframe manufacturers, airlines and pilot unions. The FAA says the airline industry is saving more than $600 million a year because of the work, after spending about $500 million on safety measures over the 10-year life of CAST. Accident costs would include loss of life and aircraft, devaluation of stock, insurance fees and other indirect legal costs.

Now that the "low-hanging fruit is gone" in terms of accident causes, in part due to the efforts of CAST, Blakey says the industry has to move "from an analysis of what has happened to an analysis of what the data shows might happen with a certain degree of probability".

The information behind Blakey's imperative will largely be accumulated through voluntary safety programmes like Flight Operations Quality Assurance (FOQA) and the Aviation Safety Action Partnership (ASAP), efforts that continuously gather data during flights, allowing analysts to discover lurking problems or analyse what happened in an incident and why it happened without naming names. With FOQA, airlines create a database of engineering data related to a flight, including flight performance indicators like airspeed and altitude, as well as analogue measurements of engine parameters and discrete measurements like the position of on/off switches. De-identified summaries are then made available to the FAA in return for immunity from prosecution, providing any errors discovered were unintentional.

Mining the data

ASAP is similar, but provides textual descriptions of what happened from the people involved. By having the two inputs, investigators can ideally reconstruct the mechanics of the incident as well as the crews' actions or immediate thoughts on the situation, elements that can allow investigators to "connect the dots". Fifty-two airlines have FAA-approved ASAP programmes for pilots and a smaller number of airlines have separate ASAP programmes for mechanics, flight attendants and dispatchers. Eighteen airlines have FOQA programmes.

To find "needle in the haystack" risks hidden in reams of FOQA and ASAP data from a large number of airlines, the FAA is looking to NASA for help. The FAA's Margaret Gilligan, who co-chairs CAST along with Hank Krakowski of United Air Lines, says CAST is driving towards two objectives. "The easier one is that when we identify something we want to ask about, we can query a database and see if it's happening elsewhere. Beyond that - and the ultimate for us - will be the ability to run in the background ways to identify things we have not seen yet," she says.

With FAA funding, researchers at the NASA Ames Research Center have been experimenting with advanced data mining software to monitor data currently being provided voluntarily by eight airlines for FOQA and five airlines for ASAP under the auspices of an FAA data-sharing aviation rule-making committee. The "distributed" database now contains more than 1 million flights of FOQA data, increasing at 100,000 flights a month. ASAP reports number 10,000 and are increasing at 2,000 a month, says Irving Statler, NASA's director of Aviation National Archives. "Distributed" in this context means that each airline keeps its own data NASA is allowed to have a one-way server at each participating airline's location. Queries can be made of de-identified reports, but only the results of analysis software can be ported out.

Ames Research Center has since the early 1990s been developing data-mining techniques that can spot indicators of abnormal performance and, in the future, may be able to predict impending problems. A key benefit of the software originally was to reduce the amount of time that subject matter experts at an airline had to spend poring over data. FOQA systems can record as many as 3,000 parameters at varying frequencies during every flight and ASAP reports, being textual, can generate a wide variety of descriptions of similar events.

In partnership with several airlines, the researchers in 2001 developed software called the "Daily Report" that is now used by Alaska Airlines and Delta Air Lines, and will soon be broadly available as a commercial product. The software compares continuous FOQA data from all flights in the past 24h hours to the observed "normal behaviour" of 1,000 previous flights, looking for items that fall outside the "normally observed preponderance of data", says Ashok Srivastava, group lead for the Intelligent Data Understanding group at Ames, developer of the software.

Automatic processes

More FOQA analysis software in development at Ames looks at discrete parameters like on/off switches to flag situations when pilots might be activating switches at unusual times during a flight, indicating a problem. Some early tests of the software have found incidents where pilots had switched on an engine igniter at an atypical time, possibly indicating an engine problem. Other data revealed overuse of speed brakes, indicating the possibility of a high-energy approach.

Srivastava and his team are now developing software that will automate the analysis of ASAP reports - a more difficult proposition since reports are typically brief and often contain acronyms and jargon. The software requires subject matter experts to classify a variety of ASAP reports first, after which researchers can build a "machine learning process" that automatically reviews the reports. Srivastava says the software can now classify 23 categories of the 32 traditional ASAP event types, and will be deployed to "an airline partner" in March. Statler says the larger challenge with ASAP is to "extract reliable information about the factors that contributed to causing the event from the free narratives of the reporters". Statler says the capability is still in its early conceptual stages and "will likely push the state-of-the-art of natural language processing", a method of representing concepts in written documents.

At the request of CAST, Srivastava's team recently completed a study of terrain warning alerts in the FOQA and ASAP distributed database, and from submissions to NASA's Aviation Safety Reporting System, an anonymous reporting system available to all pilots. Gilligan says airlines had been "getting more alerts at airports with certain configurations", and that the results of the study are being used to "figure out what's really going on" and to come up with mitigations.

Linked databases

Two additional studies are now under way, one to look at automatically measuring FOQA and ASAP metrics that evaluate CAST enhancements, and the other to look at problems generated by late runway changes, where air traffic controllers switch runways for a landing aircraft well into the approach.

For similar studies in the future, NASA would ultimately like to be able to link FOQA, ASAP and other databases that relate to a particular incident, including maintenance, to create a better understanding of what happened and why. Without a direct link, investigators have to make an educated guess as to whether an ASAP description of an incident matches the same event as described by data alone in FOQA.

Such linking is the exception rather than the rule at the moment, since airlines generally de-identify FOQA and ASAP reports without first correlating the events, a requirement levied by the pilot unions to protect the identity of the submitters. The practice is part of a broader blanket of protection meant to prevent the general public from identifying employees or airlines linked to a report.

The assurance of anonymity is critical to the free flow of data, hence the success of the FOQA and ASAP programmes. "It's a tight net of confidentiality it makes pilots willing to participate," says Krakowski. "If you want to shut the programme down quickly, get the reports out there."

Source: Flight International