Paul Phelan/CAIRNS
Ansett Australia, the country's principal domestic carrier, believes it has broken new ground in enhancing quality and standardisation in its flight operations. The source of those gains is in aircrew training: under Ansett's system, it more closely matches individuals' needs.
The airline's training department assembles a database of individual pilots' training and checking assessments to develop specific training programmes for individuals. Then assessment and training decisions are scrutinised to ensure standardisation. Airline officials believe two of the most recent refinements to pilot training and checking systems - flight operations quality assurance (FOQA) and the US-initiated advanced qualification programme (AQP) - have virtually reached the limit of their potential for the same reason: the covenant that they must de-identify information to protect individuals.
For that reason, FOQA, which uses flight data recorder information gathered in day-to-day operations through separately installed quick access recorders (QAR), has so far only been able to pass on statistical data to training departments for that reason.
The system identifies trends in operating practice across fleets of a particular aircraft type, allowing training departments to shape their offerings around the frequency of "exceedances" that the QAR is programmed to record. Exceedances are deviations from an ideal which exceed predetermined limits.
If the system identifies an unusually high frequency of, for example, over-rotation on takeoff, excessive bank angles or flap extension speeds, training will be focused on that trend. While this targets identified needs, can result in pilots receiving extra training in areas where their individual needs may not lie.
AQP is a quality improvement programme that allows airlines to develop and revise all training, both qualification and recurring, based on an analysis of what each pilot must know and accomplish to be proficient, and on feedback from simulator sessions as to how well those pilots are acquiring and maintaining their proficiencies. One immediate advantage is that it can reduce total recurrent training. Again, however, the de-identification of live flying data diminishes its value.
The key to Ansett's AQP derivative, which it calls Proficiency Assessment & Training System (PATS), has been to identify those individual needs for line pilots as well as for training and checking captains. The latter group needs better-defined guidelines to guarantee that each trainer, presented with a particular level of performance of a particular task, will deliver a uniform assessment.
PATS emerged as an initiative in the crisis of confidence resulting from an Ansett Australia Boeing 747-300's landing at Sydney with its nose gear retracted. The crew had not known of the situation, but were warned late on the approach by air traffic control, and decided to land anyway. With a mandate to overhaul the carrier's flight operations and safety systems, Ansett's general manager of operations, Capt Trevor Jensen, developed a strategy that adopted AQP as a system foundation. The next task was to focus on enhancing its benefits.
The plan's backbone was to "-look outside the industry", Jensen says. "This airline is over 60 years old, and aviation is over 100 years old. It's no good going on looking at the way other airlines do it, because if you do that you're looking at their mistakes as well."
Capt Simon Henderson, PATS programme manager, explains how Ansett expects to deliver safety and cost advantages. "In conventional systems, the cyclical training package tests everything, but if a pilot is consistently excellent in one area and consistently requires extra attention in another, that is not the most efficient and effective allocation of resources.
"Even in advanced systems like FOQA, which uses QARs to identify recurrent events to suggest a need for a specific training focus, the events are de-identified before the data are used to review training procedures in a 'systemic' approach. In the USA, the risk of the legal system obtaining AQP data for criminal or civil trials means the data have to be de-identified, and the potential safety advantages of electronic data storage are lost," Henderson says.
"Ansett pilots have long accepted responsibility for their own standards and are comfortable with the recording of personal data."
A prime advantage of AQP/PATS is that the number of annual simulator visits drops from four to two - albeit two-day sessions - with huge savings in crew travel, rostering flexibility, and trainer time.
On Day One, Ansett has adopted the American principle of a "first look", a shortened session that looks strictly at critical manoeuvres, explains Henderson. "We are not assessing individual standards at that point, we're grading the system as to how well it has prepared the pilots to perform those critical manoeuvres. Day Two is more like an individual standard assessment, on a one to five grading scale."
Adoption of that scale provides a sufficient spread of data for the database, says Henderson. "A lower scale doesn't adequately differentiate between a marginal pass and excellent performance; a simple pass/fail is incompatible with PATS principles and the wider goal of continually improving standards within the airline; and in a scale of one to 10, the increments are too small to be clearly defined, it's difficult to score, and there is the risk of losing objective assessment."
"Reason Codes" are developed to explain why an event received the score it did. "Any rating system is only as good as the people performing the assessments," Henderson says. "Providing feedback to all of our raters about how they perform their role in relation to their contemporaries or to the company standard, is a powerful tool for enhancing the reliability of the entire system. In other airlines, rating reliability information is normally de-identified, which removes the ability to arrange targeted intervention training for check captains. At Ansett, we'll have the advantage of knowing who's doing what."
Ansett produces a series of filmed simulator sessions, in which sequences to be rated are separately acted out at the five proficiency levels. Check captains watch a sequence, and individually grade it, also defining their reasons for using the reason codes. "We then review the rating and discuss with them how it relates to the company's goal standard for that rating, why they graded higher or lower, did they see the behaviour which established the grading? If they saw it, what meaning did they ascribe to it? Why did they think it was worth a particular rating?"
That process, called referent rating reliability, compares individual performance to a reference standard. Separately observed is "inter-rater reliability", which detects any trend away from established standards within the check captain group. "Once they've graded a sufficient number of people with varying degrees of proficiency, check pilots see a similar cross section of performance. So we look at the fleet average for distribution of scores, and at the reason codes to describe reasons for a particular rating.
"The system also gives the fleet average for the events themselves, and we can superimpose on all those different readouts, the individual check captain's average. Thus we can give him a picture of how he grades, compared to the rest of the fleet," Henderson says.
"Rating reliability should be a key component of any grading system," and check captain training using all of these principles is scheduled for this month.
Assessment of human factors is another issue Ansett has successfully introduced into line oriented flight training.
Ansett is also installing quick-access recorders (QARs) across its fleet. "Ansett's philosophy is training for the need, so we should be in a position to include actual line QAR data, along with other training and checking assessments to complete the feedback loop and vary training depending on observed and measured performance."
The use of simulator output to verify reason codes and automate the assessment of rater reliability also are part of the plan. "We need a QAR fitted to the simulator, so we can relieve the check captains of some of the task of assessing performance against technical standards," Henderson says.
When that equipment is fitted, they'll be able to look at the crew management issues, which is really what they're there to do."
THE ANSETT SYSTEM:
The Ansett system assesses each task against eight values - four technical and four related to human factors. The reason codes provide a major enhancement of standardisation, because each "rater" is given objective descriptions (which Ansett calls "word pictures") of the various performance levels. The performances assessed are:
TECHNICAL
M - Manipulation
K - Knowledge of systems and procedures
A - Automated systems comprehension and use
E - Execution of procedures
COCKPIT RESOURCE MANAGEMENT
C - Communication
W - Workload management
S - Situational awareness
P - Problem solving and decision making;
To produce valid data, the assessments awarded must be as consistent as possible across the check pilot group, so the marking scheme has been enhanced in an effort to minimise subjectivity in scoring. A pilot is given a 1, 2, 3, 4 or 5 against each of the eight performance measures. Although word pictures are provided for all five scores, the raters are not expected to annotate the reason if awarded a three or four. For workload management, as an example, the rater would use the guidelines:
1 "Unsatisfactory" would be awarded if there was no prioritising of tasks, leading to clear errors of flight management.
2 "Minimum standard" may result in repeats being required. This is described as organisation of tasks in such a way that essential items are barely carried out in the time available.
3 "Satisfactory" requires a debrief only. According to its description, task organisation adequately deals with high priority items.
4 "Very good" defines overall performance to a standard that would not normally require a debrief. The word picture says "all tasks are correctly organised in a manner that makes flight management efficient. Abnormal and emergency situations are quickly resolved to a good outcome."
5 "Excellent"- reflects overall performance of a very high standard. "In-flight tasks are so well organised that challenging aspects of flight management appear easy. Abnormal/emergency situations are resolved with the best possible outcome."
Similar guidelines are published for each of the other seven elements. As data accumulates in the system, the scheme's effectiveness is refined to enhance rater reliability.
Source: Flight International
