Stewart Penney/FARNBOROUGH
As the cockpit becomes more advanced and more data is received from a greater number of on- and off-board sensors, the role of the pilot is changing fundamentally.
Today, the pilot flies and fights his aircraft with limited off-board input - for instance command and control via datalink and laser designating from another platform. In the future, the pilot will have to manage data from on-board systems - radar, 360° electro-optical and infrared sensors and electronic support measures - and off-board systems, including airborne warning and control, ground surveillance systems and other tactical aircraft - while controlling other assets such as unmanned combat air vehicles (UCAVs).
Dr Helen Dudfield, assistant director cockpit systems engineering and integration at the UK's Defence Evaluation and Research Agency (DERA) says: "We are getting to the point where the pilot is moving from hands-on flying to hands-on flying and information management." She adds: "It is all about the design of the cockpit interface to maximise human and technology performance. There needs to be some balance between the power of the technology and the human."
DERA's work aims to allow the pilot to reach a decision quickly and to manage data beneath the man-machine interface display. Says Dudfield, key questions include the accessible depth of information beneath the interface and, if data fusion is such that access to the raw data is limited, how is the pilot given confidence in the displayed information? An issue affecting pilot confidence is the "automation surprises", when the machine does something unexpected. Dudfield says much can be learned from commercial aerospace, which has found solutions to this problem.
Manned aircraft acting as UCAV motherships cause further problems: the pilot needs situation awareness for his aircraft and a number of other air vehicles, says Dudfield. The problem is accentuated because the situation changes continually, meaning that pilots have to update their situation awareness constantly. During some flight phases, the UCAVs may be in close formation while at others, they could be at a combat separation of 1,500m (4,900ft)-plus. In the former case the "UCAV's view" is similar to that of the manned platform but there is a greater risk of collision. In the latter, the UCAV's view is offset from the pilot and therefore more difficult to visualise.
"This is a challenge," says Dudfield, adding: "We have to address this. We've been looking at information management of the pilot's own world but the world will be bigger with greater responsibilities."
Impact studies
DERA's work includes predicting operational workload and the study of initial designs using a range of simulators and other synthetic environments to assess the impact on aircrew.
Replacing four or five smaller liquid crystal displays with a single large head-down display (HDD) is the basis of the SMITE(Systems Management in Tactical Environments) display and FIST (Fused Imagery Simulated Testbed), a synthetic environment demonstrator used to evaluate cockpit technologies. Dudfield says there are issues with integration and providing a concept that is acceptable to all users. Key to this is the interface and data control logic.
"We know that the more options we give the operator, the better the performance," she says. If the pilot is given options to use information or perform a task in a number of ways, she adds, "we get better systems management." Total freedom is not possible as maintaining safety is crucial, and some level of standardisation must occur, requiring the system to display a given mode at a given time. The operator must understand the system logic, otherwise information will be removed automatically, potentially causing confusion and distracting the pilot from the task at hand.
If the machine and pilot are not working together the man-in-the-loop system will break down, says Dudfield, for any of several reasons: because the system is not transparent, the workload has overloaded the operator, or conversely the crew is under-used and loses concentration. To restore harmonised conditions, a good understanding of the system is needed - through training and experience - and the individual must have the "big picture of the world in their own mind, and be able to communicate it to colleagues, which includes the machine," says Dudfield.
Situation awareness is complicated if a pilot is managing other assets as spatial issues become a major factor. Maintaining a mixture of each asset's location and seeing the world from the viewpoint of another system "is challenging," says Dudfield, adding, "Do you get into the shoes of the other asset or use a symbolic representation?" Technology could, for instance, allow the user to view the world from another asset. This could be presented in a helmet mounted display (HMD) using stereoscopic pictures to provide a three-dimensional (3-D) view.
Interaction levels
FIST team member Darryl Croft says the aim is to produce technologies for around 2015, adding that large flat panel displays are the best way forward. Lockheed Martin, Sextant and Smiths Industries Aerospace are studying the concept, he notes.
Croft says replacing many displays with one creates cognitive and perceptual issues. FIST is used to rapidly prototype systems, for experimental work and to consider technical issues, says Croft. He says an operational system would probably use three surfaces closely tiled together, to give a measure of redundancy.
Three levels of man-machine interaction are available - the touch sensitive screen, hands-on-throttle and stick (HOTAS) controls and direct voice input (DVI). FIST also has a simulated HMD. SMITE can display three outside world views, continues Croft. A moving map, an "inside looking out view" and, in a similarity to many computer games, a chase view where the pilot can see the aircraft in 3-D space.
SMITE's inside looking out view combines satellite navigation with a terrain reference database and fused sensor data. Unnatural colours encourage head-out of the cockpit rather than head-in the cockpit flying. Croft says this prevents "cognitive capture when the pilot is attracted by the fidelity of the HDD". Around the outside of these screen-filling views is an artificial horizon, which uses the ability to absorb information acquired through peripheral vision. This allows the pilot to maintain altitude without the need for a dedicated flight instrumentation screen.
Pilot studies
SMITE keeps "housekeeping functions" such as monitoring fuel state in the background, but either flashes an icon or brings up the status screen if a problem occurs. Sensor displays are similar. The screens are resizable, can be moved around and can be opened in a number of ways, including touching the icon on the screen - similar to touching the keys around a traditional cockpit display - or moving a cursor over the icon.
The effect of the gap between tiled screens has been assessed while on-going studies include combining an HMD with the HDD, which is important as the former are becoming displays rather than simple attitude references. Tracey Milne, a psychologist working on the programme, says studies with test pilots have revealed that, while they like the SMITE and FIST concepts, they have made minor suggestions, such as preferring a mouse on the HOTAS controls to touchscreens.
Dr Ken Catchpole says the eyes can absorb a finite volume of data and 3-D sound allows additional information to be acquired. Layering voices in 3-D within the communications system allows a pilot to distinguish, for instance, his callsign even in heavy radio traffic - in effect similar to an individual picking out his name being said in a crowded room. The system would also allow aural presentation of threats or a wingman's position, eliminating the need to look inside the cockpit at a display. Although the technology is only now becoming viable, DERA has already considered the technique for helicopters, developing a navigation system in which the pilot flies towards the sound.
Dr Don Jarrett of DERA's systems integration department has been studying helmet displays using the VEIL (Virtual Environments Information Laboratory). He says that HMDs could become the sole source of information, fusing the data with an outside view - creating "augmented reality". This leads to alternative forms of control including head and eye pointing as well as improved DVI and novel forms of hand pointing.
Augmented reality
Jarrett says a wide field of view, high quality, full colour display is required. This is aimed primarily at the low-level ground attack role, where the pilot needs to remain head out of the cockpit. An HMD can display the same information as an HDD - the position of friendly assets, threats and targets, and waypoints.
As such a display is binocular, information can be displayed stereoscopically to ease data assimilation. Like the FIST studies, VEIL work is considering the interaction between the HMD and the HDD. The systems could be complimentary or the HMD could replace other displays, says Jarrett. Fielding of directed energy weapons - which would damage a pilot's eyes - could impose the use of a HMD mounted on an opaque visor - eliminating any view outside the helmet. "It would be silly not to have a good HDD, but we are looking at a virtual presentation to allow flying when it would be dangerous to have a view of the outside world," Jarrett says. Such an HMD would need a virtual cockpit so the pilot can "see" which switches to operate.
Techniques such as finger tracking would allow the pilot to operate virtual switches while eye and head tracking could also be used. The robustness of eye pointing under high-g has been tested in a centrifuge, with initial results suggesting that eye-pointing is more accurate than head-pointing and has other advantages. Head and eye pointing will need improvements in head-tracking technology, and Jarrett suggests that the next-generation head-trackers will combine today's electromagnetic and optical systems and add inertial measurement to improve robustness, speed and static accuracy.
Finger tracking has proved to be software intensive and early trials required the pilot's hand and fingers to be held in unnatural positions, says Jarrett. It is a valuable technique that is worth developing, he adds. Notably the DERA team is working with an RAF aircrew glove with an added light-colour strip, rather than a specially designed, and therefore expensive, piece of equipment.
As Dudfield says, "It's all about design. There needs to be a balance between the power of technology and the human's capability. We need to help the pilot reach decisions as quickly as possible."
Source: Flight International
