quarta-feira, 5 de janeiro de 2022

EYE TRACKING IN FLIGHT DECK OPERATIONS – UPCOMING FLIGHT SAFETY DEVICE: COCKPIT MONITORING THE PILOT’S EYES

 


ABRIDGE

Eye tracking metrics for workload estimation in flight deck operations

Kyle Kent Edward Ellis

University of Iowa

Reference:

Operator Performance Laboratory, NASA, Smarteye Inc.

University of Iowa

Defence Research and Development Canada

Scientific Report

 

The eye movements of aircrew during flight have been a topic of interest to military and civilian flights. Studies in flight simulators and real aircraft have used eye movements as a window onto operators’ processing of information from cockpit instruments and displays.

The interaction between the operator and the aircraft interface

Analysis of operator state in different testing scenarios in flight deck operations.

Determining workload fixation duration and blink rate

Current avionics are not aware of pilot real-time capabilities and limitations resulting from varying workload levels.

The concept of the intelligent flight deck is currently being pursued by the National Aeronautics and Space Administration (NASA), with specific interest in characterizing operator state in flight deck operations. The goal is to use operator workload and overall cognitive state effectively to optimize the flight deck interface.

Basic visual search is comprised of two components:

Fixations and

Saccadic movements.

A fixation is a set of look-points or a series of eye gaze vector data points that is focused on a stationary target in the person’s visual field (Applied Science Laboratories, 2007).

A fixation is the duration of time for which an individual is visually collecting and interpreting whatever information is available within the foveal range of the eye.

When the fixation is made on a point close to the individual, such as on a flight deck, visual angle decreases significantly depending on the distance from the eye. The central 1.5 degrees of visual field have a visual resolution many times greater than that of the peripheral vision (Rao, Zelinsky, Hayhoe, & Ballard, 1997).

This region of resolution is the only field in which the eye is capable of interpreting fine resolution information, such as words in a book.

Converting the reading information analogy to that of heads down displays on a flight deck, the highest resolution necessary of any eye tracker needs to be at least within two degrees visual angle (Rayner & Bertera, 1979). Various components of eye fixations are the duration, the frequency, and the location in which they are made.

The eye movement from one fixation to the next is called a saccade.

A saccade connects one fixation to the next, and can be measured in terms of radial degrees.


Different components to a saccade include the length of the saccade (visual angle), the speed of the saccade in degrees per second, and the direction of the saccade.

When reading, the eye makes rapid movements, as many as four to five per second, moving from one fixation to the next, focusing on a few words each time (Rayner & Bertera, 1979).

The eye does not transmit visual signals to the brain when making a saccade.

A saccade is made each time information is obtained from one fixation and another fixation is necessary to observe further information elsewhere.

Combining saccadic movements and their associated fixations a scan pattern or scan path emerges.

The quality of the eye tracking itself is not affected by differences among pilots’ Experience.

Experienced pilots will typically be more comfortable while performing a flight task with a basic knowledge of what they need to look at to obtain the information they need. This increases the efficiency of their eye behavior, resulting in a difference in eye tracking metrics in contrast to a novice pilot.

Pupil color greatly impacts the quality of eye tracking for many eye trackers.

High precision eye trackers require a sharp contrast between the pupil and the iris. Bright pupil systems require direct infrared reflection off of the retina therefore, subjects with blue eyes are often times easier to track. This is due to blue eyes containing less IRreflective melanin in the iris. In contrast to this, brown or hazel eyes are usually ideal for eye tracking systems that utilize a dark pupil contrast. (Boyce, Ross, Monaco, Hornak, & Xin, 2006); (Wang, Lin, Liu, & Kang, 2005).

Pilots who may be sleep deprived also pose another form of problem. Eyelid closure can become an issue when the eyelid itself begins to cover portions of the pupil.

Corrective lenses, such as glasses, pose reflection issues that pose as the biggest threat to eye tracking quality. Lenses posing the largest problem are lenses with hard edged bi- or tri-focal lenses due to distortion of the eye image as seen from the perspective of the eye tracking cameras. Distortions typically occur due to lens shape, causing problems with systems using corneal reflection, bright retinal reflection, dark pupil circle, limbus or iris features, etc.

Soft contact lenses typically do not cause problems however, hard contacts can cause edge problems in bright pupil systems typically caused by dirt or dust trapped beneath the lens.

Typically single vision corrective eye glasses do not cause problems unless they have an anti-reflective coating.

Lenses with curved front surfaces will often times because of problems caused by reflecting the infrared source back into the camera.

Three theories of eye tracking data analysis (Jacob & Karn, 2003):

1.  Top-down based on cognitive theory: “Longer fixations on a control element in the interface reflect a participant’s difficulty interpreting the proper use of that control.”

Top-down based on a design hypothesis: “People will look at a banner advertisement on a web page more frequently if we place it lower on the page.”

Bottom-up: “Participants are taking much longer than anticipated making selection on this screen. We wonder where they are looking.”

Post-run analysis can lead to indications of why a subject, in this case a pilot, would spend more time on the attitude indicator than the airspeed indicator, both of which are of high importance. The answers to such questions can lead to further understanding of pilot workload, and what is consuming their cognitive capacity and why.

Eye tracking data:

Average Dwell Time – The total time spent looking at an instrument divided by the total number of individual dwells on that instrument.

Dwell percentage – Dwell time on a particular instrument as a percent of total scanning time.

Dwell Time – The time spent looking within the boundary of an instrument.

Fixation – A series of continuous look points which stay within a pre-defined radius of visual degrees.

Fixations per dwell – The number of individual fixations during an instrument dwell.

Glance – A “subconscious” (i.e., non-recallable) verification of information with a duration histogram peaking at 0.1 seconds. (also referred to as an “orphan”)

Lookpoint – The current coordinates of where the pilot is looking, frequency of data points depending on the eye tracking system used.

One-way transition – The sum of all transitions from one instrument to another (one direction only) in a specified instrument pair.

Out of track – A state in which the eye tracking system cannot determine where the pilot is looking, such as during a blink or when  the subject’s head movement has exceeded the tracking capabilities of the system setup.

Saccade – The movements of the eye from one fixation to the next. Also considered to be the spatial change in fixations.

Scan – Eye movement technique used to accomplish a given task. Measures used to quantify a scan include (but are not limited to) transitions, dwell percentages, and average dwell times.

Transition – The change of a dwell from one instrument to another.

Transition rate – The number of transitions per second.

Two-way transition – The sum of all transitions between an instrument pair, regardless of direction of the transition.

(Harris, Glover, & Spady, 1986)

Fixations

Eye fixations are defined as “a relatively stable eye-in-head position within some threshold of dispersion (~2 deg) over some minimum duration (200ms), and with a velocity threshold of 15-100 degrees per second” (Jacob & Karn, 2003). Several studies have been conducted utilizing eye fixation measures. The total number of fixations has been observed to correlate negatively with efficiency; however, efficiency is seen to correlate negatively with workload (Goldberg & Kotval, 1998).

Fixation frequency that shows a positive correlation to subject workload similar to fixation total. Fixation frequency has shown to indicate more effortful search, indicating poor performance accuracy and longer search times in memory tasks (Van Orden, Limbert, Makeig, & Jung, 2001).

Fixation duration, including the mean and maximum duration, indicates increased workload in flight.

Longer fixations are indicative of increases in cognitive processing loads during a period of time (Callan, 1998).

Gaze

Very similar to the fixation metric, gaze analyzes the grouping of fixations within a single region of interest.

Analysis of the gaze metric, including gaze rate (# of gazes / minute) on each area of interest, gaze duration mean and gaze percentage (proportion of time) in each area of interest for 40 pilots flying an  aircraft landing approach (Fitts, Jones, & Milton, 1950). Gaze metrics focus more on the area of interest and what it represents, not only the measure of a fixation in any given region of space.

Saccadic Movement

Measures of saccadic movement are often times neglected in usability research initiatives because many of its close relation to fixations measures, which are easier to examine are used instead.

The length of the saccade, as well as the speed of which the saccade is made are both very easily calculated measures, simply calculating the distance from one fixation to the next in an ordered pair.

The frequency of longer length saccadic movements could indicate a correlation of decreased efficiency, and potentially an increase in perceived workload.

Scan-Path

Several research studies have been conducted that analyze scan-path as it relates to efficiency, workload, usability, effectiveness, effort, saliency, and other forms of human factors.

Scan-path is often looked at as the measurable window that depicts how a subject uses their visual sensory perception to complete any task at hand, carrying with it also the distractions and other important artifacts that are included that add or detract to an individual’s intention of completing that task. Scan-path analysis measures the transitions between fixations, including measures of transitions between areas of interest (link-analysis) as a quantifiable measure.

It is particularly useful in bottom-up analysis approaches that seek to identify where someone is looking and why, in an attempt to understand the cognitive background to an individual’s eye tracking behavior.

From a top-down approach scan-path is seemingly less useful.

Blink Rate

Research using air traffic controllers in high and low workload situations suggests that increases in workload negatively correlate with blink rate (Brookings, Wilson, & Swain, 1996); (Wilson, Purvis, Skelly, Fullenkamp, & Davis, 1987).

The fundamental belief being that workload is higher requiring more focused attention and a general increase in visual load. Blinks therefore occur less often so it is less likely to miss critical information. This requires the amount of time the eye is collecting  information to be increased thereby resulting in a decreased blink rate (Brookings, Wilson, & Swain, 1996).

Fixation Maps

Fixation mapping is the “analysis of eye-movement traces” of a given scene.

Example of Fixation Map on Standard 737 EFIS PFD

Example of Fixation Map on Standard 737 EFIS PFD 2

When analyzing fixation maps it is not the analysis of fixation order, but the location of the fixation that is important.

BOEING 737-800 FLIGHT DECK EYE TRACKING RESEARCH STUDY

A complex flight task that will yield a wide variation in relative physical and cognitive workload levels.

This will be used to observe pilot’s eye movement behavior under these varying conditions. From this it will show that eye movement measures are affected by task loading.


Smarteye Eye Tracking System

The 737 flight deck utilized a 3 camera system to achieve the visual angle of eye tracking necessary to capture the test pilots’ gaze across the flight deck areas of interest.


To obtain quantitative eye tracking data, a Smarteye eye tracking system was installed and optimized inside the OPL’s 737-800 simulator. The Smarteye eye tracker is a remote eye tracking system that uses facial recognition to calculate the position of defined points on a subjects head relative to the calibrated position of 2 or more cameras.

The camera’s use the facial features to locate the corners of each of the subject’s eyes and digitally zooms to enhance the image of the eye.

To calculate eye gaze vectors from the head origin, infrared led’s project infrared light onto the pilots face, illuminating the pilots face as well as creating two ocular reflections; a static corneal reflection and a moving pupil reflection that moves in conjunction with eye movements. By triangulating the angular difference between the corneal reflection and pupil reflection, the Smarteye eye tracking system can create a vector between the two points to create an eye gaze vector originating from the corneal reflection at the center of the subject’s eyes.

Practicing in simulator B737-800

KORD Runway 9R Approach to Land Task

 

The main objective to the design of the experiment was to develop a series of flight scenarios that utilized the same flight task but could demand several different levels of workload from the pilot. To accomplish this, a single approach task to Chicago O’Hare International airport was chosen.

 The initial point (IP) started the flight test simulation southwest of the DPA VOR at 10,000 feet. Pilots were contacted by Chicago center and instructed to descend to 7000 feet and maintain 200 knots on course to DPA. Approximately 5 NM out from DPA pilots were instructed by Chicago center to contact Chicago approach at radio frequency

119.0. Once contact with Chicago approach was established, pilots were instructed to descend to 6000 feet, continue to waypoint Burke and establish the aircraft on the localizer cleared for runway 9R. Pilots then proceeded to follow the flight plan to waypoints Pratt and Carle. Approximately 1 NM from waypoint Deana pilots were instructed by O’Hare approach to contact O’Hare tower at radio frequency 121.75. With the aircraft inside the outer marker of O’Hare, pilots were cleared to land by the tower.

The flight test engineer in the right seat of the flight deck was responsible for making calls to decision height at 1000, 500 and 200 feet to minimums. Upon reaching decision height pilots were expected to make a land or go around call and execute the procedure depending on visual acquisition of the REILs.

The KORD runway 9R approach task includes five waypoints with designated speeds and altitudes pilots were instructed to establish upon reaching that given waypoint:

• DPA – 200 knots at 7000 MSL

• Burke – 180 knots at 6000 MSL

• Pratt – 165 knots at 5000 MSL

• Carle – 165 knots at 4000 MSL

• Deana – 145 knots at 2300 MSL


Pilots were instructed to maintain a sterile cockpit, keeping verbal communication to a minimum during each test run, speaking only during radio calls and workload callbacks. A checklist was provided to the pilots listing each waypoint and the speeds and altitudes they are to establish upon arrival at each waypoint. Also provided for each waypoint were suggested flap positions, MCP engage commands, and gear down instructions.



The KORD ILS runway 9R approach plate was also provided to the pilots as a standard in flight reference of the approach task. The ILS approach plates give information to pilots in a familiar form to pilots with an IFR and above license. It lists the typical clearance altitudes to be expected and distances between approach waypoints, as well as radio frequencies of O’Hare approach and O’Hare towers. This information could be used by the pilot to pre-program the radios if so desired to make the flight tasks easier when asked to transfer radio contact to approach or tower. The approach plate was available for all test runs and was the basis for programming the flight plan into the FMS.




Test Conditions

Two methods to drive workload to show high-low workload contrasts were implemented:

 1. Visibility condition – CAT II and CAT III o Land or Go Around condition

 2. Level of automation

®     Full Autopilot and Auto-throttle

®     FD guidance and Auto-throttle

®     Manual approach with localizer course and glide slope guidance only

 For the visibility condition, outside visuals were controlled to be set to CAT II visibility, with no greater than 0.3 NM visibility, or set to CAT III, with no greater than 0.1 NM visibility. The threshold of visibility between the two visibility conditions forced the pilot to make a land-no land decision at decision height at 200 feet AGL. Upon reaching 200 feet AGL, federal air regulations (FARs) state that the pilot must be able to see the runway end indicator lights (REILs) to continue to land. If the pilot cannot see the REILs at 200 feet above the runway, the pilot must execute a go-around. If the pilot is able to see the REILs at 200 feet AGL, then the pilot was to proceed to 100 feet AGL where they are required by FARs to make visual contact with the end of the runway to continue to land. The variance in visibility conditions made no impact on the 100 foot AGL decision height. The difficulty for the pilot is found in the time for which the decision to land must be made and to maintain the proper flight path to the runway with no outside visuals obtaining guidance strictly from the HDDs.

Several times pilots would find themselves off-course due to the high mental demand required by several systems at once, such as thrust level, attitude adjustment and radio communication. Diverging oscillations in recovering the flight path intercept was very typical across pilots in the manual condition, ultimately driving the pilots’ workload to higher levels on the Bedford workload scale.









Workload

Of course, pilot errors and performance decrements can result from causes beyond a loss of SA. (Lefrancois, Matton, Gourinat, Peysakhovich, and Causse(2016). Named automation addiction due to pressure and fatigue as a factor leading to errors in monitoring flight instruments.

They examined 20 pilots who were instructed to land an Airbus A320 manually in a flight simulator.

 A quarter of the pilots was unable to stabilize the aircraft and made the decision to go around. The authors concluded that gaze patterns for these pilots were suboptimal in comparison to those of the pilots who stabilized and landed the aircraft most precisely.

 They did not sufficiently scan primary flight instruments to fly the approach. The authors assumed that these pilots were not sufficiently trained to fly manual approaches.

 In a closely related experiment, compared pilot groups who inappropriately flew on during an ill-advised approach and those who appropriately decided to go around. They observed differences in visual allocation of attention between the two groups, as well as differences between the pilot flying and the copilot navigating. The latter differences were quantified and reflected both in where the two types looked inside and outside the cockpit, and also in the qualitative style of eye movements. (Dehais, Behrend, Peysakhovich, Causse, and Wickens(2017).







 



Nenhum comentário: