Eye tracking metrics for workload
estimation in flight deck operations
Kyle Kent Edward Ellis
University
of Iowa
Reference:
Operator Performance Laboratory, NASA, Smarteye
Inc.
University of Iowa
Defence Research and Development Canada
Scientific
Report
The eye movements of aircrew during flight
have been a topic of interest to military and civilian flights. Studies in
flight simulators and real aircraft have used eye movements as a window onto
operators’ processing of information from cockpit instruments and displays.
The
interaction between the operator and the aircraft interface
Analysis of
operator state in different testing scenarios in flight deck operations.
Determining
workload fixation duration and blink rate
Current avionics
are not aware of pilot real-time
capabilities and limitations resulting from varying workload levels.
The concept of
the intelligent flight deck is
currently being pursued by the National Aeronautics and Space Administration (NASA),
with specific interest in characterizing operator state in flight deck operations.
The goal is to use operator workload and overall cognitive state effectively to
optimize the flight deck interface.
Basic visual search is comprised of two
components:
Fixations and
Saccadic movements.
A fixation is a set of look-points or a
series of eye gaze vector data points that is focused on a stationary target in
the person’s visual field (Applied Science Laboratories, 2007).
A fixation is the duration of time for which
an individual is visually collecting and interpreting whatever information is
available within the foveal range of the eye.
When the fixation is made on a point close to
the individual, such as on a flight deck, visual angle decreases significantly
depending on the distance from the eye. The central 1.5 degrees of visual field
have a visual resolution many times greater than that of the peripheral vision
(Rao, Zelinsky, Hayhoe, & Ballard, 1997).
This region of
resolution is the only field in which the eye is capable of interpreting fine
resolution information, such as words in a book.
Converting the
reading information analogy to that of heads
down displays on a flight deck, the highest resolution necessary of any eye
tracker needs to be at least within two degrees visual angle (Rayner &
Bertera, 1979). Various components of eye fixations are the duration, the frequency,
and the location in which they are made.
The eye movement
from one fixation to the next is called a saccade.
A saccade connects one fixation to the
next, and can be measured in terms of radial degrees.
Different
components to a saccade include the length of the saccade (visual angle), the speed
of the saccade in degrees per second, and the direction of the saccade.
When reading,
the eye makes rapid movements, as many as four to five per second, moving from
one fixation to the next, focusing on a few words each time (Rayner &
Bertera, 1979).
The eye does not transmit
visual signals to the brain when making a saccade.
A saccade is
made each time information is obtained from one fixation and another fixation
is necessary to observe further information elsewhere.
Combining
saccadic movements and their associated fixations a scan pattern or scan path
emerges.
The quality of the eye tracking
itself is not affected by differences among pilots’ Experience.
Experienced
pilots will typically be more comfortable while performing a flight task with a
basic knowledge of what they need to look at to obtain the information they
need. This increases the efficiency of their eye behavior, resulting in a difference
in eye tracking metrics in contrast to a novice pilot.
Pupil color greatly impacts the
quality of eye tracking for many eye trackers.
High precision
eye trackers require a sharp contrast between the pupil and the iris. Bright pupil
systems require direct infrared reflection off of the retina therefore,
subjects with blue eyes are often times easier to track. This is due to blue eyes
containing less IRreflective melanin in the iris. In contrast to this, brown or
hazel eyes are usually ideal for eye tracking systems that utilize a dark pupil
contrast. (Boyce, Ross, Monaco, Hornak, & Xin, 2006); (Wang, Lin, Liu,
& Kang, 2005).
Pilots who may
be sleep deprived also pose another form of problem. Eyelid closure can become
an issue when the eyelid itself begins to cover portions of the pupil.
Corrective
lenses, such as glasses, pose reflection issues that pose as the biggest threat
to eye tracking quality. Lenses posing the largest problem are lenses with hard
edged bi- or tri-focal lenses due to distortion of the eye image as seen from
the perspective of the eye tracking cameras. Distortions typically occur due to
lens shape, causing problems with systems using corneal reflection, bright
retinal reflection, dark pupil circle, limbus or iris features, etc.
Soft contact
lenses typically do not cause problems however, hard contacts can cause edge problems
in bright pupil systems typically caused by dirt or dust trapped beneath the
lens.
Typically single
vision corrective eye glasses do not cause problems unless they have an
anti-reflective coating.
Lenses with
curved front surfaces will often times because of problems caused by reflecting
the infrared source back into the camera.
Three theories of eye tracking data
analysis (Jacob & Karn,
2003):
1. Top-down based on cognitive theory:
“Longer fixations on a control element in the interface reflect a participant’s
difficulty interpreting the proper use of that control.”
Top-down based on a design hypothesis:
“People will look at a banner advertisement on a web page more frequently if we
place it lower on the page.”
Bottom-up: “Participants are taking much
longer than anticipated making selection on this screen. We wonder where they
are looking.”
Post-run analysis can lead to indications of why a subject,
in this case a pilot, would spend more time on the attitude indicator than the
airspeed indicator, both of which are of high importance. The answers to such
questions can lead to further understanding of pilot workload, and what is
consuming their cognitive capacity and why.
Eye tracking data:
• Average Dwell Time
– The total time spent looking at an instrument divided by the total number of
individual dwells on that instrument.
• Dwell percentage
– Dwell time on a particular instrument as a percent of total scanning time.
• Dwell Time –
The time spent looking within the boundary of an instrument.
• Fixation – A
series of continuous look points which stay within a pre-defined radius of
visual degrees.
• Fixations per dwell
– The number of individual fixations during an instrument dwell.
• Glance – A “subconscious” (i.e., non-recallable)
verification of information with a duration histogram peaking at 0.1 seconds.
(also referred to as an “orphan”)
• Lookpoint – The
current coordinates of where the pilot is looking, frequency of data points
depending on the eye tracking system used.
• One-way transition
– The sum of all transitions from one instrument to another (one direction
only) in a specified instrument pair.
• Out of track – A state in which the eye
tracking system cannot determine where the pilot is looking, such as during a blink
or when the subject’s head movement has exceeded the tracking capabilities of
the system setup.
• Saccade – The movements of the eye from one
fixation to the next. Also considered to be the spatial change in fixations.
• Scan – Eye movement technique used to accomplish
a given task. Measures used to quantify a scan include (but are not limited to)
transitions, dwell percentages, and average dwell times.
• Transition –
The change of a dwell from one instrument to another.
• Transition rate
– The number of transitions per second.
• Two-way transition
– The sum of all transitions between an instrument pair, regardless of
direction of the transition.
(Harris, Glover, & Spady, 1986)
Fixations
Eye fixations are defined as “a relatively stable eye-in-head
position within some threshold of dispersion (~2 deg) over some minimum
duration (200ms), and with a velocity threshold of 15-100 degrees per second”
(Jacob & Karn, 2003). Several studies have been conducted utilizing eye
fixation measures. The total number of fixations has been observed to correlate
negatively with efficiency; however, efficiency is seen to correlate negatively
with workload (Goldberg & Kotval, 1998).
Fixation
frequency that
shows a positive correlation to subject workload similar to fixation total.
Fixation frequency has shown to indicate more effortful search, indicating poor
performance accuracy and longer search times in memory tasks (Van Orden,
Limbert, Makeig, & Jung, 2001).
Fixation
duration, including the
mean and maximum duration, indicates increased workload in flight.
Longer fixations are indicative of increases in cognitive
processing loads during a period of time (Callan, 1998).
Gaze
Very similar to the fixation metric, gaze analyzes the
grouping of fixations within a single region of interest.
Analysis of the gaze metric,
including gaze rate (# of gazes / minute) on each area of interest, gaze
duration mean and gaze percentage (proportion of time) in each area of interest
for 40 pilots flying an aircraft landing approach (Fitts, Jones, & Milton, 1950).
Gaze metrics focus more on the area of interest and what it represents, not
only the measure of a fixation in any given region of space.
Saccadic Movement
Measures of saccadic movement are often times neglected in
usability research initiatives because many of its close relation to fixations
measures, which are easier to examine are used instead.
The length of the
saccade, as well as the speed of which the saccade is made are both very
easily calculated measures, simply calculating the distance from one fixation
to the next in an ordered pair.
The frequency of longer length saccadic movements
could indicate a correlation of decreased efficiency, and potentially an
increase in perceived workload.
Scan-Path
Several research studies have been conducted that analyze
scan-path as it relates to efficiency, workload, usability, effectiveness,
effort, saliency, and other forms of human factors.
Scan-path is often looked at as the measurable
window that depicts how a subject uses their visual sensory perception to
complete any task at hand, carrying with it also the distractions and other
important artifacts that are included that add or detract to an individual’s
intention of completing that task. Scan-path
analysis measures the transitions between fixations, including measures of
transitions between areas of interest (link-analysis) as a quantifiable
measure.
It is particularly useful in bottom-up analysis approaches
that seek to identify where someone is looking and why, in an attempt to understand
the cognitive background to an individual’s eye tracking behavior.
From a top-down
approach scan-path is seemingly less useful.
Blink Rate
Research using air
traffic controllers in high and low workload situations suggests that increases in
workload negatively correlate with blink rate (Brookings,
Wilson, & Swain, 1996); (Wilson, Purvis, Skelly, Fullenkamp, & Davis, 1987).
The fundamental
belief being that workload is higher requiring more focused attention and a
general increase in visual load. Blinks therefore occur less often so it is less
likely to miss critical information. This requires the amount of time the eye
is collecting information to be increased thereby resulting in a decreased
blink rate (Brookings, Wilson, & Swain, 1996).
Fixation Maps
Fixation mapping is the “analysis of eye-movement traces”
of a given scene.
Example of Fixation Map on Standard 737 EFIS PFD
Example of Fixation Map on Standard 737 EFIS PFD 2
When analyzing fixation maps it is not the analysis of
fixation order, but the location of the fixation that is important.
BOEING 737-800 FLIGHT DECK
EYE TRACKING RESEARCH STUDY
A complex flight task that will yield a wide variation in
relative physical and cognitive workload levels.
This will be used to observe
pilot’s eye movement behavior under these varying conditions. From this it
will show that eye
movement measures are affected by task loading.
Smarteye
Eye Tracking System
The 737 flight deck
utilized a 3 camera system to achieve
the visual angle of eye tracking necessary to capture the test pilots’ gaze
across the flight deck areas of interest.
To obtain quantitative eye tracking data, a Smarteye eye
tracking system was installed and optimized inside the OPL’s 737-800 simulator.
The Smarteye eye tracker is a remote eye tracking system that uses facial
recognition to calculate the position of defined points on a subjects head
relative to the calibrated position of 2 or more cameras.
The camera’s use the facial features to locate the corners
of each of the subject’s eyes and digitally zooms to enhance the image of the
eye.
To calculate eye gaze vectors from the head origin, infrared
led’s project infrared light onto the pilots face, illuminating the pilots face
as well as creating two ocular reflections; a static corneal reflection and a
moving pupil reflection that moves in conjunction with eye movements. By
triangulating the angular difference between the corneal reflection and pupil
reflection, the Smarteye eye tracking system can create a vector between the
two points to create an eye gaze vector originating from the corneal reflection
at the center of the subject’s eyes.
Practicing in simulator B737-800
KORD
Runway 9R Approach to Land Task
The main objective to
the design of the experiment was to develop a series of flight scenarios that
utilized the same flight task but could demand several different levels of workload from the pilot. To
accomplish this, a single approach task to Chicago O’Hare International airport
was chosen.
The initial point (IP) started the flight test simulation
southwest of the DPA VOR at 10,000 feet. Pilots were contacted by Chicago
center and instructed to descend to 7000 feet and maintain 200 knots on course
to DPA. Approximately 5 NM out from DPA pilots were instructed by Chicago
center to contact Chicago approach at radio frequency
119.0. Once contact with Chicago approach was established,
pilots were instructed to descend to 6000 feet, continue to waypoint Burke and
establish the aircraft on the localizer cleared for runway 9R. Pilots then
proceeded to follow the flight plan to waypoints Pratt and Carle. Approximately
1 NM from waypoint Deana pilots were instructed by O’Hare approach to contact
O’Hare tower at radio frequency 121.75. With the aircraft inside the outer
marker of O’Hare, pilots were cleared to land by the tower.
The flight test engineer in the right seat of the flight deck
was responsible for making calls to decision height at 1000, 500 and 200 feet
to minimums. Upon reaching decision height pilots were expected to make a land
or go around call and execute the procedure depending on visual acquisition of
the REILs.
The KORD runway 9R approach task includes five waypoints
with designated speeds and altitudes pilots were instructed to establish upon
reaching that given waypoint:
• DPA – 200 knots at 7000 MSL
• Burke – 180 knots at 6000 MSL
• Pratt – 165 knots at 5000 MSL
• Carle – 165 knots at 4000 MSL
• Deana – 145 knots at 2300 MSL
Pilots were instructed to maintain a sterile
cockpit, keeping verbal communication to a minimum during each test
run, speaking only during radio calls and workload callbacks. A checklist was
provided to the pilots listing each waypoint and the speeds and altitudes they
are to establish upon arrival at each waypoint. Also provided for each waypoint
were suggested flap positions, MCP engage commands, and gear down instructions.
The KORD ILS runway 9R approach plate was also provided to
the pilots as a standard in flight reference of the approach task. The ILS
approach plates give information to pilots in a familiar form to pilots with an
IFR and above license. It lists the typical clearance altitudes to be expected
and distances between approach waypoints, as well as radio frequencies of
O’Hare approach and O’Hare towers. This information could be used by the pilot
to pre-program the radios if so desired to make the flight tasks easier when
asked to transfer radio contact to approach or tower. The approach plate was available
for all test runs and was the basis for programming the flight plan into the
FMS.
Test Conditions
Two methods to drive workload to show high-low workload
contrasts were implemented:
1. Visibility condition – CAT II and CAT III o Land or Go
Around condition
2. Level of automation
®
Full
Autopilot and Auto-throttle
®
FD
guidance and Auto-throttle
®
Manual
approach with localizer course and glide slope guidance only
For the visibility condition, outside visuals were
controlled to be set to CAT II visibility, with no greater than 0.3 NM
visibility, or set to CAT III, with no greater than 0.1 NM visibility. The
threshold of visibility between the two visibility conditions forced the pilot
to make a land-no land decision at decision height at 200 feet AGL. Upon reaching
200 feet AGL, federal air regulations (FARs) state that the pilot must be able
to see the runway end indicator lights (REILs) to continue to land. If the
pilot cannot see the REILs at 200 feet above the runway, the pilot must execute
a go-around. If the pilot is able to see the REILs at 200 feet AGL, then the
pilot was to proceed to 100 feet AGL where they are required by FARs to make
visual contact with the end of the runway to continue to land. The variance in
visibility conditions made no impact on the 100 foot AGL decision height. The
difficulty for the pilot is found in the time for which the decision to land
must be made and to maintain the proper flight path to the runway with no
outside visuals obtaining guidance strictly from the HDDs.
Several times pilots would find themselves off-course due to
the high mental demand required by several systems at once, such as thrust
level, attitude adjustment and radio communication. Diverging oscillations in
recovering the flight path intercept was very typical across pilots in the
manual condition, ultimately driving the pilots’ workload to higher levels on
the Bedford workload scale.
Workload
Of course, pilot errors and performance
decrements can result from causes beyond a loss of SA. (Lefrancois, Matton,
Gourinat, Peysakhovich, and Causse(2016). Named automation addiction due to pressure and fatigue as a factor
leading to errors in monitoring flight instruments.
They examined 20 pilots who were instructed to
land an Airbus A320 manually in a flight simulator.
A quarter of
the pilots
was unable to stabilize the aircraft and made the decision to go around. The authors concluded that gaze patterns for these pilots were suboptimal in comparison to those of the
pilots who stabilized and landed the aircraft most precisely.
They did not sufficiently scan primary flight
instruments to fly the approach. The authors assumed that these pilots were not
sufficiently trained to fly manual approaches.
In a closely related experiment, compared pilot groups who inappropriately
flew on during an ill-advised approach and those who appropriately decided to
go around. They observed differences in visual allocation of attention between
the two groups, as well as differences between the pilot flying and the copilot
navigating. The latter differences were quantified and reflected both in where
the two types looked inside and outside the cockpit, and also in the
qualitative style of eye movements. (Dehais, Behrend, Peysakhovich, Causse, and
Wickens(2017).
