Pilots’ mental workload prediction based on timeline analysis

2.1Timeline task analysis

In practice, tasks, especially high-mobility military tasks, are complex and usually consist of one or more simple tasks [1]. Meanwhile, the flight information required by the pilot is different under different flight phases. As a result, to accurately determine the mental workload state of the pilot over time, the timeline analysis method is used to decompose the complex tasks and derive the time-variant basic task units and the time attributes of each task unit, including start time t1⁢j, end time t2⁢j, and duration Δ⁢tj (where j refers to the j-th task unit).

2.2Task type correlation coefficient solution

Since different task types have different effects on mental workload, they need to be quantified according to the internal cognitive mechanism of task types [18]. Each task unit contains one or several specific behaviors [15]. The behavior of task segmentation is executed by different parts and actions of the operator [16]. The McCracken-Aldrich Mental Stress Assessment Scale [5] divides behaviors into four channels: visual, auditory, cognitive, and motor. Each channel is further subdivided into different scales of complexity. According to the multi-resource theory [3], behaviors that are not in the same channel can be executed simultaneously. The task type workload value kq⁢j of the q-th decomposition behavior can be quantified. Then, the task type workload value kj is the summary value of Q decomposed behaviors of the j-th task unit.

The task type workload value calculated using the McCracken-Aldrich scale is considered to be overly heavy if more than 8 [5]. Therefore, the task type correlation coefficient Kjl of a task unit is calculated as follows, where m is the total number of task units decomposed by the l-th task. When Kjl=1, the mental workload caused by the l-th task is considered overwhelmingly high.

2.3Multi-factor mental workload prediction

The multi-factor mental workload prediction based on attention resource allocation [17] is triggered by two decisive factors: external task demand and internal resource allocation of mental workload, integrating the information amount Hi, time pressure Ti, visual coding Ci, and attention resource allocation fi.

The average amount of information Hi of different information appearing over time is calculated as follows, wherein the information occurrence probability of the i-th areas of interest (AOI) Pi represents a coefficient related to the information number and information value.

During the information perception, the attentional allocation factor fi is used to indicate the intrinsic mental resources that the operator needs to consume due to the information of a certain AOI. The attention resource occupied by n AOIs is A; then fi is the ratio of the attention resource to the total attention resource for the i-th AOI.

After the information is captured by the visual system and transmitted to the mental procession by the nervous system, the attention resources allocated to the i-th AOI are as follows, where each of these information has a certain probability βi of being noticed in one gaze. Information prominence Si is quantified by the applicability of the highlighted format. Ei indicates the effort that the vision system needs to pay to acquire the i-th AOI, and Vi is the information importance attribute associated with the assignment task.

Ci [17] is the visual coding comprehensive performance eigenvalue, with 0<Ci<0.1. wij is the weight coefficient corresponding to the j-th visual coding in the i-th AOI, which is obtained by the G1 method through expert evaluation. vij is the comprehensive performance eigenvalue of the j-th visual coding obtained by meta-analysis, and ⊙ is the fuzzy weighted average operator.

Related experimental results [2, 5, 17] show that the influence of visual coding on mental workload is less significant than that of information amount and time pressure. Therefore, the square root of visual coding is applied as the coordinated quantitative result in the model.

Considering the dynamic time effect of the mental workload, the present study improved the method of quantifying time pressure [17]. According to the TLAP, the time pressure affecting the mental workload is highly correlated with TR/TA on the timeline process. Therefore, the adapted calculation of the time pressure Ti is as follows, where TA⁢i is the available time of the i-th AOI, given by the system for the operator to perform a task, and TR⁢i is the required time of the i-th AOI, needed by the operator.

TR⁢i can be represented by the time required by the operator from information presentation to perception, understanding, and reaction. According to the “behavior time prediction” method, the time from perception to reaction includes the reaction time Tc⁢i and the exercise time Td⁢i. Tc⁢i can be subdivided into simple reaction time and selective time. The simple reaction time of different sensory channels can refer to empirical data [3]. Hick-Hyman Law [11] can be employed in calculating the selective time. Ic is the information quantity constant, with the value 50–157 ms, and the average is 92 ms. H is the average information amount of the stimulation signal in the i-th AOI. The exercise time Td⁢i is calculated by Fitts’ Law [7], where Im is the motion control constant, with the value 70–120 ms and the average 100 ms; log2⁡(D⁢/⁢S+1) is the motion difficulty index, where D is the target distance and S is the width of the target object.

Altogether, the multi-factor mental workload prediction value m⁢wjl of the j-th task unit decomposed by task 1 is obtained:

2.4Dynamic time effect calculation of mental workload

To monitor the pilots’ mental workload state in real time, the IMW, CMW, TMW, and AMW are established under the dynamic conceptual framework of mental workload [13, 14].

IMW is defined as the instantaneous amount of mental workload at a given time. Combined with Kjl and m⁢wjl, the IMW M⁢Wjl⁢(t) (in bit) for the j-th task unit decomposed by task l can be obtained:

The TMW M⁢WTl (in bit × s) is defined as the total accumulation of mental workload from the start time to the end time. In the formula, T0 represents the initial time, and T represents the termination time. Mathematical expressions of other mental workloads can be obtained from the mathematical expressions of IMW.

The AMW (in bit) is defined as the average intensity of mental workload throughout the time course.

3.1Participants

The study was performed on 21 participants (19 males and 2 females), with an average age of 24.5 years (N = 21, Mage = 24.5, SD = 1.0). They had an aeronautical knowledge background, right hand, normal or corrected vision, and no color blindness. The participants were required to participate in flight simulation training until they were fully familiar with the flight operations.

3.2Study design

The study was based on the actual flight tasks, using the display interface (3) × flight task (4) two-factor repeated measurement of the interior design of the test. As shown in Fig. 1, the display interface included three levels: the virtual main flight display (PFD) of B737, A320, and ARJ21, with proper simplification and abstraction. The flight phase included the takeoff phase and three cruise phases (90∘ turn, straight, and 180∘ turn). To avoid the practice and fatigue effects, the experimental sequence was designed in Latin.

Figure 1.

Simulated interface of B737, A320, and ARJ21, cruising flight phase (left) and simulation (right).

3.3Materials

The experiment was carried out in a Boeing 737–800 flight simulator, with real-time performance and eye movement indicators monitored. Within 15 min after the end of each experiment, the participants were asked to finish the subjective evaluation of different display interfaces and flight phases according to their real feelings.

Since the flight task performance was difficult to measure directly, the reaction time and the accuracy of the embedded subtask were used as performance indicators, which were automatically recorded by MATLAB programming through keyboard operation. The Swedish noncontact infrared eye tracking system Smart Eye Pro 4.5 was used to record the eye movement data, with an accuracy of better than 1∘ and a 60 Hz sampling rate. National Aeronautics and Space Administration-Task Load Index (NASA-TLX) [5] software was used for subjective evaluation.

3.4Experiment task

This experiment applied the embedded subtask method proposed by Wickens et al. [3], requiring the participant to monitor and identify the flight information on the PFD and navigation display (ND) using the remaining capabilities while ensuring that the manual control task (main task) was completed, including airspeed, altitude, pitch, and heading. During the experiment, the display interface was frozen and covered at any time, and the subject answered questions about the current value or change characteristics of the flight information within the specified time.

Manual control flight task is a simulated dynamic flight task for takeoff and cruise under a specified interface. The participants need to complete the takeoff operation by controlling the landing gear, throttle, flaps, and so on. The simulated cruise tasks consist of three phases. The participants are asked to control the steering wheel so that the aircraft can fly along the prescribed flight path and keep the airspeed, altitude, and attitude within the normal range.

4.1Theoretical predictions

First, the takeoff phase lasts 109 s. The operations using accumulated long-term memory related to operation steps through experience and learning are divided into 11 task units. Second, the cruise phase lasts for 240 s and is decomposed into a single task unit. PFD and ND are divided into five AOIs. Then, the IMW value is calculated, as shown in Fig. 2. The TMW and AMW are shown in Table 1.

Figure 2.

IMW of the three display interfaces during the takeoff phase.

Figure 3.

Distribution of gaze points (left) and correlation between mental workload prediction and some experimental values (right).

4.2Experiment result

The repeated-measures variance analysis (RMANOVA) of the two-factor design showed that the main effect of the display interface was only significant for the average gaze time (P= 0.005). The main effects of the flight phase on all indicators were significant (P< 0.005). The interaction effect between the two factors was not significant.

The multiple comparisons after single-factor test showed that the performance, subjective evaluation, and blink frequency were not significant among the three interfaces, and the gaze time was partially significant (PB⁢737,𝐴𝑅𝐽⁢21= 0.042, PA⁢320,𝐴𝑅𝐽⁢21= 0.004). Meanwhile, performance, subjective evaluation, and blink frequency were significant between the takeoff and straight phase (P= 0.001, 0.013, 0.001, and 0.001), and only the blink frequency was significant between 90∘ turn and 180∘ turn (P= 0.001).

As shown in Fig. 3, the Pearson correlation analysis between the prediction and experimental results indicated a significant positive correlation between TMW and subjective evaluation (r= 0.9, P< 0.001) and a significant negative correlation with mean gaze time (r=-0.681, P= 0.015). The AMW significantly positively correlated with the subjective evaluation (r= 0.809, P= 0.001) and reaction time (r= 0.537, P= 0.072) and significantly negatively correlated with the correct rate (r=-0.839, P= 0.001). The blink frequency was not related to the TMW and the AMW.