Review of methods to identify the critical job-tasks undertaken by the emergency services

1.1The purpose of Physical Employment Standards (PES): Why develop them?

Tipton et al. [8] state that the purpose of establishing a PES is to ensure that workers can complete the critical (in terms of performance and productivity) and generic (i.e. common or regular) tasks associated with their job without undue stress to themselves and to others working with them. Such purpose suggests that the physical demands of work may not be achieved by all people and that there is an associated risk of injury which may, within reason, be managed to its irreducible minimum. However, legislation mandating that employers enforce policies for the provision of health and safety at work (i.e. duty of care) and equal opportunities (without discrimination) have likely been the underlying drivers for PES since the 1970’s [9–11]. Recognising the risk of work-related injury and the need to maintain a safe, effective workforce of sufficient size, employers have increasingly invested in the development of PES as part of their selection process and to inform the training of recruits and incumbents. Efficiently placing people to the work for which they are best physically suited has been the premise of strategies to promote cost effectiveness in recruitment (as it is hoped that staff successfully complete their training at the first attempt, progress to deliver their roles without injury and remain in-service for the duration of their career). Increased legal scrutiny (since 2015) of the rationale for the exclusion (by many Armed Forces of various nations) of women from employment in the ground close combat roles emphasised the need for evidence-based PES.

1.2The process for developing PES: How to develop them?

There are a number of well-established frameworks for the design and development of PES. The fundamental process for developing PES has been the topic of numerous technical events since the 1990’s [12–18]. Whilst researchers in Canada responded to legal scrutiny by establishing Bona Fide Occupational Requirements (BFOR) [12] other authors continued to report a similar process for developing physical fitness standards [19]. Whilst Gledhill et al. [12] had described the development of a BFOR in 12 stages and Taylor and Groeller [20] offered their generic planning model (incorporating trade- and task-analyses), the 6-stage process described by Tipton et al. [8] has been most often cited for PES:

Common to all approaches reported in the evidence-based literature is the need to correctly identify the critical job-tasks at the very start of the process. However, varied criteria have been reported for determining critical job-tasks and the methods used to acquire the evidence (i.e. qualitative and quantitative data).

1.3The need to standardise the PES process

The precedent has been set within occupational research for use of the 6-stage process in developing PES [8, 21–24]. However, similar precedent is less evident with regard to the methods for conducting the initial stages of this process which are often contested in the Court of Law and must withstand legal scrutiny [24]. Reference to job analyses, task analyses, and trade analyses are commonplace for stage 1 in the process but they describe very different methods and data. Recognising the need for standardisation, researchers have proposed methods for each stage [22–24]. Payne and Harvey [23] provided their (often cited) comprehensive framework for the design of physical employment tests and standards, in which they discussed job-task analyses (considering component tasks and analysing the mode, frequency, duration, intensity and work: rest ratio). However, the description, by Tipton et al. [8], of job-task analysis as objectively subjective, highlighted the complexity of techniques that have been used to investigate this first stage in the PES process. Methods to select (and acquire data from) reliable, experienced, informed expert practitioners of the critical job-tasks have been suggested by leading researchers of PES [22, 24]. In 2016 an international delegation of these PES experts (representing 9 nations) formed a North Atlantic Treaty Organization (NATO) research task group [25] with the objective of designing an international standardisation agreement defining the methods to use when developing PES for combat roles.

1.4Job-task analysis

A job-task analysis (sometimes known as a job analysis, task analysis or work analysis) has been defined [21, 26] as the process for establishing an accurate accounting of the tasks or activities that take place in a job. No published guidelines exist to instruct researchers on how to conduct job-task analyses [21, 27] and there is no single correct method that fits all requirements. Hardison et al. [21] recommended that the choice of method may necessarily vary according to the intended purpose of the analysis (e.g. to develop PES or to review doctrine). They [21] added that the centrality of a job-task analysis in defending the use of a selection system or PES could not be overstated. There was a fundamental requirement to correctly identify and characterise the job-tasks which were evidently critical to the success of the emergency service. Furthermore, the job-task analysis was considered to be fundamental to ensuring that the standards for an occupation were valid predictors of critical job requirements (such assurance was considered essential in order to withstand legal scrutiny [28]). Landy and Vasey [29] reported that plaintiffs would typically assert that there was a fatal flaw in the job analysis techniques, analyses, results, or inferences. Thus, the choice of method used to define the content of the job would be vital in addressing some of the criticisms that might be raised. Larsen and Aisbett [27] asserted that given the importance of reporting accurate and legally defensible job-task analyses it was imperative to critically evaluate methods and to provide a best-practice approach for future research.

Hardison et al. [21] reviewed several methods used to develop PES for the Armed Forces. The techniques that were cited when conducting job analyses included; [a] analysis of documentation such as job descriptions, training manuals and task inventories (i.e. the employer’s list of tasks that were in-scope for specific jobs); [b] review of relevant scientific literature and professional competencies; [c] conduct of site visits to observe, and to interview, incumbents performing their jobs; [d] interviews with instructors and job-supervisors; [e] judgement panels comprising Subject Matter Experts (SME); and [f] on-line staff surveys. Zumbo [30] suggested criteria to apply when selecting SMEs as suitably qualified and experienced professionals. Use of technology (where participants rated the performance of scenario-based tasks that were observed via recorded imagery) enabled Siddall et al. [31] to acquire real-time data from anonymized e-voting systems within their expert judgement panels. Methods were cited which acquired both qualitative and quantitative data [8] and included ratings of task importance, difficulty, intensity, duration and frequency.

1.5Objective of this study

To identify the methods by which critical job-tasks have been determined when developing PES within the emergency services and to assess whether a standardised approach may exist.

2.1Eligibility for the review

Titles and abstracts of the acquired studies were assessed for their eligibility against the following criteria:

a. Qualification for review required the following conditions to be met (i.e. inclusion criteria):

b. Records were excluded if they were:

The above criteria formed a checklist which was used by two of the authors of this review to independently assess the studies for their eligibility. Only those studies which were assessed by both authors as eligible were included in the review.

2.2Processing the reported methods

Details of the methods reported were entered into a Microsoft Excel workbook. Each reported method was assigned to one of the following {1} to {9} categories:

A process for assessing (e.g. scoring) the quality of implementation for each reported method was not conducted.

2.3Analyses

The methods that were reported by each of the studies which successfully met the eligibility criteria were recorded by their category ({1} to {9}) and quantity (i.e. the number of methods used from 1 [minimum] to 9 [maximum]). The frequency with which each individual method was reported (between studies) was calculated as a proportion (%) of the total number of eligible studies in the review. The Microsoft Excel data analysis histogram tool was used to determine the frequency (calculated as a proportion [% ] of the eligible studies) with which multiple methods had been reported. The statistical mode was calculated for the number of methods reported by the studies (data were not adjusted for heterogeneity).

3.1The literature search

Use of the keywords in the initial search identified 617 records. No previously reported review was identified which sufficiently met the objective of this present study. A further 6 records were acquired from other (manual) sources which included proceedings from scientific conferences. Hence, a total of 623 records were assessed for their eligibility against the inclusion/exclusion criteria. Figure 1 presents the outcome of each stage of the assessment process which resulted in the independent assessments by 2 of the authors of this present study concurring that 42 of the records qualified for this review.

Fig.1

Results of each stage of the process in assessing eligibility for inclusion in this present review.

3.2Findings from the studies that were reviewed

The number and type of methods used by each of the 42 records included in this review have been described in Table 1. The value for the mode number of methods or techniques reported by a record was 4 (methods reported ranged from 1 [least] to 9 [most]). Twenty-five records reported using 4 or more method categories (Fig. 2), whilst 2 of the remaining 17 records relied exclusively upon the use of surveys (i.e. applied only 1 method). Within the 42 studies that were reviewed the use of survey methods were evident in 38 (90%) cases whilst consultation with Subject Matter Experts (SME) and stakeholder panels (empowered to make decisions regarding the emergency service), i.e. {8}, was cited in 34 (81%) cases (Fig. 3).

Fig.2

Use of methods to identify critical job-tasks in the emergency services (as reported by the [n = 42] studies in this review).

In 22 (52%) cases the studies described observing incumbents conducting their service roles, i.e. {6}, (either during operational duty or scheduled training exercises) as well as seeking assurance by cross-reference to policy and doctrine, i.e. {3} (established tactics, techniques and procedures for use in both training and emergency scenarios).

The miscellaneous methods which were categorised within {9} included: (a) research staff undertaking the training of the emergency service (to gain understanding and practical experience of the tasks and their associated physical demands); (b) use of mathematical modelling techniques; (c) development of operational scenarios in which the critical job-tasks were conducted; (d) development, or review, of operational task lists; and (e) reference to critical job-tasks which had been defined by previous research. Thirteen (31%) records cited the use of task lists developed by, or on behalf of, the emergency service (e.g. Military Occupational Specialisations [MOS], and Mission Essential Task Lists [METL]). Only 5 (12%) records sought information to describe how job-tasks may be conducted in future scenarios (obtained by method category {1}, {8} or {9}), or acquired guidance from SMEs regarding strategic plans for the immediate future of the emergency service (e.g. the planned introduction of new equipment and operating procedures). Records (n = 37 [88%]) tended to limit methods to acquiring evidence from past or present operations and doctrine.

Fig.3

Number (and proportion) of the 42 studies reporting use of multiple methods to identify critical job tasks in the emergency services.

4.1Standardising the identification of critical job-tasks: Factors to consider

Dukalskis and Beadle [32] suggested that there were many commonly used methods for systematically obtaining details regarding a job, task or role (i.e. job analysis [33]). However, the choice of method varied and depended upon the type of information that was hoped to be gained and the purpose of the investigation. It was evident that the variance between services regarding the availability of appropriately managed information and data was an important factor in the cases that were reviewed (e.g. MOS; competency frameworks; training manuals; incident reports and investigations; incident lessons; performance databases; continuous professional development records etc.). Hence, any future intent to standardise the methods for determining critical job-tasks [25] (and their performance criteria) must encourage employers to maintain robust and accessible information management systems. Operational data are needed to understand the determinants of success and failure when describing job-task performance.

It is likely that PES will be subjected to further legal challenge in the future. Subsequent reviews of this nature should consider the methods that were used to establish critical job-tasks in cases where PES was successfully defended. Where such methods are found to be consistently successful in their legal defence, they should be incorporated as best practice when developing PES.

Establishment of an international standard (for developing PES) will facilitate data sharing and help to determine reliable success criteria when setting standards or cut-off scores in performance assessments. However, a paradox may exist where the intent to reflect the reality of conduct on operations is in conflict with the method of best practice as advocated in doctrine and underpinning PES. An example of a physically demanding job-task has included the replacement of a wheel on a truck [34]. Although doctrine may describe procedures for loosening the wheel nuts using upper body muscle strength, in practice applying body weight to the wrench (by standing on it) may reflect the reality of common practice on operations and result in a lesser requirement for upper body muscle strength. Implications for strict adherence to doctrine (in such cases) when setting PES need to be understood and explained to the employer.

4.2Implications of strategic plans on the priority and performance of future job-tasks

It can take several years to develop PES [24, 34, 35]. Strategic review of the roles of the emergency services (including procurement of new equipment, infrastructure, protective clothing and operating procedures) is typically conducted within a similar timescale [36–38]. Therefore, criteria used to develop PES must remain relevant to the operational roles for which they are expected to represent else they risk being out-of-date as soon as they are implemented [82]. When designing a BFOR, Gledhill et al. [12] stated that the job-task analysis needed to understand the evolution of the job both past and future as well as considering the changing nature of the job (including legitimate variations in how the job-tasks may be achieved). Authors of guidance concerning the process for developing BFOR and PES suggest the need to consider future tasks but they do not offer any specific methods for achieving this. Only 5 of the 42 records included in this review considered the implications of future developments in their critical job-tasks. However, employers undertaking strategic review of the roles in the emergency services typically apply techniques such as Wargaming [39] and red teaming to assess resilience and to investigate future ways of operating. Wargaming roles [39] and scenarios [83] planned for the future workforce (an established scenario-based model where the outcome and sequence of events affect, and are affected by, the decisions made by SME players) was not evident in any study in this review. Wargaming facilitates realistic role-play for specific scenarios [83] which may allow incumbents to repeatedly practice common incidents as well as those emergencies which may occur only rarely in a typical career. Existing guidance for conducting job-task analyses when developing PES [40] does not consider the techniques (such as Wargaming) that are increasingly used by these employers.

Reference to the emergency services describes the collaboration of multiple agencies responding to the same incident. However, PES may differ between services despite the similarity in the emergency job-tasks they conduct. Although some differences may be reasonably defended, there remains a risk of legal challenge which would be wise to anticipate and address [84]. Furthermore, emergency services rarely, if ever, deploy individuals to manage an incident. However, PES are applied to individual performances. Consideration should be given to determinants for effective team working as well as individual competence.

4.3Potential opportunities provided by emerging technology

PES that are valid are those which correctly distinguish between people who are likely to be able to perform the requirements of the job from those who are not. Hardison et al. [21] suggest that without careful implementation and ongoing monitoring and updating, even well designed standards will fail to screen individuals appropriately if the testing is done improperly or as occupational tasks and equipment change over time. Emerging technology is increasingly enabling personnel to be monitored in ways that had not previously been possible. Wearable sensors and stand-off sensing continue to develop rapidly and may soon enable the concept of rolling validation (i.e. on-going evaluation of the physical demands of job-tasks) to be achieved [85]. GPS-based techniques for conducting notational analysis of individuals operating in large complex teams is becoming increasingly available to the emergency services. Process may endure but the methods must progress with such advances in technology in order to benefit from greater ability to monitor true work performance and to improve the sensitivity and specificity of PES.

4.4Employment standards an holistic approach

In 1999 Dukalskis and Beadle [32] developed the FLAG system (a system for conducting a job analysis based upon Fleishman’s algorithm [41]). Their system recognised that jobs are not purely physical in nature but require timely and effective coordination of various competencies (e.g. cognitive and physical). Future standards may need to consider an holistic approach [86] to assessing job-task performance rather than discriminating individual competencies in isolation. Sothmann et al. [42] included cognitive tasks within their job-task inventory as did Padula et al. [86].

4.5Recommendations

The results suggest that there is a need for the emergency services to:

Research to inform techniques which develop and assure employment standards should consider emerging advances in practical, pervasive monitoring technologies (including smart technologies/environments and wearable systems). There is also a need to develop reliable methods to assess the contribution (and performance) of individuals whilst they work in teams during real-time operations. Methods must be applicable to all emergency services and must be readily integrated within their operational systems (without unduly adding to the demands of conducting the job-tasks). Data from real-time operations (undertaken by the emergency services) remain an enduring and critical requirement in the assurance of evidence-based employment standards.