Recognizing human activities in Industry 4.0 scenarios through an analysis-modeling- recognition algorithm and context labels

3.1Analysis phase

In an Industry 4.0 scenario we are considering a pervasive hardware platform composed of P information sources (sensors, computing elements, etc.). These information sources provide information about production processes carried out by all people and activities in the environment. We are considering N different users developing M independent business actions.

Thus, after acquiring and aggregating all information sources, we obtain a time-variant vector X→⁢(t) of P components Eq. (1).

First, in the general case, X→⁢(t) is an analog (or analog-like) vector. Thus, signals must be digitalized through a sampling and retention scheme. This scheme, besides, must transform all information signals into integer time series represented all of them using the same number of bits, B. This is essential to compensate differences in the hardware devices characteristics (precision, resolution, etc.), and avoid numerical problems when operating with data Eq. (3.1).

The sampling period Ts must be selected according to human behavior characteristics. Thus, and according to the Nyquist theorem, any sampling frequency above 20 Hz is adequate Eq. (3).

This frequency may be modified according to the scenario and the considered hardware devices (for example, if cameras are also employed), but is adequate for environmental sensors, wearables and smartphones.

Considering this very reduced bandwidth, and the extremely high resolution required to maximize the precision of the following recognition algorithms, the sampling and retention scheme we are employing is a sigma-delta encoder. Figure 2 describes the block diagram of a standard Σ-Δ encoder.

Now, this initial digital vector, in general, is affected by physical random phenomena such as electronic noise, interferences, etc. These high frequency components may affect the following steps, so they must be removed. To do that, an exponential smoothing filter or exponential moving average (EMA) has been proved to be the most effective technique with time series. However, people tend to evolve with the workday, what creates trends in signals which may be removed by EMA. Besides, these trends are seasonal, as they are repeated every day. Because of these characteristics, we are not employing a simple EMA but a triple exponential smoothing (or Holt-Winters method), consisting of three EMA applied in a recursive manner. The first EMA applies an overall smoothing Eq. (3.1). The second EMA preserves the trends in signals Eq. (3.1). And the third and final EMA must preserve the seasonal information Eq. (6). The smoothed time series Y→⁢[n] (output) are obtained considering three real parameters α,β,γ∈[0,1]. Moreover, L is the discrete period of the seasonal components (in Industry 4.0 scenarios, twenty-four hours).

Now, in the smoothed vector of time series Y→⁢[n], not all components will be independent. In fact, as information sources belong to a pervasive platform, they are (in general) linked by C constraints. These constraints may belong to three different types. Namely:

These constraints, in industrial scenarios, are typically scleronomic (i.e. they are independent from time); and besides they are holonomic (i.e. they are independent from differential operations on the coordinates). In those conditions, constraints may be expressed as simple functions Eq. (7). These functions may be employed to remove redundant components in the smoothed vector Y→⁢[n] of time series, obtaining a new generalized vector Q→⁢[n] where all components (time series) are totally independent Eq. (3.1).

In general, this new vector will have P-C components. The benefit of this approach is that, now, every component may be analyzed independently from the others. We can imagine the obtained vector in a P-C dimensional space, where notions such as the Euclidian distances are applicable. Besides, vector Q→⁢[n] has a generic format where no particularities from physical sensors are affecting the signals. An additional normalization process can be carried out if required.

3.2 Atomic action recognition

In this context, it is possible to evaluate the similarity of two generalized vectors (or patterns) using simple distance functions, what enables doing a large number of comparisons in a short time (Euclidian distances are extremely computationally low-cost). In that way, the distance between two patterns or generalized vectors Qa→⁢[n] and Qb→⁢[n] at each time instant may be expressed through simple mathematical operations Eq. (9). Where each vector potentially contains information about atomic actions executed by workers.

However, in Industry 4.0 scenarios, atomic actions take a time period to be executed, T𝑎𝑐𝑡𝑖𝑜𝑛, so the proposed Euclidian distance would evolve with time. Besides, a standard quadratic subtraction as mechanism to measure the distance in each one of the P-C dimensions (components) is only valid if all actions are always executed at the same speed (what is not true in human performed actions). Therefore, we are evaluating the distance in each dimension using dynamic time warping technologies -DTW- Eq. (10), being function 𝑑𝑡𝑤⁢(⋅,⋅) the standard DTW algorithm [15]. Using DTW technologies, variations in the execution speed do not affect the final result, and a global estimation about the difference between two time series is directly obtained (the obtained result is independent from time). On the other hand, DTW algorithm is only valid for signals with a similar structure. This similarity level in the signals’ format is reached in the analysis phase, where signals are orthonormalized and segmented (aligned and synchronized).

Theoretically, DTW distance could be directly applied to patterns Qa→⁢[n] and Qb→⁢[n] (as, for example, in speech recognition systems), but this approach assumes all components inside each vector evolve at the same speech (although it tolerates that this speech is different in different vectors). Nevertheless, this assumption is not true in general in Industry 4.0 (where components represent sensors that evolve independently), so in our technology (as shown above) DTW mechanism must be applied to every information source independently and, later, aggregate all the obtained costs in a global distance.

As said, in this initial analysis phase, atomic actions are recognized. To do that, the P-C time series making up the generalized vector Q→⁢[n] are obtained as data streams. All the recognition process is performed at real-time (as required by Industry 4.0 applications) but, for clarity, we are describing the referred recognition process considering the whole Q→⁢[n] vector has been already received. This approach does not affect the described mathematical operations and algorithms.

This atomic action recognition process, basically, calculates the distance between elements si in a repository of patterns 𝒮={s→ii=0,…,Z} and the current values of the generalized vector Q→⁢[n].

The pattern repository 𝒮 contains Z different action patterns which are experimentally determined. Different users and experts (industry workers) are requested to perform those actions to capture the patterns and feed the repository. In this approach, the required time to completely analyze an industry scenario grows exponentially with the number of users and processes to be considered. Therefore, in very complex scenarios, the deploying cost of this solution might be high. On the contrary, the control and monitoring capacity also grows in precision and efficiency.

The pattern s→i* that is the closest (later we are describing this point with all details) to the generalized vector is selected as the atomic action being performed. However, atomic actions are characterized by being limited in time and space. Thus, several atomic actions could be performed at the same time and in the same global space. The generalized vector Q→⁢[n] will contain information about all of them, and (in this situation) DTW distance cannot be directly calculated. Then, a problem to be solved is to segment the generalized vector into sets of samples gi→⁢[n] containing only one atomic action Eq. (11), being n𝑖𝑛𝑖𝑡-n𝑓𝑖𝑛 the execution period of the atomic actions.

To separate atomic actions, we are grouping components qi⁢[n] in the generalized vector Q→⁢[n] that comes from devices that are together at a certain moment. To recover this geographical information at this point, it could be stored as metadata in the acquisition process. Besides, information about the user performing the action could be acquired. In this paper, this information is presented as semantic annotations (metadata) [42]. As different people may perform actions in a different manner, two areas are defined (see Fig. 3):

Limits for these areas (r1 and r2) are fixed according to the scenario under study. For example, in an Industry 4.0 scenario of hand-made basketry, they would take values of some centimeters.

All possible sets gi→⁢[n] generated by grouping elements in A2 in all exiting manners (regardless the order and without repeating elements) potentially represent the atomic action being performed. However, although the number of possible actions grows up exponentially with the number of components in A2 area Eq. (12), it is not required a long time to solve this calculation: only combinations (atomic actions) s→i which are also present in the pattern repository 𝒮 must be evaluated. Hereinafter, 𝑐𝑎𝑟𝑑⁢{A2} represents the number of elements in set A2. If different combinations are describing atomic actions stored in the repository, all of them will be considered and evaluated using the DTW distance. Components qi⁢[n] which are not finally attached to any atomic action, they are considered context signals.

Segmenting time series qi⁢[n] into time intervals (n𝑖𝑛𝑖𝑡-n𝑓𝑖𝑛) describing only one atomic action is a more complex problem.

To perform this action, we are proposing a sliding window scheme. This window s⁢[n] will have a square envelope and a duration of D samples. Besides, we are defining a core c⁢o⁢[n] in center of this window with a duration of Dc samples. The sliding window moves with an overlap of Do samples, which must include at least one sample in the window core (see Fig. 4 – dashed window is represented just for clarity as it represents the window in the previous time instant –). Parameter Dc is selected to adjust to the fastest atomic action. D is selected to adjust to the slowest atomic action, including (probably) a certain error margin. Finally, Do is selected to adapt to the average transition period between atomic actions performed by workers. The objective of this window structure is to locate all samples belonging to the same atomic action.

The proposed solution operates in the following manner (see Algorithm 1). The windowed time series (segment) is compared to the patterns (using the proposed DTW technique) considering as initial sample every sample from the initial one to the (D-DC)2-th sample. At the same time, the final sample is selected in the range [(D+DC)2,D]. It is defined a maximum admissible distance dmax. If no path has a cost below dmax, the sliding window advances D-Do samples. All these samples are considered empty noise between actions. If some paths have a cost below dmax, then the longest path (with more steps) is selected as the segment describing the atomic action, and the sliding window advances D-Do samples.

This sliding window and segmentation process is meant to, mainly, reduce the false negative elements, increasing the system recall. In noiseless scenarios, DTW technologies are tolerant to add or remove several samples from the signals. However, in noisy scenarios as distance thresholds must be more restrictive to avoid false positive elements, the segmentation process is essential to ensure all samples and contributions are considered.

In the most basic approach, the detected atomic actions s→i* are those which are closest to each segment gi→⁢[n]. However, this approach is very weak, and we are here employing a k-nearest neighbors (K-NN) algorithm [33] but modified to adapt to Industry 4.0 scenarios. First, if no pattern is closest than a certain threshold distance dt⁢h, no action is recognized. Second, as some atomic actions may be more common in the pattern repository 𝒮 than others, not all neighbors can be considered in the same manner. In that way, contributions to the estimation functions must be weighted Eq. (3.2) using a real parameter αj. Basically, patterns that are closer than a distance of d𝑏𝑟𝑒𝑎𝑘 units are considered “close actions” and weighted in a similar way. Patterns further than d𝑏𝑟𝑒𝑎𝑘 units are considered different actions and are penalized.

This weighting parameter αj is calculated through a function that may take different expressions. For this work, we are considering a linear piecewise function, depending on the distance between patterns Eq. (3.2). Besides, parameters β1 and β2 must fulfill a relation in order to define a valid weighting function Eq. (15).

After recognizing the atomic actions being performed, all components qi describing any of these actions are not considered anymore. Components qi which have not been identified to be part of any atomic action are injected into the following steps as context information.

3.3Modeling phase: User activities recognition

At this point, we have obtained two data structures. On the one hand, a set A→ of recognized atomic actions, labeled with a discrete temporal stamp T and a piece of information 𝔲, indicating the user that performed the action. Although any user recognition solution could be employed to collect this piece of information 𝔲, in order to reduce the acquisition cost we are employing a deterministic scheme. Wearable sensors and smartphones are directly associated to specific users (so each sensor monitors a worker), while environmental sensors may monitor several different users. Regarding the timestamp, in this model we are assuming that computational and acquisition delays are constant and independent from the specific action being recognized or sensor being employed. So, actions are recognized in the same order they are performed, and actions can be aggregated easily following a strict temporal order.

In the general case, we are considering V atomic actions are recognized in T0 time units Eq. (3.3).

On the other hand, a set of context signals C→ (those time series that do not contain data describing any atomic action and are not empty noise, as said in the previous section) is also obtained Eq. (3.3).

In the modeling phase, these two data structures are employed to evaluate user action and context models. Each one with a different approach (Conditional Random Fields and Neural Networks). As a result, user actions and context labels are recognized.

First, we are discussing how user actions are recognized.

A repository 𝒰 of user actions, where actions U→i are described as sequences of atomic actions uij is considered Eq. (3.3). This repository is easily built by monitoring the industrial scenario and recognizing atomic actions in a supervised manner.

As referred for the atomic action repository, in this case the cost of supervising users and creating the repository of user actions is not negligible. Specifically, this cost grows up exponentially with the number of workers and activities under consideration.

Now we are evaluating the conditional probability of a user 𝔲 to be executing a certain user action U→i considered the observed and recognized atomic actions A→ Eq. (19). The user action U→i maximizing this conditional probability is the recognized user action U*→ Eq. (20).

However, set A→ contains atomic actions performed by different users and, besides, actions belonging to different user actions. Then, we must split set A→ in different subsets before applying the discriminative model Eq. (20). Figure 5 presents the proposed splitting mechanism, which may operate, even, at real-time.

Then, N different subsets A𝔲→ are obtained Eq. (21). One for each user in the scenario. This process may be easily performed using the piece of information 𝔲.

Now, atomic actions in each subset A𝔲→ must be separated into new subsets describing only one user action. Thus, they can be compared to patterns stored in repository 𝒰. This process is based on the timestamp of atomic actions and a sliding window (see Fig. 6 – dashed window is represented just for clarity as it represents the window in the previous time instant –).

Recognized atomic actions are ordered as a time series, according to their timestamp. Then, a window e⁢[n] with a square envelope is employed to aggregate E atomic actions in a subset A𝔲,[n𝑖𝑛𝑖𝑡,n𝑓𝑖𝑛]→ Eq. (22).

Besides, we are defining a core e⁢o⁢[n] in center of this window with a width of Ec actions. The sliding window moves with no overlap. A window starts exactly where it finishes the last one. Contrary to the analysis phase, where empty noise may appear in time series, in this scenario every atomic action must be part of a user action. Thus, the sliding window has not a central structure (see Fig. 4), but a left aligned one (see Fig. 6); and no overlap is considered. Any case, the purpose of this structure is the same: to locate all atomic actions belonging to a user action, that are spread along a segment whose duration varies between the minimum (Ec) and the maximum (E).

The windowed (aggregated) atomic actions A𝔲,[n𝑖𝑛𝑖𝑡,n𝑓𝑖𝑛]→ are compared to patterns (using the conditional probability) in the user activity repository 𝒰. Different possible combinations will be considered, by selecting the final atomic action in the range [Ec,E]. All possible combinations are evaluated. The one with the highest probability is selected as the performed and recognized user action U*→. All the included atomic actions in that user action are removed, and the sliding window moves to start exactly where the previous user action finished. Algorithm 2 describes the proposed mechanism.

At this point, we only must discuss how the conditional probability P(U→i|A𝔲,k→) may be numerically evaluated from the observed atomic actions A→ and the patterns in the user action repository 𝒰.

Each one of the random variables considered in the conditional probability (U→i and A𝔲,k→) are, in fact, a vector of Pu and k (respectively) atomic actions (or elemental random variables) Eq. (3.3). As seen before, these vectors represent the sequence of atomic actions aiT,𝔲 performed by a certain user 𝔲 during a certain time period [n𝑖𝑛𝑖𝑡,n𝑓𝑖𝑛].

In real Industry 4.0 scenarios, despite the variable and flexible character of activities, actions tend to be performed following a minimum common structure (according to the production process, for example). Thus, once a certain atomic action aiTi,𝔲 is observed, the probability distribution of the next atomic action ai+1Ti+1,𝔲 varies (for example, atomic actions belonging to the same production process that the first action aiTi,𝔲 will be more probable). In other words, observed atomic actions are depending on each other. However, in this work, we are considering all atomic actions are independent. This assumption is introducing a certain error, but it will be more stable and more easily evaluable (aggregated into the model’s precision).

In those conditions, it is possible to develop the conditional probability as a product or partial probabilities (one for each atomic action uij in U→i). Besides, in order to allow this mechanism to be implemented using software tools, it must be casual, i.e. it must only depend on the atomic actions aiTi,𝔲 currently observed or in the past, but not in the future, as A𝔲,k→ contains all of them Eq. (24). Being Z a parameter to maintain the global value of the product in the range [0,1], according to the Kolmogorov’s definition of probability.

At this point, in order to improve the precision of the model, we can add (artificially) information about previously recognized user actions, contained in the set 𝒰*. This new information takes the form of new conditions in the conditional probability Eq. (25).

Each elemental probability P(uij|{alTl,𝔲l=1,…,j},𝒰*) must now be evaluated numerically, so it can be understood as an unknown function fj depending on atomic actions uij and alTl,𝔲 and the set 𝒰*. Then, these expressions Eq. (25) may be rewritten in a more compact manner Eq. (26).

Now, as humans might freely perform any action at any time, function fj(uij,{alTl,𝔲l=1,…,j},𝒰*) is never taking the zero value (no action is an impossible event). Then, function fj(uij,{alTl,𝔲l=1,…,j},𝒰*) may be understood as a Gibbs random field (GRF), whose probability distribution is based on exponential functions Eq. (3.3). Being Hj(uij,{alTl,𝔲l=1,…,j},𝒰*) a new function called the energy function of the GRF. Besides, with this new view, Z parameter may be calculated as the partition function of the GRF Eq. (3.3)

On the other hand, as function fj(uij,{alTl,𝔲l=1,…,i},𝒰*) is a GRF, we can consider the first lemma of Hammersley-Clifford theorem. Then, function fj may be factorized into two terms Eq. (29), fja and fji⁢t, separating the influence of the observed atomic actions alTl,𝔲 and the previously recognized user actions.

fja is called the “action function” and represents the influence of observed atomic actions. fji⁢t is called the “interaction function” and represents the influence of previously recognized user actions.

Each, factor, as said before Eq. (3.3) may be expressed as an exponential function considering an energy function. Thus, it is induced a new factorization Eq. (30).

Now, we can rewrite the expression for the conditional probability considering the GFR Eqs (3.3) and (3.3).

Functions Hja and Hji⁢t must be selected to represent the restrictions of industrial processes, and the human behavior. In order to do that, we are needing a learning process to capture that information in a systematic manner. However, to allow the utilization of existing learning mechanisms we must rewrite functions Hja and Hji⁢t in another manner Eqs (32) and (33).

Functions gl,ja⁢(uij,alTl,𝔲) and gU*→,ji⁢t⁢(uij,U*→) are unitary functions. Typically, they can be expressed as combinations of Kronecker’s delta functions. In this most simple formulation, they will be only one delta function Eqs (3.3) and (3.3).

Parameters θla and θU*→i⁢t are real values which weight the contribution of each function and must be learnt automatically. Different strategies could be employed to learn those parameters, but in order to select the optimal weighting scheme we are representing them as a vector Θ* Eq. (3.3). With this new formulation, the obtained model is formally identical to a General Conditional Random Field (GCRF); although the deduction process and meaning of each element is different. However, mathematically, the same numerical methods employed to train GCRF may be employed in our case. Particularly we are employing an optimization algorithm of the maximum verisimilitude logarithm.

In our approach, we are not using a generic model, but a model that is adapted to the industrial scenarios since the beginning and the initial mathematical definition. That is a novelty compared to existing solutions, which causes a relevant increase in the system precision and justifies the higher processing delay.

3.4 Modeling phase: Context label recognition

Now, we are paying attention to the context signals Eq. (3.3), which must be transformed into high-level context labels in this phase (to enable the business action recognition process in the final phase).

In this case, context series do not represent a behavior as complex as human behavior (like atomic actions), but the evolution of the environment (which is, in general, much slower, and predictable). Thus, a more standard approach may be employed to create context labels from these context signals.

First, we are defining a set Λ of context labels λc including 𝔏 different labels Eq. (37). The objective of this new module (context recognition, se Fig. 1) is to deduct which context labels are applicable at each time instant, according to the information contained in the context signals. This labeling problem adapts perfectly to the functionality of neural networks; so, we are employing a recognition module based on this technology (see Fig. 7).

In the proposed context recognition module, we are first segmenting context signals cm⁢[n] using a sliding rectangular window s⁢w⁢[n]. This window has a width of Wc samples Eq. (38). In this case, this window is moving with no overlap. Later, from each segment Eq. (39) brm⁢[n] is extracted a vector of features Vrm Eq. (40), including statistical and waveform characteristics. Extracted features (see Table 2) are selected to be referred as good quality features for context recognition [25].

Then, for each position of the sliding windows we are obtaining a large vector Vr→ including all features extracted from all context signal segments brm⁢[n] Eq. (41).

This vector becomes the input of a neural network. For this neural network, we propose a multilayer network, specifically a Multilayer Perceptron (MLP) formed by a stack of five hidden layers (see Fig. 8). There are two dense layers followed by Dropout layers with a rate of 12 which randomly switch off 50% of the MLP’s neurons in each epoch. This allows the neurons to independently develop meaningful features and avoids overfitting when processing the input feature vectors. Six hundred and twenty-five different trainable parameters are then defined in this network. The output of this network (a tensor) is a vector encoding the class probability for the input vector. The tensor resulting from this multilayer network (where the input to one layer is the output of the previous one) is then fed into a classifier bank with L different one-unit classifiers (one per each context label to be recognized), where a 2-class classification (context label recognized or not) is performed. As activation functions, we used ReLU (Rectified Linear Unit) non-linearity for Fully-Connected layer and sigmoid for the last Dense layer (encoding the probability of a class or the other).

For the training process, we use a stochastic gradient descent algorithm and Adam optimization (considered to be the fastest to converge) with a small learning rate of 1×e-3 to optimize the binary crossentropy function (which measures the similarity between the prediction and the ground truth when working with a network ending in a sigmoid function). The network was trained for one hundred (100) epochs. After each epoch, using never seen data, the classification error is measured, evaluated, and distributed across the entire network using backpropagation. For the testing phase (see Section 4) neither the training data nor the evaluation data are employed, so performance metrics are obtained using never seen data, which increases the experiment reliability.

We chose this architecture for its simplicity, computational efficiency and flexibility, which allow us to reach the real-time requirements of Industry 4.0 scenarios. As in the user activity recognition module, this neural network must be trained to capture information about the application scenario. To perform this process, we are also employing standard instruments.

Finally, after the classification process, we obtain for each time instant (position of the sliding window) a set {Λr⁢r∈ℕ} containing the recognized context labels.

3.5 Recognition phase

For this final phase, we need a new classification technique being able to chop input information and analyze the parts independently, although partial results must be later composed to obtain a global result. This description perfectly fits with a classifier based on Random Forests [37].

Random Forest technique consists of a set of D decision trees (see Fig. 9), which are fed with different subsets Id of the input information (in our case we can include discontinuous information in each subset) Eq. (3.5). We are only ensuring that each subset contains both, information about user activities and information about context.

Each tree, then, evaluates the input subset (or sub-vector) and decides about which business action Ei from the business action repository ℰ Eq. (43) is being executed.

This repository, as the other ones described in this paper, is created by supervising users and workers in the scenario under study for a period. All repositories may be created at the same time through a unique configuration phase. Atomic, user and business actions and activities should be defined by managers or industry experts according to the production processes, manufactured products, and business objectives. Any modeling language could be employed for this purpose.

Besides, we are modifying classic Random Forest; and all business actions Ei* which are recognized by more than Dt⁢h decision trees are globally recognized as current actions Ei**. Despite this modification in the final step, standard frameworks may be employed to create the proposed classifier, considering as input information sets 𝒰*, {Λr⁢r∈ℕ}, ℰ and ℐ. All decision trees are built during the training phase, which also may follow a standard procedure.

4.1Experimental validation: Materials and methods

To evaluate the performance of the proposed technology, two experiments are planned and performed. The first experiment was focused on evaluating the quality of the described technique through a set of standard indicators in the field of activity recognition solutions. The second experiment was planned to evaluate the performance of the new technology in terms of execution time and scalability. To perform these studies, the new activity recognition mechanism was implemented and executed using MATLAB 2017 software suite. In order to guarantee the obtained results for the new technology during the planned experiments are comparable to results previously reported in the state of the art, we are basing both experiments in standard datasets commonly employed to evaluate activity recognition technologies. Specifically, we have selected two different datasets: ExtraSensory [23] dataset and UJAmI dataset [16].

ExtraSensory dataset contains, mainly, information provided by personal mobile sensors integrated into mobile phones. Sensors such as accelerometers, gyroscopes and magnetometers are included in this dataset. On the other hand, UJAmI dataset contains information from a pervasive hardware platform including sensors such as NFC tags or temperature and CO2 sensors. Besides, to guarantee the statistical significance and validity of the results, the performance of the new activity recognition technology is analyzed through a k-fold cross-validation scheme. In that methodology, the working dataset is divided into k different sub-sets, equal and interchangeable These sub-sets are employed for training and testing the proposed technology and, then, interchanged. The process is repeated k times, one for each sub-set. Considering the number of records in the datasets, we are using a scheme with five iterations. Final results, presented in this paper, are the statistical mean values extracted from all these previous partial results.

Datasets employed in these experiments were selected according to different criteria:

With these criteria, two datasets were selected. ExtraSensory dataset describes sixty (60) users performing up to one hundred and sixteen (116) different activities in a multi-tasking scheme. More than 300.000 minutes of monitoring are present in the dataset. Personal mobile sensors are employed. Raw signals are available. On the other hand, UJAmI dataset represents workers performing activities in a pervasive sensing scenario, as it is envisioned to happen in Industry 4.0 applications. Only twenty-four different actions are monitored. Ten days of monitoring are available. UJAmI dataset was initially published in 1996. However, different actualizations and versions have been released and, for this work, we are considering the last 2018 version, so we can guarantee the dataset reflects current Industry 4.0 scenarios.

In order to guarantee that obtained results are not user-conditioned, when creating the five folds in the validation process independent groups of users were considered. Specifically, 80% of users considered in each experiment were employed to train the model and additional 20% of participants were employed to test the performance. Although this approach may cause overfitting under certain circumstances, in our experiment we saw a very high and constant performance in every k-fold. No subset where this performance is significantly lower has been detected. As a result, we can conclude the generalization capacity of our model is very high.

As said, two different experiments were conducted. For both experiments, the proposed new activity recognition technique was configured with a particular set of parameters, which are shown in Table 3.

Most of these parameters must be selected according to the activities to be recognized, although parameters related to the smoothing effect have optimum values that have been analyzed and reported in the state of the art [26]. In order to tune activity-dependent parameters, a “silence” detection analysis based on elemental signal processing may be done, so the length and duration of the different activities may be easily identified and calculated. The spectrogram tool is employed for this purpose in this work.

The first experiment was focused on analyzing the recognition and classification capabilities of the proposed technology. To do that, a standard collection of relevant performance indicators was considered (see Table 4). The entire datasets were employed to train and evaluate the proposed technique during this experiment. Three different situations were defined. In the first one, we are only using the ExtraSensory dataset. In the second one, we are only using the UJAmI dataset. And in the third one we are creating a new dataset, obtained by merging ExtraSensory and UJAmI datasets.

In Table 4, t⁢p indicates the number of activities that are correctly recognized; t⁢n indicates the number of activities that are correctly non recognized; f⁢p indicates the number of activities that are falsely recognized; and f⁢n indicates the number of activities that are falsely non-recognized. Besides, yi represents the recognized label for an activity, and xi indicates the real label for that activity. NT denotes the total amount of samples in the dataset. Finally, p⁢o refers to the observed probability in the entire dataset, and p⁢c represents the probability of chance.

In order to highlight the novelty of the proposed solution, and provide a relevant data comparison, the obtained results are statistically compared to the state-of-the-art hybrid mechanisms [40]. For this purpose, we have selected as reference the hybrid technology showing the highest accuracy [40] among all reported solutions in the last five years. Other more recent proposals [22, 50] could be found, but they show a worse performance. Although different tests could be employed, in this experiment we are using the Mann-Whitney U test, as it has been proved to be effective to compare activity recognition solutions. The p value indicates the significance level of the Mann-Whitney U test. Different tests for various significance levels (alpha parameter) were conducted. Significance levels have been selected to be the most usual and standard in the state of the art. The error associated to this test may be considered negligible given the size of the datasets we are employing [3].

On the other hand, the second experiment was focused on the performance and scalability analysis of the proposed technology. Considering the dataset generated by merging ExtraSensory and UJAmI datasets, the required time for the training process and the recognition delay are measured. From this dataset, different folds were extracted containing different numbers of users. For each fold, the training and recognition delay was measured. From this experiment, the required processing time, the scheme scalability and the algorithm temporal order was calculated and discussed.

4.2Results

Table 5 provides the obtained results for the first experiment. Globally, these results are coherent both, internally (among the different indicators) and externally [37]. No dissonant value or result may be seen, so they may be considered valid and statistically representative of the technology’s behavior. This conclusion is also supported by the high values in the Cohen’s kappa score. From Table 5 it can be deducted the proposed mechanism present a very good behavior as activity recognition technique in Industry 4.0 scenarios: F1-Score is near 0.9 for all experiments (even significantly above this value for the UJAmI dataset).

In crowdsensing scenarios (represented by ExtraSensory dataset), precision is almost 87%. This value considers all business activities represented in the ExtraSensory dataset together (such as driving, cooking, or working in the lab). Activities are heterogenous enough to represent a large catalogue of potential Industry 4.0 scenarios. The same catalogue of activities has been previously recognized using other approaches, some of them even similar to the proposed solution, and obtained results with our proposal (globally) improve up to 10% the performance of these state-of-the-art techniques [37] applied to the same dataset. In general, activities that are performed in a continuous and homogeneous manner (such driving or walking) are recognized with a better precision than activities that are non-continuous (such as cooking or bathing). The difference in precision between both kinds of activities is around 2.5%.

The best results are obtained for UJAmI dataset, which represents environments based on pervasive sensing platforms, and shows a F1-Score around 10% higher than experiments with other datasets. On the other hand, the proposed scheme in this work improves the precision around 8% compared to the state-of-the-art proposals where the entire catalogue of activities in the UJAmI datasets are considered [27].

More complex Industry 4.0 scenario will include both, personal sensors and pervasive sensing platforms. These scenarios are represented by the merged ExtraSensory + UJAmI dataset. In this case, the precision, as well as the F1-Score, is a little bit lower than the value for the ExtraSensory or UJAmI datasets independently (a reduction about 2% and 10% respectively). However, results are still improving, although in a more moderate manner (around 7%–8%, depending on the indicator), the performance of techniques in the state of the art.

Although some discussions have been provided, comparing the results with state-of-the-art mechanisms, Table 6 shows a formal statistical comparison with existing hybrid approaches [40] using the Mann-Whitney U test. As it can be seen, in general for all metrics the proposed solution is significantly better than the state-of-the-art hybrid mechanisms [40] applied to the same datasets.

First, in general, all metrics improve with a significance level of 0.005. However, in our approach, F1-score shows a more similar behavior to previous proposals than other metrics, and the significance level reduced in one magnitude order. Even, for the experiment considering the ExtraSensory and UJAmI datasets together no difference is detected. Any case, globally, we can conclude the proposed scheme improves the performance of state-of-the-art mechanisms, as Kappa parameter shows a relevant improvement with a significance level of p= 0.05.

In order to add more information to the discussion, we are analyzing some relevant activity types. Namely, the continuous (C) and non-continuous (NC) activities, and the activities performed by one (I) or by several (G) workers together. These disaggregated results are represented in Fig. 10. Besides, Table 7 present the confusion matrix for these four groups and all the considered datasets.

Continuous activities are those that last for a long period generating a homogeneous and almost permanent sensor outputs (such as driving or sitting). On the other hand, non-continuous activities are those that last a short time or have a variable behavior (for example, cooking).

As can be seen there is no big difference between performance for individual and group activities. Indicators are slightly higher for individual activities, but differences are below 1%. Only errors originated in this last phase affect the differences between individual and group activities (contrary to other approaches based on monolithic solutions).

However, there is a significant difference (in the environment of 5%) between the performance for continuous and non-continuous activities. In this case, discontinuities affect both, the modeling, and the recognition phases. As a general idea, complex business activities (with discontinuities and several users collaborating together) are recognized with a lower precision (e.g., working in the laboratory) than activities with a simpler structure such as driving or lying.

In order to analyze with more details which kinds of activities are recognized with the best precision, Table 8 shows a statistical comparison of the obtained results with the state of the art, using the Mann-Whitney U test. Besides, in order to enable a heuristic comparison, Table 9 shows the values for the main indicators (precision, recall, specificity and F1-score).

First, in general, all kind of activities shows a significant improvement in all metrics compared to the state of the art. In general, precision and F1-score improvement have a significance level of p= 0.05; while recall improvement shows a significance of p= 0.005. However, as it can be seen, differences are more significant for continuous and individual activities (such as driving). Besides, sensor information from ExtraSensory datasets (smartphone, mainly) also allows a more significant improvement than information from pervasive platforms. That may be caused by the precise user identification enabled by phones’ sensors.

Regarding the different activity types, continuous and individual activities (as they have a simpler structure) are recognized with a better precision, recall and F1-Score. This includes activities such as lying, sitting, running or driving. In this case, the significance level of the improvement is close to p= 0.005. On the contrary, non-continuous and group activities are more complex, and the improvement is less significant. The increase in precision and recall, in this case, has a significance level one magnitude order lower: p= 0.05; while F1-Score does not show any significant difference. Activities such working in the laboratory or cooking belong this second group.

Figure 11 shows the results of the second experiment. As can be seen (Fig. 11b), only around 200 milliseconds are required to recognize a business activity using the proposed framework. Almost-random variations may be observed in the figure, of 3% between the maximum and minimum values, but they can be easily explained by exogenous processes affecting the experiment, such as delays caused by the operating system and other applications that are sharing the resources. Any case, seen the obtained graphic, the temporal order of the proposed solution during the operation phase is almost linear with respect to the number of users. This is the most desired behavior for industrial solutions.

On the other hand, in Fig. 11a the required time for the training process is shown. Values range between only twenty minutes (approximately) for scenarios where only one user is employed to train the algorithm; to around four hours, required in scenarios where sixty workers are involved in the training process. These results are coherent with the idea that training processes in our solution are performed using standard mechanisms, which have showed similar behaviors in other previously reported works [52].

In this case, and using a fitting mechanism, we have found that the temporal order of the proposed solution during the training process is n⋅log⁡(n) with respect to the number of users involved in the training.