Emotional manifestations of humanoids of the future

2.1.Emotion recognition

Humans learn the ability to recognize emotional patterns after having made sufficient number of informed observations of what different types of emotions look like, e.g. happy vs sad faces. It is with this principle that the essence of machine-enabled emotion recognition, which is a special case of pattern recognition, has been so designed (Picard, 2000).

In particular, during the so-called training phase, the machine is shown samples of different categories of emotions. Each emotional sample is processed by a feature extraction process in order to extract a concise and most precise representation of the sample. For instance, we recognize emotions shown on faces based on certain cues observed on the face, e.g. smiley eyes, upturned mouth corners indicate a positive emotion. In analogous fashion, each output feature from this process is a concise representation of the different categories of emotion samples. These features are stored in a database, ready for use for subsequent recognition.

During the recognition phase, when a sample is observed for which the manifested emotion is unknown, the sample is put through a similar feature extraction process, and then a classification (i.e. matching) function is performed that compares the currently extracted feature with the features stored in the database. The feature in the database that is the closest match to the current feature will lead to the conclusion that the current unknown sample belongs to the same emotion category as that matching feature in the database. This is the gist of the emotion recognition process as handled by the most advanced machine learning and AI techniques in the literature, to date able to achieve accuracies of over 97%.

Thus far, researchers have demonstrated that machines are able to recognize emotions when given data of various modalities, including facial videos, human speech, gestures such as keystrokes and key presses, and even gait (how we walk). Indeed, the science has advanced to even recognizing hidden emotions, exhibited as so-called micro-expressions. These typically are only observable within a fraction of a second on a human’s face because s/he would tend to immediately suppress such facial expressions in a bid to conceal the felt emotion.

2.2.Emotional manifestation: Synthesis vs emotions transfer

To complete the emotional manifestation capability of machines, beyond just recognizing emotions, there is a need to be able to react to the sensed emotions, so-called affective computing (Picard, 2000). What is more realistically convincing is if the machine has the capability to reciprocate the emotional expressions.

Research into emotional synthesis includes enhancing speech with emotional effects (Schröder, 2001). While efforts had been made to directly synthesize such emotional speeches from scratch, the initial attempts sounded mechanical and therefore unrealistic. The situation is quite different now with the latest advances of deepfakes, which refers to using AI techniques to fabricate realistic fake images, video and audio. In particular (Floridi, 2018), researchers synthesized what was supposed to have been JF Kennedy’s last speech if he had not been assassinated.

The main technology underlying deepfakes is known as generative adversarial networks (GANs) (Goodfellow et al., 2014), which can be seen as a specific type of deep learning, the in-trend term used to refer to neural networks with multiple inner layers of neurons. GANs are so powerful as they comprise two types of networks essentially. A generative network whose aim is to generate realistic outputs (images, video, audio) from random data, guided by the responses from a discriminative network that aims to distinguish between whether inputs given to it are real or fake. The generator learns from the responses of the discriminator, and they take turns until they reach equilibrium, i.e. no further changes in each side’s actions will affect the optimality of its outcome. Since 2017, GANs and GAN-enabled deepfakes have taken the world by storm, with numerous victims in celebrities, public figures and even people on the streets who have had deepfakes generated of their images or videos shown in compromising situations.

Beyond swapping faces, these types of technologies also enable facial style transfer (Thies et al., 2019), including transplanting facial expressions (which infer emotions) from one to another, leading to a way to manifest emotions despite the original facial video not exhibiting such emotions. Transferring the facial expressions results in more realistic videos than synthesis from scratch as in this case one starts with an already realistic facial video except that no desired facial expressions are exhibited yet.

Facial expression transfer techniques mean that it is now possible to also imbue neutral-faced animated avatars with human-like emotional expressions. In the physical realm, if humanoid faces, e.g. Sophia’s (Raymundo, 2016) had motorized facial landmarks, then these can be triggered to move in synch with the facial expressions of the originating human subject, further contributing to the emotional realism of humanoid faces.

2.3.Mapping facial expressions to underlying emotions

In order to enable machines to recognize human emotions, there is a need to map the observed facial expressions to underlying emotions felt by the human. Psychology tells us that this mapping is no simple issue. Before one can even attempt to measure the accuracy with which an entity (human or machine) can read the emotion of another human, it is necessary to know the true mental state or emotion of that individual (henceforth referred to as ‘the target’), which (West and Kenny, 2011) refer to as the ‘truth criterion’. Establishing the truth criterion when it comes to emotions is challenging, and perhaps for this reason much of the research on emotion recognition in psychology has focused on questions of process rather than accuracy (Zaki and Ochsner, 2011).

Very often, then, emotion recognition has been investigated by presenting people with photographs (or, more recently, videos) of actors who have been asked to pose different emotional states. Accuracy in this context can be established either by examining the extent to which observers’ judgments match with the emotion that target was asked to pose, or by observer consensus whereby the ‘correct’ emotion is the one agreed by observers; e.g. (Baron-Cohen et al., 1997). While such methods can usefully shed light on the processes involved in making judgments about emotional states, it is not clear that they tell us anything about the actual mental states of the target in question or how they relate to the expressions produced. Indeed, the target who is posing the emotion may not feel any emotion at all or may be feeling an entirely different emotion from the one he/she is posing. If we want to map emotional expressions onto internal states it seems apparent that this needs to be done using natural displays of emotion rather than posed stimuli.

Suppose that, rather than asking the target to pose specific emotions, a video has been recorded of a target who is naturally engaged in a conversation, throughout which the target’s face is animated. Probably in this context the target is experiencing some genuine emotional states, offering opportunity to carry out such mapping. But how might we know what emotional state(s) the target is experiencing? Perhaps the simplest solution is to ask the target him/herself. This technique was devised by (Ickes, 2001; Ickes, 2009), who asked targets to view videos of themselves engaged in conversation and asked them to state what emotions they were experiencing throughout the interaction. Later Ickes presented the same videos to other observers who were asked to infer the emotions of the target and accuracy was operationalised as the agreement between the observers’ judgments and the target’s self-report. Using this method, known as the empathic accuracy task, Ickes reports high levels of agreement between targets and observers, suggesting a close mapping between inner mental states and expressions that observers are able to read.

However, the technique is open to criticism in that when viewing the video playback, the target may not recall or may not even know what emotion he/she was experiencing at the time of the conversation, so arguably may make an inference based on observable behaviour in much the same way as any other observer (the accuracy of which would be not known (Teoh et al., 2017)). Further difficulty arises when we acknowledge that in real life situations, people are likely to experience emotions that are fleeting and subtle, and blends of different emotions and mental states, which may be difficult for a target to report. Finally, self-reports of emotions or mental states are constrained by the labels that have developed within language to describe these states. These may not perfectly match the wide range of states that people experience over time.

On the other hand, if the aim is to create machines that are indistinguishable from humans in their emotion recognition capabilities, it is not necessary to know whether the robot is accurate in its recognition: it merely needs to make judgments in the same way as humans do. This means that mapping emotions onto real internal states may not be necessary – it is just necessary to know what mental states people infer in these situations.