PFA-GAN: Pose Face Augmentation Based on Generative Adversarial Network

2.12D/3D Model-Based

Feng et al. (2017) proposed a 2D-based method to generate profile virtual faces with out-of-plane pose variations. They built a PCA-based shape model to control only the yaw rotations since the pose varies with the same rotation direction of the original shape, i.e. left or right. Meanwhile, many approaches employed 3D face models for face pose translation (Crispell et al., 2017; Blanz and Vetter, 1999; Zhu et al., 2016; Guo et al., 2017). Crispell et al. (2017) use a 3D face shape estimation method, followed by a rendering pipeline for arbitrarily reposing of faces and altering the light conditions. Although the results of the face re-lighting method are good, reposing of the face produces many distortions in its structure. In addition, the background is not fixed since it is rotated along with the direction of the face rotation. Blanz and Vetter (1999) proposed a method to estimate a 3D morphable face model by transforming the shape and the texture of a face image into a vector space representation. Then, faces with new poses and expressions can be modelled by modifying the estimated parameters to match the target 3D face model. This method is good at generating faces with small poses, but it failed with large poses due to the serious loss of the facial texture. Furthermore, some additional steps are required at synthesizing facial expressions such as smiling to generate the hidden regions (e.g. the teeth). Zhu et al. (2016) introduced the 3D Dense Face Alignment (3DDFA) algorithm to solve face alignment in large poses. 3DDFA has also been used to profile faces, which means synthesizing the face appearances in profile view from medium pose samples by predicting the depth of face image. However, this augmentation method reduces the realism of the generated images. Guo et al. (2017) proposed a face inverse rendering method (3DFaceNet) to recover geometry and lighting from a single image. With that, they can generate new face images with different attributes. Nevertheless, their inverse rendering procedure has limitations, and it may lead to inaccurate fitting for face images (e.g. estimating the coarse face geometry and pose parameters from a face image).

2.2GANs-Based

Recently, generative adversarial network model learning (Tran et al., 2017; Tian et al., 2018; Cao et al., 2018a; Antoniou et al., 2018; Yin et al., 2017; Huang et al., 2017; Zeno et al., 2019a) demonstrated an outstanding ability to synthesize face images with new poses. Tran et al. (2017) introduced Disentangled Representation Learning-Generative Adversarial Network (DR-GAN), where the model takes a face image of any pose as input and outputs a synthetic face, frontal or rotated with the target pose, even for extreme profiles ( ±90∘). The discriminator in DR-GAN is trained also to predict the identity and the pose of the generated face. Tian et al. (2018) proposed the Complete Representation GAN-based method (CR-GAN) following a single-pathway design, and a two-pathway learning scheme to learn the “complete” representations. Cao et al. (2018a) have introduced Load Balanced Generative Adversarial Networks (LB-GAN) to rotate the yaw angle of an input face image to the target angle from a specified set of learned poses. The LB-GAN consists of two modules: a normalizer, which first frontalizes the face images, and an editor, which rotates the frontal face after that. Antoniou et al. (2018) introduced Data Augmentation Generative Adversarial Network (DAGAN) based on conditional GAN (cGAN). DAGAN captures the cross-class transformations since it takes any data item and generates other points of the equivalent class. A particular case of a pose transformation is the face frontalization. It is often used to increase the accuracy of face recognition systems by rotating faces to the frontal view, which is more convenient for a recognition model. Many methods have been introduced to frontalize profile faces, such as GAN-based methods (Yin et al., 2017; Huang et al., 2017). The FF-GAN method (Yin et al., 2017) relies on any 3D knowledge for geometry shape estimation, while the TP-GAN method (Huang et al., 2017) infers it through data-driven learning. TP-GAN is a Two-Pathway Generative Adversarial Network for synthesizing photorealistic frontal views from profile images by simultaneously perceiving global structures and local details. FF-GAN is a Face Frontalization Generative Adversarial Network framework, which incorporates elements from both deep 3DMM and face recognition CNNs to achieve high-quality and identity-preserving frontalization with less training data. Both TP-GAN and FF-GAN methods obtained impressive results on face frontalization, but they need explicit front-view annotations.

2.3IP-GAN

Zeno et al. (2019a) proposed a framework for Learning Identity and Pose Disentanglement in Generative Adversarial Networks (IP-GAN). To generate a face image of any specific identity with an arbitrary target pose, IP-GAN incorporates the pose information in the synthesis process. Different from the recent work (Yin et al., 2017) that uses a 3D morphable face simulator to generate pose information and the works (Tran et al., 2017; Tian et al., 2018; Huang et al., 2017) that encode pose annotation in a one-hot vector, IP-GAN can learn such information by explicitly disentangling identity and pose representation from a face image in fully self-supervised settings. The overall architecture of the IP-GAN framework is depicted in Zeno et al. (2019a), and consists of five parts: 1) the identity encoder network EI to extract the identity latent code; 2) the head pose encoder network EP to extract the pose latent code; 3) the generative network G to produce the final output image using the combined identity latent code and the extracted pose latent code; 4) the identity classification network C to preserve the identity by measuring the posterior probability of the subject identities; 5) the discriminative network D to distinguish between real and generated images. To train these networks Zeno et al. (2019a) proposed a learning method in order to learn complete representations in fully self-supervised settings. The learning method consists of two learning pathways, generation, and transformation. While the generation pathway focuses on mapping the entire latent spaces of encoders to high-quality images, the transformation pathway focuses on the synthesis of new face images with the target poses. This framework has many drawbacks and when it was trained on an unconstrained dataset of face images, it failed to learn disentangled representation of pose and identity. Besides, the learning scheme is very complex that makes it difficult for the GAN to converge.

3.1PFA-GAN

To simplify the proposed architecture in Zeno et al. (2019a), and according to the specific goal of the PFA-GAN in generating face images with new poses, we remove the Classification Network C, as there is no need to add a new task of face recognition to PFA-GAN, and preserving the subject identity in the generated face image is guaranteed by the use of the content loss function. To reduce the complexity of the learning method, we propose removing the generation pathway and focusing on the work of the transformation pathway, which consists of two sub-paths: reconstruction and transformation. The task of the head pose encoder network EP is to learn a pose representation as it has to isolate the pose information from the other information in a face image such as age, gender, skin color, and identity. Isolating pose information in unconstrained images is a challenging task. To facilitate it, we reduce the amount of input information for the network EP by replacing a face image with an image of its landmarks.

3.2Model Description

Let xS∈X be a source face image of a certain subject identity, and xD∈X be a driver face image to extract the target pose features. Our goal is to generate a new face image x´S of the subject of xS with the extracted face pose of xD. To achieve this goal, we assume that each image x∈X is generated from an identity embedding vector a∈A and a pose embedding vector p∈P. In other words, xS, xD are synthesized by the pair (aS,pS) and the pair ( aD, pD), respectively. As a result, the new face image x´S is generated by the pair (aS,pD).

3.3Framework Architecture

Fig. 1

The proposed framework architecture, pose encoder network, identity encoder network, generator, discriminator. Learning scheme from left to right: the reconstruction sub-path, the transformation sub-path.

The proposed framework consists of the following four components, see Fig. 1:

4.1Reconstruction Sub-Path

The reconstruction pathway trains the generator G, the pose encoder network EP and the discriminator D. Here the identity encoder network EA is not involved in the learning process since the network EP learns the pose representations of the driving face images and the generator G tries to synthesize a face image using the driving pose embedding vector, while the identity embedding vector contains random values. Hence, given a random noise vector from noise uniform distribution az∈Z and the pose embedding vector of the driving landmark image pD=EP(lD), we concatenate them in the latent space z=[az,pD] and feed them to the generator which aims to generate a realistic face image xz=G(az,pD) under the driving pose latent vector pD. Similar to the original GAN work (Huang et al., 2007), the generative network G and the discriminative network D compete with each other in a two-player min-max game. While the discriminator D tries to distinguish real images from the output of G, the generator G tries to fool the network D. Specifically, D is trained to differentiate the fake image xz from the real one xD. This D minimizes:

4.2Transformation Sub-Path

The transformation sub-path trains the networks EA, G, and D, but keeps the pose encoder network EP fixed. The output of the EA network should ensure preserving the identity of the source face image. We introduce a cross reconstruction task to make EP and EA disentangle the pose from the identity information. More specifically, we sample a real image pair (xSi,xDi) that shares the same identity but different appearance, poses, and facial expressions. The goal is to transform xSi to a new face image x´Si where its pose matches the pose of the driving face image xDi. To achieve this, EA receives xSi as an input and outputs a pose-invariant face representation aSi, while EP takes lDi landmarks image of xDi as an input and outputs a pose representation vector pDi. We concatenate the embedding vectors and feed the combined vector, zi=(aSi,pDi)=[(EA(xSi),EP(lDi)] into the network G. The generator G should produce x´Si, the transformation of xSi. D is trained to distinguish the fake image x´Si from the real one xDi. Thus, D minimizes:

4.3The Overall Loss Function

The final loss function is a weighted sum of all losses defined in Eqs. (1)–(10):

5.1Dataset

The PFA-GAN is trained on a subset of the MS-Celeb-1M (Guo et al., 2016) dataset, which contains about 5M images of 80K celebrities with unbalanced viewpoint distributions and with a very large appearance variation (e.g. due to gender, race, age, or even makeups). We use 36K face images belonging to 528 different identities, while no pose or identity annotations are employed in the training process. For each face image, we first detect the facial region using the multi-task cascaded CNN detector (MTCNN) (Zhang et al., 2016) and then align and resize the detected face to 128×128 pixels.

5.2Implementation Details

We use the same implementations of the generator G and the discriminator D in IP-GAN that were introduced by Tian et al. (2018). For the pose encoder network EP and the identity encoder network EA, we use ResNet50 (He et al., 2016) network architecture, where the skip connections in the network allow to learn the desired representation (e.g. the identity or the pose) since the performance of the upper layers will be at least as good as the lower layers. The following parameters were used: λcnt=1, λadv=0.005, and λreg=0.001. The values of the random noise az are in the range [−1,+1]. To implement the model, a set of Pytorch deep learning tools was used. The batch size was set in 16 and one Nvidia graphic card (GTX 1080 Ti) was used. The Adam optimizer (Kingma and Ba, 2015) was used and configured with the learning rate of 0.0005, and the momentum of [0,0.9].

5.3Interpolation of Pose Latent Space

In this section, we demonstrate that a pose of the generated face images can be gradually changed with the latent vector. We call this phenomenon face pose morphing. We have tested our model on the selected subset from the MS-Celeb-1M (Guo et al., 2016) dataset. We first choose a pair of images xS and xD, and then extract the pose latent vectors pS and pD using the pose encoder network EP. Then, we obtain a series of pose embedding vectors p˜i by linear interpolation, i.e.:

Fig. 2

Interpolation of the pose latent space.

5.4Visual Data Augmentation Strategies

The traditional visual data augmentation methods alter the entire face image by transferring image pixel values to new positions or by shifting pixel colours to new values. These generic methods ignore high-level content such as moving the head or adding a smile, so in this section, we show the effectiveness of using our model as an alternative face specific augmentation method. The ability of the PFA-GAN model to perform a controlled synthesis of a face image allows enlarging the volume of data for training or testing by generating new face images with new poses. So, using our proposed model, for each image in the dataset, an unlimited number of images can be generated for the same identity subject with a great variety of face poses. Assuming the original dataset is R, pose face augmentation can be represented by the following transformation:

Fig. 3

Face image examples from the original and augmented datasets. From left to right: the original dataset R, the augmented datasets T1, T2, T3 in the second, third and fourth columns, respectively.

5.5Face Verification Task

In this subsection, we evaluate whether the augmented datasets will improve the performance of the face verification task or not. In general, face verification needs the following steps: training a convolution neural network classifier on a dataset, then using it as a feature extraction network to extract the embedding vectors for a pair of face images from testing datasets. Next, the extracted two vectors are sent to the distance function to calculate the similarity between them, and according to the threshold, the function judges whether it is the face of the same person or not. Two classifiers are used, C1 and C2, the backbones Restnet50, Restnet101 are chosen for the C1, C2, respectively. Both classifiers use ArcFace (Deng et al., 2019) and Focal loss function. We use different datasets for face verification, such as LFW (Huang et al., 2007), CFP-FP (Sengupta et al., 2016), AgeDB (Moschoglou et al., 2017), CFP-FF (Sengupta et al., 2016) and VGGFace2-FP (Cao et al., 2018b). Apart from the most widely used LFW dataset, we also report the performance of our augmentation model on the recent large-pose and large-age datasets (e.g. CPLFW. Zheng and Deng, 2018 and CALFW, Zheng et al., 2017). Table 1 shows the statistics of both training and testing datasets in a verification scenario.

We feed the augmented datasets to the classifiers (C1,C2) for training, then we use the learned model to extract embedding vectors for each image in the testing datasets. Therefore, the verification accuracies are calculated as shown in Table 2 and Table 3, and it is clear, that the verification accuracy is higher than it on the dataset without augmentation. For instance, the verification accuracy on AgeDB increased from using the CNN model (RestNet50) that trained on the augmented dataset (Aug-D3). As a comparison between the proposed augmentation strategies, the difference between augmented datasets (T1,T2,T3) is that the pose face in each of them is transformed from the original dataset baseline with a different degree of change, from small, as in the case of the T1, to randomness, as in the case of the T3. Consequently, by comparing the tables’ results, it can be noted that a small change in pose transformation improves the results of face verification, but when the change is random as in the augmentation strategy (Aug-D3), the results of verification using the more deeper feature extractor Resnet101 are the best. Since on the publicly available dataset CFP-FP, the increase in verification accuracy has been achieved up to 4.5%.