Artificial & Biological Vision

Results: Biases in Facial Emotion Recognition Models

by Brenda Aguilar & Alyssa Wu

From now on, the model built from fine-tuning Dense-Net 161 with images of faces from KDEF will be referred to as the KDEF Model and the model is fine-tuned with images of faces from FER-2013 will be referred to as the FER-2013 Model.

Following the processes of Akhand et al., each time the entire training portion of the KDEF/FER-2013 datasets were passed through the model for fine-tuning (or for every epoch), we assessed the training and test accuracy of that given model. The weights of the final fine-tuned KDEF Model and FER-2013 Model were selected based on the best performing epoch (where testing accuracy was highest).

The best testing accuracy for the KDEF Model was 96.67 percent. At that same epoch, its training accuracy was 99.40 percent. Such high testing accuracy follows Akhand et al’s findings in their 2021 findings. 

Figure 1. A series of 8 incorrectly categorized images by the KDEF model, made on the KDEF training set, are captured above. Of the eight, six of them had disgust as either the incorrect prediction or the original image’s classification, which is worth exploring as a connection.

Some of these incorrect predictions are rather understandable. We see that the KDEF Model mislabes faces with gaping mouths and or widened eyes as surprise, an understandable mistake as these features are often characterized by surprise. 

The best testing accuracy for the FER-2013 Model was 58.45 percent, though for that same epoch, the training accuracy was at 99.45 percent.

Figure 2. The images below are the incorrectly labeled images of the FER-2013 Model on the FER-2013 training set. Similar to the KDEF Model’s output, it appears that disgust comes up for this specific set of input.

This discrepancy between train and test accuracy in the FER-2013 Model can be explained by the variability of the images in the FER-2013 dataset. Unlike the KDEF and JAFFE datasets (the two dataset in which Akhand et al. test their transfer-learning methods on), the faces in FER-2013 are not standardized in lighting, cropping, or context. Both KDEF and JAFFE are rather homogenous datasets, where models of a singular race are asked to pose according to seven emotions against a standardized background and standardized head orientations and wearing no face-obscuring accessories (like glasses). The FER-2013 dataset is instead composed of miscellaneous images across Google Image Search, introducing a sense of randomness. Thus, while the FER-2013 Model can find the weights to accurately label the images it is trained on to near perfect accuracy, the model is inadequate in generalizing across the FER-2013 dataset due to its unpredictability and inconsistencies. 

This, however, has very interesting applications when the KDEF Model and the FER-2013 Model are evaluated on their abilities to accurately label the emotions across the JAFFE dataset. Out of the 213 images in JAFFE, the KDEF Model accurately labeled 58 out of the 213 images, a 27.23 percent accuracy rate. This is a stark drop from the testing accuracy the KDEF Model has when evaluated with test images from KDEF (96.00 percent). Ironically, the FER-2013 Model, which has a much lower testing accuracy when evaluated on test images from FER-2013 (at 53.59 percent), was able to accurately label more facial emotions in JAFFE. The FER-2013 Model accurately labeled 73 of 213 images, a 34.27 percent accuracy rate. The performance between the FER-2013 Model’s best training and JAFFE accuracy are much closer than their corresponding accuracies for the KDEF Model. 

Figure 3. This set of images is the output of incorrect predictions from the KDEF Model when tested on the JAFFE dataset. As mentioned earlier, it had an approximate 21% accuracy rate, proving to be quite low and inaccurate.
Figure 4. This dataset depicts some of the incorrect predictions the FER-2013 Model made when tested on the JAFFE dataset. Though still low, the FER-2013 Model had an approximate 34% accuracy rate, effectively being more accurate than the KDEF Model.

For the DiverseFACES dataset, which model performs better is reversed. The FER-2013 Model was able to correctly label 130 out of the 432 (30.09 percent). The KDEF Model was able to correctly categorize 152 images (35.19 percent accuracy).

Figure 4. The 8 images above depict the FER-2013 Model’s incorrect predictions on the DiverseFACES dataset emotions. They are a small representation of the 30% accuracy rate by the FER-2013 model.
Figure 5. The images above are from the incorrect predictions that the KDEF model made on the DiverseFACES dataset, which had an ultimate 35% accuracy rate.

These findings highlight a shortcoming in Akhand et al.’s methods and results. While their transfer learning method has been able to produce Facial Emotion Recognitions models with unprecedented accuracy rates, their high test and accuracy rates only apply to highly regulated FER datasets like KDEF and JAFFE. In datasets like KDEF and JAFFE, the randomly split training images and the test images likely share the same human models and definitely share the same backdrop, head orientation, gaze, image quality, etc.

Even though the diversity of the FER-2013 dataset causes the FER-2013 Model to see a low testing accuracy when tested on the training images in FER-2013, it is the diversity of FER-2013 dataset that allows the model to better generalize. Therefore, we see better emotion recognition abilities of the FER-2013 Model for the JAFFE and DiverseFACES dataset in comparison to the KDEF Model. 

The confusion matrices below highlight the distribution of predictions across each model on each dataset, answering questions like: Does one model incorrectly think subjects tend to be one particular emotion most of the time? Does one consistently mis-categorize sadness as neutral?

Figure 6. The confusion matrix above visualizes the discrepancies between the FER-2013 Model predictions and the DiverseFACES emotion labels for each emotion. 

We see that for the FER-2013 Model on DiverseFACES, the model was only able to consistently categorize happiness and neutral. However, because all other emotions were presumed to be neutral, it may not be that the FER-2013 Model is accurately categorizing neutral, but instead defaulting to neutral as its prediction. The FER-2013 Model incorrectly labels 3 of the faces as surprise, reflecting the fact that none of the faces in the DiverseFACES dataset were expressing surprise.

Figure 7. The confusion matrix above reflects how each emotion in the JAFFE dataset was mispredicted by the FER-2013 Model. 

Even though the overall accuracy of the FER-2013 Model on the JAFFE and DiverseFACES datasets are similar, we can see that the model is better at categorizing emotions in the JAFFE dataset. Here, the model also shows some accuracy in categorizing sadness and surprise as well (although similarly, it completely fails for emotions like fear, anger, and disgust). We see that while there is still a common “default categorization” of faces as neutral, a lot of faces across emotions are being characterized as sadness. 

Figure 8. The confusion matrix above reflects how each emotion in the JAFFE dataset was mispredicted by the KDEF Model. 

We see that the KDEF model has a tendency to classify emotions in the JAFFE dataset as surprise. However, there is some level of accuracy in classifying happiness, disgust, and anger. 

Figure 9. The confusion matrix above reflects how each emotion in the DiverseFACES dataset was mispredicted by the KDEF Model. 

Across the diagonal, we can see significant accuracy for the KDEF model to classify happiness, neutrality, sadness, and anger. There are significant mislabelings of fear as disgust and disgust as anger. Notably, there is a tendency for the KDEF Model to misclassify emotions as anger, a result which does feed into preexisting notions of Black and Latino aggressiveness.

Our research question was initially posed in the context of uncovering possible racial bias, and while it is true that the more racially diverse dataset FER-2013 performs better on labeling human emotions for the faces in JAFFE, it does not outperform the KDEF model in categorized images from DiverseFACES. 

For future research, it may be interesting to consider a third FER model that has been created by fine-tuning with a dataset which expanded upon the DiverseFaces dataset to include more than Black and Latino faces–a dataset diverse across race, age, and sex, but is similarly standardized with a select number of models, head orientation, backdrop, etc like KDEF and JAFFE. While such a dataset does not exist, it would allow researchers to draw conclusions on how racial diversity alone may cause biases in FER models. 

Further evaluations of FER models may observe how age, race, and gender impact a model’s misclassifications. Paired with a more social sciences approach, research may reveal how FER models may reinforce emotional stereotypes (e.x. Seeing if a model tends to over-label POC faces with negative emotions).