Developing and Evaluating Models for Biases in Facial Emotion Recognition
by Brenda Aguilar & Alyssa Wu
Building a Facial Emotion Recognition Model
For developing a Facial Emotion Recognition (FER) system, we will use the same methods as in the 2021 paper “Facial Emotion Recognition Using Transfer Learning in the Deep CNN” by Akhand et al. The authors created a FER system by taking a pre-trained Deep Convolutional Neural Network (DCNN) that performs object recognition in images and doing Transfer Learning, a process which replaces the DCNN’s dense upper layer(s) so that its classification categories are now emotions rather than object names and fine-tuning the model with facial emotion data.
Akhand et al. verified their proposed FER system on eight different pre-trained DCNN models (VGG-16, VGG-19, ResNet-18, ResNet-34, ResNet-50, ResNet-152, Inception-v3 and DenseNet-161) and on two facial emotion datasets, KDEF and JAFFE. Models were trained with 90% of available images from KDEF and tested with the remaining 10% in KDEF, and the same process with the JAFFE dataset (Akhand et. al 2021). We build upon their methods to then test the racial implications of FER models trained on diverse and non-diverse datasets.
We will use DenseNet-161 as our pre-trained DCNN. DenseNet-161 is an object-categorizing DCNN that was trained on ImageNet-1k, a dataset which consists of over one million images categorized into 1,000 objects. When Akhand et al. applied their transfer learning methods to DenseNet-161, they were able to achieve high accuracies on test sets of KDEF (96.51%) and JAFFE (99.52%) (Akhand et. al 2021). We will therefore also be utilizing the same image transformations–random horizontal image flips, image rotations, and image sizing–to best mimic Akhand et al.’s methods to ideally achieve similar levels of accuracy.
We will then apply Akhand et. al’s transfer learning methodology to create two models: one which is fine-tuned with images of faces from KDEF (a racially and age-wise non-diverse dataset) and another is is fine-tuned with images of faces from a much more diverse dataset FER-2013.
Facial Emotion Datasets Used
FER-2013: This dataset consists of 35,887 grayscale images that are categorized into seven emotions (anger, disgust, fear, happy, neutral, sad and surprise). These images vary in race, age and profile (facing forward, left or right). This dataset was created at the University of Montreal for machine learning research by using Google search to collect images that corresponded with emotion-related keywords, one example being “blissful”, that were combined with features, like age, ethnicity and gender, to create queries (Anonymous, n.d.). The leading 1000 images for each query were then given a “boundary box” around the detected facial features using OpenCV (programming library) face recognition. These images were then manually categorized into their correct emotion label and cropped, ridded of duplicates, resized to 48x 48 pixels and converted to grayscale. Finally, these images were confined to a subset that categorized the detailed keywords into the seven emotions previously mentioned (Anonymous, n.d.).
KDEF: The KDEF (Karolinska Directed Emotional Faces) dataset consists of 4900 images of faces that are all white and estimated to be around 20s but differ in profile. These images are classified as one of seven emotions, which were previously mentioned: anger, disgust, fear, happy, sad, surprise and neutral. The emotion portrayed in these images were contrived and collected in a lab setting for the initial purpose of researching memory, emotion, perception, attention and backward masking (Akhand et al., 2021). Despite the dataset’s initial goal not being centralized around emotion identification, it still proves to be helpful in FER because of its variance in profiles. Though the variance in profiles makes it increasingly challenging to distinguish between emotions, it demonstrates a network’s efficiency in FER.
JAFFE: The JAFFE (Japanese Female Facial Expression) dataset contains 213 grayscale images, each depicting one of the previously mentioned 7 categories of emotion from 10 different Japanese women for each category. All images within this dataset are from the front-facing perspective and, similar to KDEF, were obtained in a controlled environment where emotions were instructed (Akhand et al., 2021). More specifically, the images were collected for the psychology department at Kyushu University in Japan. This dataset’s relevance lies in its small size and non-white racial unity, as these attributes both present deviations from training sets like KDEF and FER-2013, and has been of frequent use by various studies on FER.
Diverse FACES: This dataset consists of 432 images that display six different emotions (anger, disgust, fear, sadness, happiness and neutral) from 36 different Black/ African American and Hispanic/ Latinx individuals. These images were collected by following the FACES protocol (photographing individuals showing 6 different emotions through 3 phases: emotion induction, personal experience and controlled expression) with people of the Black and Latinx communities (Ebner & Lindenberger, 2010). Extending past the collection of images, the images were used to gather data on how people in the images would be ranked and sorted between different criteria: age, race, emotion, representativeness of race and trustworthiness (Gonzalez & Seaman, 2024). Diverse FACES plays an important role in FER, as it assists in depicting whether or not a network can account for non-white emotions accurately and explores whether or not there’s a racial bias.
Bias Evaluation Stage
We will compare the accuracies of each of the two neural networks (one fine-tuned with KDEF, the other fine-tuned with FER-2013) when they are tested with images from facial emotion datasets DiverseFaces and JAFFE. We will evaluate whether or not the model fine-tuned on FER-2013 (the more diverse dataset) outperforms the model fine-tuned on KDEF when asked to classify the emotions of images from DiverseFACES and JAFFE.
For each dataset each model is assessed on, we will create a confusion matrix. This chart will provide a breakdown of the model’s predictions compared to the actual labels, displaying the types and distribution of errors made by each model. Images the model misclassifies will be saved alongside the model’s incorrect prediction and the actual label. Images labels in the DiverseFACES dataset also store images about the pictured face’s age, race, and gender, which can allow for deeper quantitative (distribution of mistakes) and qualitative (social implications) analysis on the biases each model may contain.
References
- Akhand, M. A. H., et al. “Facial Emotion Recognition Using Transfer Learning in the Deep CNN.” MDPI, Multidisciplinary Digital Publishing Institute, 27 Apr. 2021, www.mdpi.com/2079-9292/10/9/1036.
- Anonymous. (n.d.). Troubled datasets.FER2013.https://recognitionmachine.vandal.ist/p/fer2013/
- Ebner, N. C., Riediger, M., & Lindenberger, U. (2010, February). Faces–a database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior research methods. https://pubmed.ncbi.nlm.nih.gov/20160315/ Gonzalez, S. M., & Seaman, K. L. (2024). Diverse FACES – A database of facial expressions in young, middle-aged, and older individuals from underrepresented racial populations. OSF. https://osf.io/preprints/psyarxiv/mey2h_v1