Emotion – Artificial & Biological Vision

Introduction to Facial Emotion Recognition Models and their Biases

by Brenda Aguilar & Alyssa Wu

Question/ Background

How well can neural networks recognize emotions in human faces? What key facial features are used by these models to identify and categorize emotion? What are the societal implications of these processes and results? Are these models racist? Let’s find out!

Known:

Facial emotion recognition consists of 2 steps: feature extraction and emotion recognition
FER is challenging due to the subjectivity of emotion recognition by human visual systems, though there is a general conception of an emotion’s characteristics

Unknown:

Unless you delve directly into a neural network’s architecture, training data, and training methods, the cause of bias in public (but frequently used) models often exists in a “blackbox”

Pre-existing methods (models used) and findings/ shortcomings:

	Akhand et. al 2021	Wei 2021	Raina et. al 2022
Methods	CNNs have been used because of their extraction properties, this study uses a deep CNN that uses a Transfer Learning (TL) technique (where a pre-trained DCNN model is adopted by replacing its dense upper layer(s) compatible with FER, and the model is fine-tuned with facial emotion data).	Incorporation of saliency maps to replicate how human’s identify salient features to determine emotion. The preprocessing of an image is performed to detect facial features, crop and align them. These input images are then fed into 2 channels: one that calculates saliency and normalizes it and the other that divides the CNN into low, medium and high convolution parts.	Artificially generated faces w/ FaceGen Modeller software, standardizing facial muscle activation to represent different emotions, and evaluate three publicly available FER models: 1. Extended MTL Model for Pain-Estimation 2. Multi-task EfficientNet-B2 Emotion Classification 3. Intel’s Open Visual Inference and Neural Network Optimization (OpenVINO) *Trained on AffectNet, where 64.4% of the faces are white
Findings and short-comings	Findings: -remarkable accuracy on test sets KDEF (96.51%) and JAFFE (99.52%) Shortcomings: – Unable to identify emotions from high-resolution images with standard shallow CNNs	Findings: -this method makes up for the visual guidance of a CNN and solves having complex background noise Shortcomings: – CNNs only identifies statistic characteristics from images	Findings: – Noticeable disparities in emotional classifications and intensities within “African-Black” vs “African-White” not noticed within “European-Black” vs “European-White,” indicating both racial and skin-tone biases Shortcomings: – Unable to detect source of bias in models

**Figure 3 in Akhand et. al 2021.** Illustration of the proposed FER system based on transfer learning in deep CNN.

**Figure 2 in Wei 2021**. Illustration of the proposed Saliency Maps-CNN based FER strategy. The input images are sent into the preprocessing module at first and the features are extracted from both Channel 1 and Channel 2. The output normalized saliency maps are multiplied by the preprocessed input images. In Channel 2, the CNN network is divided into low-level, median-level and high-level convolutional parts. The residual connection technique is adopted to combine the median-CNN, the high-CNN and the saliency features.

**Figure 1 in Raina et. al 2022:** Synthetic face images used for experimental evaluations. Each row shows the same expression for each of African, African White, European, and European Black settings respectively. (First row) This row involves the activation of AU7-Lid Tightener (100% activation) with rest of the AU activations to be set at 0%. (Second row) In this row AU10-Upper Lip Raiser is activated to 100% for all face settings. (Third row) In this row AU25-Lips part is activated to 100% for all face settings. (Fourth row) In this row AU26-Jaw Drop is activated to 100% for all face settings.

Why should you care?

Facial emotion recognition (FER) neural networks are still in early stages of development; however, proponents propose that once sufficient, FER networks can be applied in areas like wellness and clinical care. FER models would be used to support or incite diagnosis, intervention, and treatment in mental health settings or integrated into health IT solutions, as seen with the Fast Facial Emotion Monitoring (FFEM) tool.

Emerging technologies often carry an air of objectivity; however, neural networks detect and recognize patterns in the data they are trained upon. If the data carries biases on age, gender, race, etc., then the network will also contain those biases. Therefore, while FER neural networks have the possibility of making mental health care more accessible, if models contain significant racial bias, this would exacerbate race-based misdiagnosis and underdiagnosis (e.x. a Black person is ~2.4 times more likely to receive a diagnosis of schizophrenia than a white person).

Citations

Akhand, M. A. H., et al. “Facial Emotion Recognition Using Transfer Learning in the Deep CNN.” MDPI, Multidisciplinary Digital Publishing Institute, 27 Apr. 2021, www.mdpi.com/2079-9292/10/9/1036.
Raina, Ritik, et al. “Exploring Biases in Facial Expression Analysis Using Synthetic Faces.” University of California San Diego, 3 Oct. 2022, https://pages.ucsd.edu/~desa/NeurIPS_SyntheticData4ML_workshop_paper.pdf.
Wei, Qinglan. “Saliency Maps-Based Convolutional Neural Networks for Facial Expression Recognition”, IEEE ACCESS, 21 May 2021, ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9438697.