Artificial & Biological Vision

Introduction to Color in Convolutional Neural Networks

by Charlotte Imbert & Yufei Meng

What is color in neurobiology?

We have learned that we live in a colorful world, the rainbow has 7 colors, and that people can sometimes be colorblind. But what is color, and how is color represented biologically? 

In our vision system, we have two major types of photoreceptors, the rods and cones. The majority of our photoreceptors are rods, which perceive the light/darkness of objects, but lack fine details of objects and specifically color. Cones, on the other hand, are the cells responsible for perceiving colors. There are three types of cones: the Long (L), Medium (M), and Short (S) cones, which are responsible for receiving red, green, and blue, respectively. There are two important features that we hope to introduce to you here about color perception that will become important as we move along. 

(1) Color is a spectrum. We know that light consists of photons that behave simultaneously as rays and waves. As a result, different colors physically just mean lights with different wavelengths. Correspondingly, our biological vision also perceives color in a spectrum manner.

As shown in the figure above, our cones also have absorption curves that will respond to a range of wavelengths. For example, if an incoming light has a wavelength around 570-580 nm, then both the L and M cones will be activated, which means we cannot distinguish color just by one type of cone. And this is exactly how yellow (570-580 nm) is represented – by combining the processing of the L and M cones. There are also colors like blue that can be perceived mainly by the S cone.

(2) Color opponency. If you remember the color wheel, you probably also recall that complementary colors are those with the highest contrast, or the opposite in the color wheel. Interestingly, our visual system also perceives color in this manner. We have L cone and M cone that are most responsive to red and green, respectively. Together they form a red-green contrast to the brain. Similarly, as L and M cones process yellow, it forms another contrast with blue perceived by the S cone in the brain. Though this contrasting relationship does not strictly follow the color wheel, it still illustrates how our biological system perceives vision with opponent colors. 

How is the human color system different from that of other animals?

Different animals have different color processing systems, which make each animal view the world differently from how humans would imagine. Dogs, for example, are dichromatic compared to humans’ trichromatic color vision, which means that dogs can only perceive blues and yellows. Colors such as green in dogs’ view were thus suspected to be some shades of grey or white. In contrast, insects with compound eyes can have as many as 8 distinct photoreceptor types, allowing the animals to distinguish colors very well. We may hypothesize that more cone types would lead to more fine-tuned color processing. However, this is not always the case. The mantis shrimp, for instance, is a well-known organism with 12 photoreceptor types. Naturally, we might think that the mantis shrimp will perceive the most types of colors in the world. However, Thoen et al. found through behavioral experiments that instead of a multiple color-opponent system, each photoreceptor bins colors into 12 separate channels; each channel recognizes one specific color, but there is no combination of sensory processing between these bins  (Thoen et al., 2014). As a result, the mantis shrimp’s color perception is not as well-defined as we hypothesized.

Biological versus artificial vision

Convolutional neural networks (CNNs) are a type of deep neural network. Their most common use is for object recognition. Their architecture includes filters, which have many similarities to biological receptive fields. For this reason, CNNs have previously been used to model biological vision, as when trained to do tasks that animals with brains do, they display brain-like activity. This has led to questions regarding the internal representations of CNNs. What does ‘red’ look like for a CNN? How different is that from what ‘red’ looks like for a human?

Some of the already known similarities between artificial neural networks and the human brain include:

  • Both consist of individual units: neurons in the brain, nodes in ANNs
  • In both cases, units are connected to each other: via synapses in the brain, and via edges in ANNs
  • The strength/efficacy of these connections is represented by weights
  • Both can be trained to perform tasks: synaptic plasticity in the brain, modifying weights in an ANN

Perhaps most relevant here, CNNs mirror the parallel hierarchical processing of biological visual systems: both artificial and biological systems can extract information about color, shape, motion, and depth from the same input. CNNs do this via filters, known as kernels, which are slid over an input image to extract relevant features (think edges, lines, textures, etc.) Does this sound similar to anything in the human visual system? V1 simple cells, which respond selectively to line orientations, or V1 complex cells, which respond to specific colors or movement!

For this project, we’re particularly interested in comparing how humans and machines encode color.

How is color represented in computational models?

Rafegas and Vanrell’s 2018 paper, ‘Color encoding in biologically inspired convolutional neural networks’, was really the spark for this project as it provided an explanation of how color is represented in CNNs. The authors measured and classified the color selectivity of individual neurons across the convolutional layers of a VGG-M CNN, pre-trained on the ImageNet dataset. Color selectivity is the property of a neuron that shows high activation when a particular color appears in the input image, and low activation when that color is absent. They proposed a color selectivity index (CSI), which estimates the variation between a neuron’s activation to color patches and its activation to corresponding grayscale patches. The method in this paper we found most intriguing was the Neuron Feature (NF), which is a weighted average of the top 100 image patches that maximally activate each neuron. How the NF looks is a decent representation of the spatio-chromatic features that each neuron likes the most.

The CSI classified neurons as color-selective, low color-selective, and non-color selective, based on thresholds determined from observed neuron selectivity over its NF and through a chromatic transformation of the same image patch. Within the color-selective subcategory, neurons are further split into single (showing selectivity to one color) or double (showing selectivity to a pair of colors). Using a Gaussian mixture model and an EM algorithm, the authors obtained one representative hue for each single color neuron, and a pair of representative hues for each double color neuron. No neuron was selective to more than 2 colors.

Even further, double color-selective neurons were then split into opponent or non-opponent. Neurons exhibiting high selectivity to two opposing hues (based on the color wheel) were classified as opponent neurons. It is easier to focus on the neuron’s activation patterns rather than directly measuring excitation and inhibition, which is very challenging to assess in ANNs. 

So what did they find?

  1. In the first convolutional layer, they identified neurons that responded selectively to color pairs such as Red-Cyan, Blue-Yellow, Magenta-Green, and Black-White. This is similar to the color opponency mechanisms in biological visual systems
  2. As the network depth increased, the prominence of these opponent axes diminished. By the second layer, opponency was largely reduced to a single axis, Bluish-Orangish (the two most frequently occurring colors in ImageNet)
  3. In the deeper layers 3 to 5, neurons became more specialized and responded to specific color-object combinations (e.g. orangish faces, blue skies, or blue objects against green backgrounds). This is pretty similar to the hierarchical processing in biological visual systems!

Our question

We were excited by this paper’s findings that certain neurons are color-selective. Since the input images to this model were static, we wondered how robust or stable these learned color representations are when the input image is subjected to controlled color transformations. So, we want to build on this foundation by testing the robustness of color representations in a pre-trained CNN. Specifically, we want to know: How do internal color representations in a CNN respond to systematic alterations in image properties such as hue, brightness, and saturation? Can these transformations ‘fool’ the model’s color identification/perception? Essentially,  given the impressive performance of the existing model on static images, we want to test how persistent the model’s color representations are when transformations are applied to its inputs. Since biological organisms need to be able to encode color in a variety of different environments, their color representations must be able to be resilient to these changes. Let’s see if the same holds for CNNs!

Works cited