Artificial & Biological Vision

Color Representation in CNNs: Results & Discussion

by Charlotte Imbert & Yufei Meng

Recap

You may or may not remember the previous blog posts, which we all understand :), so here is a quick recap of our project’s main aims.

  1. We are mainly inspired by the Rafegas & Vanrell paper in 2018, which thoroughly examines the representation of color and color selectivity in CNNs.
  2. We are further interested in the robustness of the neuron color selectivity proposed in the paper. Therefore, the main aim of our project is to
    1. Replicate the neuron feature maps and the color selectivity indices of neurons in the paper, specifically those at the first two convolutional layers, because the paper has suggested a higher percentage of color-selective neurons in earlier layers.
    2. Examine the robustness of the neurons’ color selectivity using transformed image datasets.
    3. Compare the pretrained model’s performance with a model fine-tuned to the transformed images to investigate how the neuron characteristics could potentially change.

Method Implementation

VGG16: The paper originally used VGG-M, a model from Chatfield et al., 2014. However, we have not found publicly available code from the referenced paper. Instead, we used VGG16, which is also a pretrained model on ImageNet and is readily available in the PyTorch module.

Neuron Features (NF): The convolutional neural network (CNN) is named as such because it comprises convolutional layers as the main processing layers, which contain the artificial ‘neurons’ as interconnected units to process inputs. Therefore, the weights and activation patterns represented in these neurons can give us information on the role of these neurons. This is why the original Rafegas & Vanrell paper defines neuron features (NF), which is:

Essentially, NF is a way to quantify and visualize the activation patterns of a neuron towards the image patches. More specifically, the NF is the normalized sum of relative activation (wj,i,L), which is already normalized against the maximum activation among the top patches, in the range of Nmax images. 

Color Selectivity Index (CSI): This is probably the most important concept both for the paper and our project. This index, again, is dependent on the relative activation of neurons (wj,i,L). The numerator is the sum of activation values of the neuron towards the gray-scaled images, while the denominator is the sum of activation values of the same neuron towards the RGB images, and CSI is computed as 1 minus the quotient. Therefore, the greater the difference in a neuron’s activity between colored and gray images, the larger the CSI value will be. 

For both NF and CSI, we have tried several methods to implement them. At first, we tried to define the functions ourselves, but the gray scale image conversion was not aligning with the paper’s description, which requires a conversion from RGB to gray scale to OPP (the opponent space). Later, we found Nefesi (https://github.com/CVC-CIC/nefesi), (neuron-feature-selectivity-index), a GitHub repository with source code for implementing some of the paper’s methods. We based our scripts on the codes from Nefesi and modified them to match our later experiments.

Color Transformations

Since we do not have access to the original ImageNet data, all the images used in our analysis were from the Tiny ImageNet training set. Due to computational resource limitations, we only used a subset of the training images. The subset corresponded to 10 object classes, each containing 500 images, comprising a total of 5000 images. The classes were: goldfish, European fire salamander, bullfrog, tailed frog, American alligator, boa constrictor, trilobite, scorpion, black widow, tarantula.

After implementing the paper’s related functions and getting similar results on the original dataset, we also transformed the images as testing conditions to examine the robustness of neurons’ color selectivity. Thus, the original images in the subset were transformed as follows:

  • Brightness: decreased by 100%, decreased by 50%, increased by 50%, increased by 100%
  • Contrast: decreased by 100%, decreased by 50%, increased by 50%, increased by 100%

This was done using the digital image processing functions from https://github.com/TheAlgorithms/Python/tree/master/digital_image_processing, specifically in the change_brightness.py and change_contrast.py scripts.

A batch of image transformation examples can be seen below:

Figure 1. Original and transformed images from the goldfish class and the American alligator class

Model Fine-tuning

To further compare the effects of training on neurons’ color selectivity, we also fine-tuned the original VGG16 model by training its conv1, conv2, and final classification layer using our transformed images. We hypothesize that the fine-tuned model’s neurons should display higher robustness than the pretrained model. 

Results

1. Successful implementation of neuron features and color selectivity

    Despite the trials and errors, our results have shown a correct representation of neuron features as the sample image (Fig. 2). 

    Figure 2. Sample neuron feature grid from conv1_1, untransformed images.

    Similarly, we also managed to gather the CSI values for each neuron in our layers of interest, conv1_1, conv1_2, conv2_1, and conv2_2. If you are interested, please check our github repo for the code and the .CSV results (https://github.com/cameraperture/colorcnn/tree/main)!

    An interesting difference from the paper is that we did not find any non-color-selective neurons, and that the vast majority of our neurons were high-color-selective. In fact, on the original (untransformed) images, all the neurons were high-color-selective. The only CSI values below 0.25 (the threshold for high-color-selectivity) were observed after brightness image transformations (Figure 7). This surprised us, since in Rafegas & Vanrell’s paper, all three types of selectivity were well-represented throughout the network. There is a possible explanation for this discrepancy. Our dataset is only a subset of the Tiny Imagenet training set, whereas the paper used the entire ImageNet training set. Also, the Tiny Imagenet only consists of 64×64 images of the original 224×224 ImageNet images. We had to resample the images from 64×64 to 224×224, which may have altered pixel values in a way that affects CSI.

    2. Color-selective neurons show robust activation patterns

      Next, we compared both the changes in NFs and CSIs of individual neurons when facing image transformations. Overall, the color-selective neurons have shown relatively stable neuron feature characteristics with shifts towards the corresponding transformation. For example, when brightness goes higher, the high color-selective neurons will demonstrate the corresponding lightening of the NF (Fig.3-4, left panels). Similarly, when facing contrast transformation, the NF patches also change accordingly to a more or less saturated color representation (Fig. 3-4, right panels). While our resulting neurons are mostly high color-selective and have demonstrated the behaviors above, there are a few relatively less color-selective neurons (i.e. CSI still above 0.25 but smaller than the majority of the neurons in the same layer) that do demonstrate relatively different NF and changes. For example, the NF patches generally show less diversity in the colors represented, and the less color selective neurons also seem to show less change in responses facing different transformations (Fig. 4). This matches the results discussed in the Rafegas & Vanrell paper, where low color-selective neurons might be orientation-selective or shape-selective, thus showing less responses towards the change of colors. In the case of our sample neuron shown here (neuron 24), for instance, it seems like it might represent an orientation bar more than color (Fig. 4).

      Figure 3. Neuron features (NFs) and color selectivity index values (𝛼) for 2 neurons in layer conv1_1 across brightness and contrast transformations. Neuron 24 is technically a high color selective neuron (CSI > 0.25), but it has relatively lower CSI values than the other neurons in the layer. Neuron 46 is a highly color selective neuron with a higher CSI value (CSI>0.5).
      Figure 4. Neuron features (NFs) and color selectivity index values (𝛼) for 2 neurons in layer conv2_1 across brightness and contrast transformations. Neuron 68 is relatively much less color-selective than neuron 91, which is a very highly color-selective neuron.

      3. Color-selective neurons demonstrate differing activation patterns for brightness and contrast transformations

        Next, we quantitatively analyzed how neuron activity changes under different image transformations. As shown in Fig. 5, neurons exhibit reduced robustness to brightness changes, with peak activation levels clustering around the original brightness setting (Fig. 5A, C). In contrast (no pun intended), activations steadily increase with rising contrast (Fig. 5B, D), following a near-linear trend. This pattern could be intuitive: color-selective neurons are likely to respond more strongly when color distinctions are enhanced (i.e., higher contrast), and less so when color information is obscured, either through excessive brightness or darkness.

        4. Neurons show a similar change of color selectivity index (CSI) for brightness and contrast transformations

        Similarly, we observed a peak in CSI values around the original image condition for brightness transformations, and a near-linear increase in CSI with increasing contrast (Fig. 6). This indicates that neurons not only vary in activation in response to color transformations but also exhibit corresponding shifts in their color selectivity. The parallel patterns in activation and CSI can be attributed to the definition of CSI itself: it is computed as 1 minus the ratio of a neuron’s activation to grayscale images relative to its activation to the same color images. Thus, as a neuron’s responsiveness to color increases, its CSI value also rises proportionally.

        Figure 5. The max activation values across transformations for the first high color-selective neurons (A, B) and all neurons (C, D).
        Figure 6. The color selectivity index (CSI) values across transformations for the first high color-selective neurons (A, B) and all neurons (C, D).

        5. Adaptive changes in neuronal color sensitivity across image conditions

        Furthermore, we analyzed the distribution of neurons across more finely segmented color selectivity index (CSI) ranges (i.e. bins of size 𝛂 = 0.05, like in the paper) to examine how the population profile shifts across transformations in different convolutional layers. In the original setting, highly color-selective neurons were relatively evenly distributed across the first two convolutional layers. This uniformity is preserved only under the high-contrast transformation condition, which also yields the highest overall activation and color selectivity values (Fig. 7A,B). In contrast, relative to their CSI in response to the original images, the neurons showed an increased proportion of lower color-selective neurons in response to brightness transformations (in either direction) and to the decreased contrast transformation (Fig. 7B,C), which aligns with the overall decrease of activation and CSI values in these conditions. This shift suggests that color selectivity might be impaired when fewer color cues in the image are available. So, it might be that the neurons are losing some of their color discrimination ability after those specific transformations.

        Figure 7. Distribution of neuron CSI values across the first two convolutional layers. (A) Original images. (B) Contrast transformations: comparison of distributions with +100% and -100% contrast. (C) Brightness transformations: comparison of distributions with +100% and -100% brightness.

        6. Fine-tuning the model for future direction

          Unfortunately, due to computational limitations, we did not have a chance to retrain the model on a GPU (we are still in the queue!). Therefore, one future direction will be finishing the fine-tuning process and comparing the resulting neuron population profiles across models.

          Summary

          In this final project, we successfully implemented some of the main components of the Rafegas & Vanrell framework to pull out neuron features, color selectivity, and activation patterns. It seems to have worked! We were able to reproduce several of the results that were reported in the original study, like the pretty noticeably different neuron features of low vs. high color-selective neurons. We took their analysis a step further by exploring the robustness of these neurons under various image transformations. Notably, we found that color selectivity not only decreases in values, but also shrinks in the overall proportion relative to the neuron group when the color cues are less prominent after the transformation.

          Building this meant lots of trial and error with our code throughout the process, which gave us a lot of technical practice. We also spent a lot of time in the queue for GPU on the HPC, which we unfortunately never get out of. We are grateful for the foundational work by Rafegas and Vanrell (and their paper that got us excited in the first place!), as well as for the contributions from the Nefesi repo authors. Lastly, we want to thank Prof. Zhu for her help and feedback throughout this project, and for her awesome Vision class this semester that has taught us so much both conceptually and technically 🙂 ❤️

          Works Cited: