Artificial & Biological Vision

Methods of Investigation

by Charlotte Imbert & Yufei Meng

In the last blog post, we introduced how color perception can be represented and examined in biological and hence computational models. In this blog post, we will have an overview of the methods and metrics that we plan to implement to further extract color-selectivity information about the model.

The Dataset

    Due to computing resource limitations, we will use a subset of ImageNet, a standard natural image dataset. Tiny ImageNet was prepared by Stanford University for their CS231n course, available at https://github.com/pat-coady/tiny_imagenet/blob/master/README.md. In the tiny dataset, there are a total of 200 object classes available for classification, with 500 training, 50 validation, and 50 test samples per class (so it is not as tiny as you think). Another ‘tiny’ dimension is that images are all downsampled to 64 x 64 pixels from the 256 x 256 pixels of the original ImageNet. Though the dataset is not as massive as ImageNet, it is representative enough in terms of a number of classes and training and test samples to evaluate the pretrained model that we will be using.

    Testing the Model Performance 

      Before we proceed to dissect the model structure and implement color-related metrics, we will first test the model we will be using, the VGG-16 pre-trained model (https://pytorch.org/vision/main/models/generated/torchvision.models.vgg16.html), in order to quantify its performance specific to the Tiny dataset. The architecture of VGG-16 is shown in Figure 1. Since VGG-16 was trained on the full ImageNet, we need all 1000 class names instead of 200 to decode the output. Moreover, since only the validation images and not the test images are labeled in the Tiny dataset, and we will not be training the model using these images, we will use the classified validation images as test images for our model in this case. Also, since the Tiny dataset only contains 200 classes, we are not expecting the model to perform the same way as when tested with a larger dataset. Rather, we are more interested in the approximate performance level of the model before we dive deeper into its structures (i.e. can it get a good sense of what is in the image even if it doesn’t get the exact label right?)

      Figure 1. VGG-16 model architecture.

      Activation Curves and Color Selectivity Index

        We will be implementing some of the methods from Rafegas & Vanrell’s 2018 paper, ‘Color encoding in biologically-inspired convolutional neural networks’. Specifically, we want to compute the color selectivity index and plot activation curves for model neurons in a particular layer of the CNN, as shown in Figure 2. The layers we’ll be focusing on are Conv1 and Conv2. This decision was motivated by Rafegas & Vanrell’s findings that earlier layers contained the highest proportion of color selective neurons, and were less selective to complex features than the 3 deepest layers. Conv1 and Conv 2 are therefore of most interest to us since we’re interested in color selectivity.

        First, we’ll need to identify the patches  in each Tiny ImageNet  image that cause the highest activations in a specific neuron. This is referred to by Rafegas & Vanrell as the N-top image patches. The authors do this by:

        • Obtaining the feature map of a neuron for all images in the Tiny ImageNet training set
        • For each feature map, finding the maximum activation and cropping to keep the maximum activation
        • Ordering the cropped images by activation (highest to lowest)
        • Picking N-top image patches

        The N-top image patches are then used to create an activation curve for the neurons corresponding to that receptive field (Figure 1(b) and 1(c)). We will obtain these activations for neurons in VGG-16 layers Conv1 and Conv2 by using hooks in PyTorch. Hooks provide a way to capture the neuron activations in each layer every time an image is input to the model. Those activations can be stored in a dictionary; since the activations will be along 3 dimensions, we’ll need to compute an average activation for each neuron and each image patch.

        Figure 2. (a) Low color-selective neuronal activations to color images versus grayscale versions of those same images. There is very little difference between the color and grayscale activations. (b) High color-selective neuronal activations to color images versus grayscale versions of those same images. There is a drastic difference between the neuron’s activations to color images compared to grayscale images. (c) Neuron activations for 100 low color selective neurons to the neuron feature shown in the framed blue box. (d) Neuron activations for 100 high color selective neurons to the neuron feature shown in the framed blue box.

        After extracting the information about the color-corresponding activation patterns of neurons, we will move on to compute the color selectivity index (CSI) to find the high and low selective neurons and their indices for later use. To compute the color selectivity index (CSI), we’ll first define a function that turns images in Tiny ImageNet to grayscale. We’ll then define a function to calculate the CSI, following the paper’s methods. The authors define the CSI as follows:

        Where N is the number of top image patches, j is the index of those image patches, i is the ith neuron, and L is the Lth layer. The denominator is the sum of activations to the original images, and the numerator is the sum of activations to the grayscale version of those images. Essentially, this index is defined to quantify the relative activeness of neurons by color itself compared to that by the grey-scaled stimulus. By calculating CSI to neurons, we will obtain a list of high- and low-color-selective neurons whose performance we will compare for the original image patches versus the color-transformed image patches.

        Transformed Images

          We will apply color transformations to the N-top image patches (see Figure 3) used to calculate neuronal activations and see how these transformations affect the CSI. The two transformations we will look at are changes in brightness and changes in contrast. We will apply these transformations via functions that we will define in Python.

          After the transformation, we will then test the transformed image patches on the same model again and compare the activations for the same high- and low-color-selective neurons identified above. We will plot activation curves for these neurons, like those in Figure 2(c) and 2(d), as brightness and contrast increase.

          Figure 3. Sample steps of the algorithm used by Rafegas & Vanrell to determine the N-top image patches for each neuron

          Fine-tuning

            Besides the testing on the pretrained model, we also plan to do some transfer learning by training the model on the transformed data to examine potential changes of the model structure in convolutional layers 1 and 2. More specifically, we are curious about whether transforming the data would change the CSI of neurons, and about how these transformations may change the resulting patterns of weights in these two layers. If time allows, we will also compare the overall model performance between the original VGG-16 model and the model trained on color-modified datasets.

            Works Cited