Foveated Model Construction and Evaluation
by Dashiell Fairborn & Ezra Ford
Model Architecture
We will start by pre-training the spatial transformer version of the Killick et al. (2023) architecture on its original object classification task. This network identifies the ideal location for fixation in an image, applies foveation, and uses a graph convolutional neural network to predict the image class. To adapt this pretrained architecture to predict the binary output “blocked” or “unblocked,” we freeze the pretrained network but replace the classification layer with a one-hot encoded binary classification layer and train this layer to predict “blocked” or “unblocked.”
We will use the network (Figure 1) to process a still image frame from the video feed at every timestep and output a “blocked” or “unblocked” path classification. If the output is “unblocked,” the robot will drive forward; if the output is “blocked,” the robot will stop and turn left until it has determined that it is unblocked.
For each image, we need to first generate a fixation coordinate and sample the image based on the foveation. To generate the fixation coordinate, each image will first be uniformly downsampled to a smaller size to make processing lighter, and then a small CNN will process the downsampled image. This CNN outputs a feature map, which is fed into an attention module that applies a 1×1 convolution and a softmax function, generating a saliency map containing the probabilities of a given location containing salient information. These probabilities are multiplied by their corresponding (x, y) coordinate, and all the resulting values are added to give a final fixation coordinate. For the control model, the center of the image is used as the fixation point.
Next, the image is sampled using the foveation or uniform distribution (in the case of the control model) and fed into the graph convolutional network for feature extraction. The foveation version densely samples points within the fovea radius (60% of image diameter), and then logarithmically in the periphery, leading to increasingly sparse sampling in the periphery. The control version samples points uniformly across the image based on Vogel’s original sunflower model (see Figure 3 from Blog Post I).
Because the sampled points do not lie on a regular grid, the graph convolutional network computes feature maps by treating the points as vertices of a graph: it finds each point’s nearest neighbors and dynamically adjusts the weights based on their relative position to compute the features. This allows the network to extract meaningful spatial features even from non-uniformly sampled data.
Finally, a classifier head applies global average pooling over features and includes a final linear layer that predicts object category in the pre-training and obstruction status in the final task.
Dataset Information
The pretrained networks are pre-trained on ImageNet-100 and then fine-tuned on a custom dataset manually created using the JetBot (see video explaining the process). The custom dataset consists of pictures taken through the JetBot camera labeled as either “Blocked” or “Unblocked” depending on whether or not the JetBot’s path is obstructed when the photo was taken. Figure 2 shows an example of collecting this data. In the case of Figure 2, the JetBot is being trained to stay inside the neon tape, so when it is at the edge of its enclosure it is “Blocked.”
Model Evaluation
The models will be evaluated on a JetBot. The JetBot is a portable, battery-powered AI robot built on the NVIDIA Jetson Nano. It has been used to compare CNNs and training algorithms in terms of self-driving capabilities (Leigh & Xing, 2022), to evaluate the usefulness of deep learning networks performing real-world warehouse jobs (Kawakura & Shibasaki, 2020), and to follow targets and avoid obstacles (Lee & Lee, 2022). Overall, it’s a very common choice for the evaluation of neural networks in robotic settings.
Our task will be one of collision avoidance. The JetBot will be placed on top of a table surrounded with obstacles and must continuously move around without colliding with anything. An example of this process is shown in Figure 3, where multiple JetBots are placed on a messy desk and must avoid obstacles. If it hits something, that indicates that the collision avoidance system (i.e. the network) has failed. During testing, we will progressively increase the speed of the JetBot until the network is too slow to fully avoid a collision. Essentially, we are testing the JetBot’s reaction time in the foveated and control conditions.
We predict that the maximum speed of the foveated model will be faster than the max speed of the control model. In other words, the reaction time with a fovea will be shorter because foveation offers computational benefits that can be exploited to reduce latency of response times.
Citations
- Chitoku Yato NV. (2019, Mar 12). Launching of JetBots [Video]. YouTube. https://www.youtube.com/watch?v=6cLk9TSgFSw
- Esteves, C., Allen-Blanchette, C., Zhou, X., & Daniilidis, K. (2018). Polar transformer networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, BC, Canada, April 30–May 3, 2018. OpenReview.net. https://openreview.net/forum?id=HktRlUlAZ
- jaybdub, & Yato, C. (2019). Collision avoidance. JetBot. https://jetbot.org/master/examples/collision_avoidance.html
- Kawakura, S., & Shibasaki, R. (2020). Deep Learning-Based Self-Driving Car: JetBot with NVIDIA AI Board to Deliver Items at Agricultural Workplace with Object-Finding and Avoidance Functions. European Journal of Agriculture and Food Sciences, 2(3). https://doi.org/10.24018/ejfood.2020.2.3.45.
- Killick, G., Henderson, P., Siebert, P., & Aragon-Camarasa, G. (2023). Foveation in the Era of Deep Learning. British Machine Vision Conference, Aberdeen, UK.
- Lee, M. C., & Lee, M. (2022) Deep Learning-based Target Following and Obstacle Avoidance Methods in Mobile Robots. IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia), Yeosu, Korea, Republic of, 2022, pp. 1-4. https://doi.org/10.1109/ICCE-Asia57006.2022.9954653.
- Leigh P., & Xing, Y. (2022). Evaluation of Multiple Convolutional Neural Networks in Training the NVIDIA JetBot. IEEE 13th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, NY, USA, 2022, pp. 0524-0530. https://doi.org/10.1109/UEMCON54665.2022.9965727.
- NVIDIA Developer. (2020, Oct 15). Jetson AI Fundamentals – S2E3 – JetBot Collision Avoidance [Video]. YouTube. https://www.youtube.com/watch?v=mwOVFXkc1WI