Introduction: Advantages of Foveation for Robot Object Recognition
by Dashiell Fairborn & Ezra Ford
As we sat in the Benton Museum of Art contemplating an untitled 1968 work by Lee Bontecou, we realized that the eye is a three-dimensional structure, so it can’t just be receiving two-dimensional images, as computer vision models assume. We then wondered about other differences between computational models and physical reality, which led us to the fovea.
In the human eye, the fovea is the point of highest visual resolution. That means it’s the center of your gaze, where you can see the best. The rest of your visual field is your periphery, which is characterized by its lower resolution but wider field of view. In typical artificial visual systems, i.e. Convolutional Neural Networks (CNNs), there is no difference between the center and the periphery of an image. The image has equal resolution everywhere. This can cause issues in robots, where storage space and processing power is limited. We thought that, by implementing a foveal system that discards unnecessary information, we could reduce the memory requirements of image processing through image size reduction while maintaining accuracy. We thought that bringing biologically inspired ideas to robotics would improve performance.
So, with help from Professor Anthony Clark, we started brainstorming different biological features and potential ways to implement them (Figure 1). To approximate the differences in resolution, we could separate our image into 2 parts: center and periphery. However, recombining these parts to create a single image with two different resolutions would be difficult as traditional CNNs expect the input to have a single, constant resolution. To get around this, we considered either passing each part to the CNN independently, as 2 different inputs. This would additionally let us process the periphery in black and white, as the human eye does not have many color-sensitive cones outside of the fovea. We also considered passing each image part to different CNNs before combining the outputs of each network. This would allow for independent processing or the center and the periphery.
As it turns out, though, this is not a very new idea. Researchers have wondered for quite a long time about foveated vision, so methods for its approximation already exist. One such approximation is the use of log-polar coordinates for image representation (Traver et al. 2010). By using polar coordinates, the image is represented as a circle rather than a rectangle, which anyone familiar with the word “eyeball” can recognize as being more biologically realistic. With log coordinates, the density of pixels is greater at the center of the image, effectively replicating the high-resolution center and low-resolution periphery in a continuous manner (Figure 2). Other methods include the use of lenses with variant spatial resolution or the use of two cameras in each “eye”: a wide angle, low resolution camera as the periphery and a narrow angle, high resolution camera as the fovea (Ude et al. 2004). Ude et al. used the dual camera approach and actually implemented their model of foveation in a robot, where it improved performance in an object recognition task, but their network lacked the complexity of CNNs, meaning their work should be updated with more modern network architecture.
The most inspiring approach we have found to date is the implementation of a foveated graph convolutional network for processing images by Killick et al. (2023). The advantage of this graph approach is that it removes the necessity of processing an image as a uniform n by m grid, which leads to difficulties processing the higher resolution fovea in conjunction with the lower resolution periphery. Their approach also allows for differentiable training using backpropagation with cross-entropy loss, eliminating the need for reinforcement learning.
One version of the architecture uses a sequential model that can attend to relevant image regions through iterative fixations, and another uses a spatial transformer that identifies the ideal location for the fixation. Both share a differentiable foveated sensor that samples the input image, a graph convolutional feature extractor, an attention module based on Polar Transformer Networks (Esteves et al., 2018), and a classifier head with a global average pooling layer and a linear layer. The foveated sensor is given a fixation coordinate, calculated by the attention module, and produces a foveated image processed by the feature extractor. The resulting feature map is passed into the classification head, which generates the class prediction.
We are currently iterating on our planned approach. Our current plan is to take inspiration from Killick et al. and use a modified object recognition task to improve navigation in our robot (Figure 4).
Citations
- Esteves, C., Allen-Blanchette, C., Zhou, X., & Daniilidis, K. (2018). Polar transformer networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, BC, Canada, April 30–May 3, 2018. OpenReview.net. https://openreview.net/forum?id=HktRlUlAZ
- Vogel, H. (1979). A better way to construct the sunflower head. Mathematical Biosciences, 44, 179–189. https://doi.org/10.1016/0025-5564(79)90080-4
- Javier Traver, V., & Bernardino, A. (2010). A review of log-polar imaging for visual perception in robotics. Robotics and Autonomous Systems, 58(4), 378–398. https://doi.org/10.1016/j.robot.2009.10.002
- Ude, A., & Cheng, G. (2004). Object recognition on humanoids with foveated vision. Humanoids. IEEE-RAS/RSJ International Conference on Humanoid Robots, Los Angeles, California, USA.Killick, G., Henderson, P., Siebert, P., & Aragon-Camarasa, G. (2023). Foveation in the Era of Deep Learning. British Machine Vision Conference, Aberdeen, UK.