Foveated Model Results & Discussion
by Dashiell Fairborn & Ezra Ford
We trained foveated and unfoveated versions of the sequential model from Killick et al. (2023) for 200 epochs, and saved the model checkpoint from the best epoch. Although our original plan was to use the spatial transformer version from Killick et al., the sequential model was chosen because that was the code that was available to us, but adapted to use a single fixation. We then loaded this checkpoint and fine-tuned the model on our custom dataset manually created using the JetBot camera (see Figure 1, Dataset Information in Blog Post II). For this custom dataset, we used blue tape as borders and also placed objects within the borders for the JetBot, which we named Conehead (after the cones of the fovea), to avoid.
We then tried for an entire day to get set up for the collision avoidance task, but as we got close to success, more errors seemed to inevitably crop up. Unfortunately, even an all-nighter just led to more errors. Due to time constraints on the project, we decided to pivot to obtain some results to report. We took the dataset we had created, and decided to make the models race. Each model was given the full dataset to classify, and we measured their average inference time and accuracy over 100 trials.
While the unfoveated version was slightly faster (Foveated: M = 1.48, SD = 0.71; Unfoveated: M = 1.37, SD = 0.09), there was no significant difference between the inference time of the models (p = .152). However there was an outlier in the foveated model with an inference time of 8.42 seconds on the first trial, which may have been due to some hardware aspect of initializing the model, as this was the first model that was run. This was confirmed by switching the first model to the unfoveated model and seeing an outlier inference time of 31.67 seconds on the first trial, with no outlier on the foveated version. After dropping the original outlier, the difference between interference times between models became tiny (Foveated: M = 0.141, SD = 0.14; Unfoveated: M = 1.37, SD = 0.09), but significant using a Welch’s t-test (t(166.93) = 2.01, p = .046). In addition, the unfoveated model had a higher accuracy of 90%, compared to the foveated model’s accuracy of 80%.
Interestingly, the foveated model consistently outperformed the unfoveated model, in terms of accuracy, right up until the end of the fine-tuning process. This is probably partly due to the distorted camera (as you can clearly see in Figure 2, the Jetbot camera uses a fisheye lens, leading to immense spatial distortion) that emphasizes the center of the image over the periphery. This makes the foveated model, which already has greater resolution in the center, even more information dense. This likely leads to better performance because it can more effectively exploit the camera’s physical characteristics.
This does, then, call into question its seeming sluggishness compared to the unfoveated model when they raced against each other. This may be explained by another one of the great challenges of the Jetbot: model optimization. Typically, the Jetson Nano board will optimize models using NVIDIA’s TensorRT architecture so they can operate better with the limited computational power. Unfortunately, the rather hefty mathematical operations involved in graph neural networks and in foveation proved too complicated for the simple architecture, especially when considering the relatively outdated nature of the Jetbot’s software. This created quite the headache when trying to get the model onto the robot, and it also reveals one of the challenges of robotics. Using a graph CNN to facilitate the foveation proved MORE computationally expensive than just using a traditional CNN, even though the presence of a fovea should serve to alleviate that burden. This means that with limited computational power, the constant creation of complex spiral patterns may prove a curse rather than a blessing. The graph CNN was too much for little old Conehead, implying that our intuitions about computational benefits may be ahead of their time.
Unlike Killick et al., we did not see vast computational advantages of foveation. We also did not see the same gains as the 2-camera models of foveation. Indeed, in some ways our fovea actually did worse and proved more challenging than a standard model. Even so, there remains hope. The Jetbot seemed to lack the computational power necessary to make full use of the GNN. Our results tell us that our hypothesized advantage of a foveated model—namely, faster and more efficient processing does not materialize on the JetBot robot. It could be that a smaller model or a more powerful robot would yield our expected results. The most desirable extension, of course, would be to acquire a functional robot that could actually run our intended task. Further, it would be incredibly interesting to compare robots with different amounts of computational power. Our results seem to suggest that there is a lower limit of power below which a fovea is not beneficial. Even so, that is merely a limitation of the GNN architecture. Comparing this architecture to the 2-camera strategy would be an interesting comparison of different foveation models to see which is actually more desirable. Another possibility, to make proper use of the Jetbot, is to run the model externally. This relieves computational pressure from the robot and allows us to make use of greater power. The model (on a separate machine) would communicate wirelessly with the robot, giving it movement instructions while reading its camera data. In this way, we could test foveated models in real-world applications without waiting for bigger and better bots.
Doubtless, there’s lots more to explore in the realm of robotic foveas, but this is a good start.