We present Recurrent Vision Transformers (RVTs), a novel backbone for object detection with event cameras. Event cameras provide visual information with sub-millisecond latency at a high-dynamic range and with strong robustness against motion blur. These unique properties offer great potential for low-latency object detection and tracking in time-critical scenarios. Prior work in event-based vision has achieved outstanding detection performance but at the cost of substantial inference time, typically beyond 40 milliseconds. By revisiting the high-level design of recurrent vision backbones, we reduce inference time by a factor of 6 while retaining similar performance. To achieve this, we explore a multi-stage design that utilizes three key concepts in each stage: First, a convolutional prior that can be regarded as a conditional positional embedding. Second, local and dilated global self-attention for spatial feature interaction. Third, recurrent temporal feature aggregation to minimize latency while retaining temporal information. RVTs can be trained from scratch to reach state-of-the-art performance on event-based object detection - achieving an mAP of 47.2% on the Gen1 automotive dataset. At the same time, RVTs offer fast inference (less than 12 ms on a T4 GPU) and favorable parameter efficiency (5 times fewer than prior art). Our study brings new insights into effective design choices that can be fruitful for research beyond event-based vision.

Reference:
M. Gehrig, D. Scaramuzza
"Recurrent Vision Transformers for Object Detection with Event Cameras"
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, 2023
PDF: https://arxiv.org/abs/2212.05598
Code: https://github.com/uzh-rpg/RVT

For more information about our research, visit these pages:
1. Event-based Vision: http://rpg.ifi.uzh.ch/research_dvs.html
2. Machine Learning: http://rpg.ifi.uzh.ch/research_learni...

Affiliations:
M. Gehrig and D. Scaramuzza are with the Robotics and Perception Group, Dep. of Informatics, University of Zurich, Switzerland.