Publication

SDformerFlow: Spiking neural network transformer for event-based optical flow

Conference Article

Conference

International Conference on Pattern Recognition (ICPR)

Edition

27th

Pages

475-491

Doc link

https://doi.org/10.1007/978-3-031-78354-8_30

File

Download the digital copy of the doc pdf document

Abstract

Event cameras produce asynchronous and sparse event streams capturing changes in light intensity. Overcoming limitations of conventional frame-based cameras, such as low dynamic range and data rate, event cameras prove advantageous, particularly in scenarios with fast motion or challenging illumination conditions. Leveraging similar asynchronous and sparse characteristics, Spiking Neural Networks (SNNs) emerge as natural counterparts for processing event camera data.
Recent advancements in Visual Transformer architectures have demonstrated enhanced performance in both Artificial Neural Networks (ANNs) and SNNs across various computer vision tasks. Motivated by the potential of transformers and spikeformers, we propose two solutions for fast and robust optical flow estimation: STTFlowNet and SDformerFlow. STTFlowNet adopts a U-shaped ANN architecture with spatiotemporal Swin transformer encoders, while SDformerFlow presents its full spike counterpart with spike-driven Swin transformer encoders.
Notably, our work marks the first utilization of spikeformer for dense optical flow estimation. We conduct end-to-end training for both models using supervised learning on the DSEC-flow Dataset. Our results indicate comparable performance with state-of-the-art SNNs and significant improvement in power consumption compared to the best-performing ANNs for the same task.
Our code will be open-sourced at https://github.com/yitian97/SDformerFlow.

Categories

computer vision.

Author keywords

Spiking neural networks, event camera, transformer, optical flow

Scientific reference

Y. Tian and J. Andrade-Cetto. SDformerFlow: Spiking neural network transformer for event-based optical flow, 27th International Conference on Pattern Recognition, 2024, Kolkata, in Pattern Recognition, Vol 15315 of Lecture Notes in Computer Science, pp. 475-491, 2024, Cham.