Automatic Radar-Camera Calibration and Fusion for Traffic Perception

In intelligent transportation systems (ITS), millimeter-wave (MMW) radar-camera fusion has emerged as a cost-effective and viable solution due to low sensor prices. However, deploying such a fusion system in practical far-range scenes faces significant challenges in both sensor calibration and fusion processes. To address these challenges, this paper presents a systematic study from theoretical analysis to practical system deployment. First, we review the status quo of radar-camera fusion systems, comparing existing calibration and fusion paradigms carefully. Through comparative analysis, we find that though feature-level fusion is popular in related research, the target-level fusion is more practical for roadside applications because it is computationally efficient and more robust to depth ambiguity. Second, we introduce an automatic radar-camera calibration and fusion system for real-world traffic perception. This system implements a trajectory-based calibration scheme for spatio-temporal synchronization, specifically tackling the difficulty of identifying distinguishable calibration targets in far-range environments. After calibration, this system applies a robust two-stage target-level fusion method to achieve effective radar-camera fusion in traffic scenes. Finally, we introduce the promising advancements of the proposed system and discuss several open challenges for large-scale and high-safety commercialization. We believe physics-aware self-supervised learning, cooperative perception across roadside devices, and end-to-end perception foundation models are important for future traffic perception systems.