Abstract: Federated learning (FL) faces severe backdoor threats. Due to the inaccessibility of clean samples, the parameter server cannot clean them up in real time even if poisoning features are discovered. Meanwhile, existing backdoor defense methods always require sacrificing model accuracy or increasing communication delay in exchange for better FL trustworthiness. To address these challenges, we propose a novel data-free and swift backdoor purification (DSBP) scheme based on multi-teacher adversarial distillation to effectively erase various backdoor variants in FL. DSBP treats the purification task as an adversarial game process between knowledge inheritance and backdoor inhibition by enforcing the student model to learn ensemble results of multiple teacher models on reconstructed clean samples, while being insensitive to synthetic poisoned samples. In DSBP, we utilize the self-similarity of poisoned features to optimize the trigger generator and accelerate the convergence of DSBP during the adversarial distillation process. We validate the effectiveness of DBSP by comparing it with four state-of-the-art defense methods against three backdoor variants on three datasets. The average attack success rate can be reduced from 96.6% to 2.3% with only 300 rounds.