We use cookies to improve your experience with our site.

Indexed in:

SCIE, EI, Scopus, INSPEC, DBLP, CSCD, etc.

Submission System
(Author / Reviewer / Editor)
Advanced Search
Volume 40 Issue 2
May  2025
Turn off MathJax
Article Contents
Abstract
Ethical Declaration
References
Related Articles
Supplements
Ge JW, Cao JX, Zhao ZX et al. FSD-GAN: Generative adversarial training for face swap detection via the latent noise fingerprint. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY, 40(2): 397−412, Mar. 2025. DOI: 10.1007/s11390-024-3337-8
Citation: Ge JW, Cao JX, Zhao ZX et al. FSD-GAN: Generative adversarial training for face swap detection via the latent noise fingerprint. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY, 40(2): 397−412, Mar. 2025. DOI: 10.1007/s11390-024-3337-8
shu

FSD-GAN: Generative Adversarial Training for Face Swap Detection via the Latent Noise Fingerprint

  • 1.

    School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China

  • 2.

    Purple Mountain Laboratories, Nanjing 211111, China

  • 3.

    School of Computer Science and Engineering, Southeast University, Nanjing 211189, China

Funds: 

This work was supported by the National Natural Science Foundation of China under Grant Nos. 62472092, 62172089, and 62106045, the Natural Science Foundation of Jiangsu Province of China under Grant No. BK20241751, the Jiangsu Provincial Key Laboratory of Computer Networking Technology, the Jiangsu Provincial Key Laboratory of Network and Information Security under Grant No. BM2003201, the Key Laboratory of Computer Network and Information Integration of the Ministry of Education of China under Grant No. 93K-9, and the Nanjing Purple Mountain Laboratories. We appreciate the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations.

More Information
  • Author Bio:

    Jia-Wei Ge received his B.S. degree in computer science and technology from Hohai University, Nanjing, in 2021. Now he is working towards his Ph.D. degree at the School of Cyber Science and Engineering, Southeast University, Nanjing. His research interests include DeepFake detection, visual and language inference, and video understanding

    Jiu-Xin Cao received his Ph.D. degree from Xi'an Jiaotong University, Xi'an, in 2003. He is currently a professor with the School of Cyber Science and Engineering, Southeast University, Nanjing. He is also the director of the Jiangsu Provincial Key Laboratory of Computer Networking Technology, Nanjing. His research interests include social analysis and behavior prediction in online social networks, user behavior analysis based on big data, and crowd-sensing based trustworthy location service in mobile social networks

    Zhi-Xiang Zhao received his M.S. degree at the School of Cyber Science and Engineering, Southeast University, Nanjing, in 2022. His research interests include DeepFake detection and image processing. Since then, he has been working at the Tencent

    Bo Liu received her Ph.D. degree from Southeast University, Nanjing, in 2007. She is currently a professor with the School of Computer Science and Engineering, Southeast University, Nanjing. Her main research interests include spammer detection in social network, the evolution of social community, social influence, and multi-agent technology

  • Corresponding author:

    jx.cao@seu.edu.cn

  • Received Date: May 08, 2023
  • Accepted Date: October 21, 2024
    Abstract

  • Current studies against DeepFake attacks are mostly passive methods that detect specific defects of DeepFake algorithms, lacking generalization ability. Meanwhile, existing active defense methods only focus on defending against face attribute manipulations, and there remain enormous challenges to establishing an active and sustainable defense mechanism for face swap detection. Therefore, we propose a novel training framework called FSD-GAN (Face Swap Detection based on Generative Adversarial Network), immune to the evolution of face swap attacks. Specifically, FSD-GAN contains three modules: the data processing module, the attack module that generates fake faces only used in training, and the defense module that consists of a fingerprint generator and a fingerprint discriminator. We embed the latent noise fingerprints generated by the fingerprint generator into face images, unperceivable to attackers visually and statistically. Once an attacker uses these protected faces to perform face swap attacks, these fingerprints will be transferred from training data (protected faces) to generative models (real-world face swap models), and they also exist in generated results (swapped faces). Our discriminator can easily detect latent noise fingerprints embedded in face images, converting the problem of face swap detection to verifying if fingerprints exist in swapped face images or not. Moreover, we alternately train the attack and defense modules under an adversarial framework, making the defense module more robust. We illustrate the effectiveness and robustness of FSD-GAN through extensive experiments, demonstrating that it can confront various face images, mainstream face swap models, and JPEG compression under different qualities.

    • FullText(HTML)
    • References (47)
  • [1]
    Tolosana R, Vera-Rodriguez R, Fierrez J, Morales A, Ortega-Garcia J. DeepFakes and beyond: A survey of face manipulation and fake detection. Information Fusion, 2020, 64: 131–148. DOI: 10.1016/J.INFFUS.2020.06.014.
    [2]
    Barni M. Steganography in digital media: Principles, algorithms, and applications (Fridrich, J. 2010) [Book Reviews]. IEEE Signal Processing Magazine, 2011, 28(5): 142–144. DOI: 10.1109/MSP.2011.941841.
    [3]
    You J, Wang Y G, Zhu G, Wu L, Zhang H, Kwong S. Estimating the secret key of spread spectrum watermarking based on equivalent keys. IEEE Trans. Multimedia, 2023, 25: 2459–2473. DOI: 10.1109/TMM.2022.3147379.
    [4]
    Chen H, Rouhani B D, Fu C, Zhao J, Koushanfar F. DeepMarks: A secure fingerprinting framework for digital rights management of deep learning models. In Proc. the 2019 International Conference on Multimedia Retrieval, Jun. 2019, pp.105–113. DOI: 10.1145/3323873.3325042.
    [5]
    Cayre F, Fontaine C, Furon T. Watermarking security: Theory and practice. IEEE Trans. Signal Processing, 2005, 53(10): 3976–3987. DOI: 10.1109/TSP.2005.855418.
    [6]
    Holub V, Fridrich J, Denemark T. Universal distortion function for steganography in an arbitrary domain. EURASIP Journal on Information Security, 2014, 2014: 1. DOI: 10.1186/1687-417X-2014-1.
    [7]
    Zhang R, Dong S, Liu J. Invisible steganography via generative adversarial networks. Multimedia Tools and Applications, 2019, 78(7): 8559–8575. DOI: 10.1007/S11042-018-6951-Z.
    [8]
    Tancik M, Mildenhall B, Ng R. StegaStamp: Invisible hyperlinks in physical photographs. In Proc. the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.2114–2123. DOI: 10.1109/CVPR42600.2020.00219.
    [9]
    Adi Y, Baum C, Cissé M, Pinkas B, Keshet J. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In Proc. the 27th USENIX Conference on Security Symposium, Aug. 2018, pp.1615–1631. DOI: 10.5555/3277203.3277324.
    [10]
    Fan L, Ng K W, Chan C S. Rethinking deep neural network ownership verification: Embedding passports to defeat ambiguity attacks. In Proc. the 33rd International Conference on Neural Information Processing Systems, Dec. 2019, Article No.424. DOI: 10.5555/3454287.3454711.
    [11]
    Rouhani B D, Chen H, Koushanfar F. DeepSigns: An end-to-end watermarking framework for ownership protection of deep neural networks. In Proc. the 24th International Conference on Architectural Support for Programming Languages and Operating Systems, Apr. 2019, pp.485–497. DOI: 10.1145/3297858.3304051.
    [12]
    Kingma D P, Welling M. Auto-encoding variational Bayes. In Proc. the 2nd International Conference on Learning Representations, Apr. 2014.
    [13]
    Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. In Proc. the 28th International Conference on Neural Information Processing Systems, Dec. 2014, pp.2672–2680. DOI: 10.5555/2969033.2969125.
    [14]
    Lu Z, Li Z, Cao J, He R, Sun Z. Recent progress of face image synthesis. In Proc. the 4th IAPR Asian Conference on Pattern Recognition, Nov. 2017, pp.7–12. DOI: 10.1109/ACPR.2017.2.
    [15]
    Chen R, Chen X, Ni B, Ge Y. SimSwap: An efficient framework for high fidelity face swapping. In Proc. the 28th ACM International Conference on Multimedia, Oct. 2020, pp.2003–2011. DOI: 10.1145/3394171.3413630.
    [16]
    Gao G, Huang H, Fu C, Li Z, He R. Information bottleneck disentanglement for identity swapping. In Proc. the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2021, pp.3403–3412. DOI: 10.1109/CVPR46437.2021.00341.
    [17]
    Choi Y, Choi M, Kim M, Ha J W, Kim S, Choo J. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp.8789–8797. DOI: 10.1109/CVPR.2018.00916.
    [18]
    Zhang H, Xu T, Li H, Zhang S, Wang X, Huang X, Metaxas D N. StackGAN++: Realistic image synthesis with stacked generative adversarial networks. IEEE Trans. Pattern Analysis and Machine Intelligence, 2019, 41(8): 1947–1962. DOI: 10.1109/TPAMI.2018.2856256.
    [19]
    Liu M, Ding Y, Xia M, Liu X, Ding E, Zuo W, Wen S. STGAN: A unified selective transfer network for arbitrary image attribute editing. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.3668–3677. DOI: 10.1109/CVPR.2019.00379.
    [20]
    He Z, Zuo W, Kan M, Shan S, Chen X. AttGAN: Facial attribute editing by only changing what you want. IEEE Trans. Image Processing, 2019, 28(11): 5464–5478. DOI: 10.1109/TIP.2019.2916751.
    [21]
    Zhang X, Zheng Z, Gao D, Zhang B, Yang Y, Chua T S. Multi-view consistent generative adversarial networks for compositional 3D-aware image synthesis. International Journal of Computer Vision, 2023, 131(8): 2219–2242. DOI: 10.1007/S11263-023-01805-X.
    [22]
    Nataraj L, Mohammed T M, Manjunath B S, Chandrasekaran S, Flenner A, Bappy J H, Roy-Chowdhury A K. Detecting GAN generated fake images using co-occurrence matrices. In Proc. the IS&T International Symposium on Electronic Imaging: Media Watermarking, Security, and Forensics, Jan. 2019.
    [23]
    Xuan X, Peng B, Wang W, Dong J. On the generalization of GAN image forensics. In Proc. the 14th Chinese Conference on Biometric Recognition, Oct. 2019, pp.134–141. DOI: 10.1007/978-3-030-31456-9_15.
    [24]
    McCloskey S, Albright M. Detecting GAN-generated imagery using color cues. arXiv: 1812.08247, 2018. https://arxiv.org/abs/1812.08247, Mar. 2025.
    [25]
    Marra F, Gragnaniello D, Verdoliva L, Poggi G. Do GANs leave artificial fingerprints? In Proc. the 2019 IEEE Conference on Multimedia Information Processing and Retrieval, Mar. 2019, pp.506–511. DOI: 10.1109/MIPR.2019.00103.
    [26]
    Huang S H, Papernot N, Goodfellow I J, Duan Y, Abbeel P. Adversarial attacks on neural network policies. In Proc. the 5th International Conference on Learning Representations, Apr. 2017.
    [27]
    Ruiz N, Bargal S A, Sclaroff S. Disrupting DeepFakes: Adversarial attacks against conditional image translation networks and facial manipulation systems. In Proc. the European Conference on Computer Vision, Aug. 2020, pp.236–251. DOI: 10.1007/978-3-030-66823-5_14.
    [28]
    Huang Q, Zhang J, Zhou W, Zhang W, Yu N. Initiative defense against facial manipulation. In Proc. the 35th AAAI Conference on Artificial Intelligence, Feb. 2021, pp.1619–1627. DOI: 10.1609/aaai.v35i2.16254.
    [29]
    Yang C, Ding L, Chen Y, Li H. Defending against GAN-based DeepFake attacks via transformation-aware adversarial faces. In Proc. the 2021 International Joint Conference on Neural Networks, Jul. 2021. DOI: 10.1109/IJCNN52387.2021.9533868.
    [30]
    Dong J, Xie X. Visually maintained image disturbance against DeepFake face swapping. In Proc. the 2021 IEEE International Conference on Multimedia and Expo, Jul. 2021. DOI: 10.1109/ICME51207.2021.9428173.
    [31]
    Wang R, Juefei-Xu F, Luo M, Liu Y, Wang L. Faketagger: Robust safeguards against DeepFake dissemination via provenance tracking. In Proc. the 29th ACM International Conference on Multimedia, Oct. 2021, pp.3546–3555. DOI: 10.1145/3474085.3475518.
    [32]
    Bossert M. Channel Coding for Telecommunications. Wiley, 1999.
    [33]
    He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.770–778. DOI: 10.1109/CVPR.2016.90.
    [34]
    Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks. In Proc. the 34th International Conference on Machine Learning, Aug. 2017, pp.214–223. DOI: 10.5555/3305381.3305404.
    [35]
    Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville A C. Improved training of Wasserstein GANs. In Proc. the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp.5769–5779. DOI: 10.5555/3295222.3295327.
    [36]
    Rössler A, Cozzolino D, Verdoliva L, Riess C, Thies J, Niessner M. Faceforensics++: Learning to detect manipulated facial images. In Proc. the 2019 IEEE/CVF International Conference on Computer Vision, Oct. 27–Nov. 2, 2019, pp.1–11. DOI: 10.1109/ICCV.2019.00009.
    [37]
    Liu Z, Luo P, Wang X, Tang X. Deep learning face attributes in the wild. In Proc. the 2015 IEEE International Conference on Computer Vision, Dec. 2015, pp.3730–3738. DOI: 10.1109/ICCV.2015.425.
    [38]
    Zhu J Y, Park T, Isola P, Efros A A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. the 2017 IEEE International Conference on Computer Vision, Oct. 2017, pp.2242–2251. DOI: 10.1109/ICCV.2017.244.
    [39]
    Shi W, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.1874–1883. DOI: 10.1109/CVPR.2016.207.
    [40]
    Chollet F. Xception: Deep learning with depthwise separable convolutions. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.1800–1807. DOI: 10.1109/CVPR.2017.195.
    [41]
    Shiohara K, Yamasaki T. Detecting DeepFakes with self-blended images. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.18699–18708. DOI: 10.1109/CVPR52688.2022.01816.
    [42]
    Dong S, Wang J, Ji R, Liang J, Fan H, Ge Z. Implicit identity leakage: The stumbling block to improving DeepFake detection generalization. In Proc. the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2023, pp.3994–4004. DOI: 10.1109/CVPR52729.2023.00389.
    [43]
    Wang W, Dong J, Tan T. Exploring DCT coefficient quantization effects for local tampering detection. IEEE Trans. Information Forensics and Security, 2014, 9(10): 1653–1666. DOI: 10.1109/TIFS.2014.2345479.
    [44]
    Yu N, Skripniuk V, Abdelnabi S, Fritz M. Artificial fingerprinting for generative models: Rooting DeepFake attribution in training data. In Proc. the 2021 IEEE/CVF International Conference on Computer Vision, Oct. 2021, pp.14428–14437. DOI: 10.1109/ICCV48922.2021.01418.
    [45]
    Sun W, Zhou J, Li Y, Cheung M, She J. Robust high-capacity watermarking over online social network shared images. IEEE Trans. Circuits and Systems for Video Technology, 2021, 31(3): 1208–1221. DOI: 10.1109/TCSVT.2020.2998476.
    [46]
    Zhu J, Kaplan R, Johnson J, Fei-Fei L. HiDDeN: Hiding data with deep networks. In Proc. the 15th European Conference on Computer Vision, Sept. 2018, pp.682–697. DOI: 10.1007/978-3-030-01267-0_40.
    [47]
    Lerch-Hostalot D, Megías D. Unsupervised steganalysis based on artificial training sets. Engineering Applications of Artificial Intelligence, 2016, 50: 45–59. DOI: 10.1016/J.ENGAPPAI.2015.12.013.
    • Relative Articles

  • Related Articles

    [1]Lei Guan, Dong-Sheng Li, Ji-Ye Liang, Wen-Jian Wang, Ke-Shi Ge, Xi-Cheng Lu. Advances of Pipeline Model Parallelism for Deep Learning Training: An Overview[J]. Journal of Computer Science and Technology, 2024, 39(3): 567-584. DOI: 10.1007/s11390-024-3872-3
    [2]Adam Weingram, Yuke Li, Hao Qi, Darren Ng, Liuyao Dai, Xiaoyi Lu. xCCL: A Survey of Industry-Led Collective Communication Libraries for Deep Learning[J]. Journal of Computer Science and Technology, 2023, 38(1): 166-195. DOI: 10.1007/s11390-023-2894-6
    [3]Xin Zhang, Siyuan Lu, Shui-Hua Wang, Xiang Yu, Su-Jing Wang, Lun Yao, Yi Pan, Yu-Dong Zhang. Diagnosis of COVID-19 Pneumonia via a Novel Deep Learning Architecture[J]. Journal of Computer Science and Technology, 2022, 37(2): 330-343. DOI: 10.1007/s11390-020-0679-8
    [4]Sheng-Luan Hou, Xi-Kun Huang, Chao-Qun Fei, Shu-Han Zhang, Yang-Yang Li, Qi-Lin Sun, Chuan-Qing Wang. A Survey of Text Summarization Approaches Based on Deep Learning[J]. Journal of Computer Science and Technology, 2021, 36(3): 633-663. DOI: 10.1007/s11390-020-0207-x
    [5]Hua Chen, Juan Liu, Qing-Man Wen, Zhi-Qun Zuo, Jia-Sheng Liu, Jing Feng, Bao-Chuan Pang, Di Xiao. CytoBrain: Cervical Cancer Screening System Based on Deep Learning Technology[J]. Journal of Computer Science and Technology, 2021, 36(2): 347-360. DOI: 10.1007/s11390-021-0849-3
    [6]Jun Gao, Paul Liu, Guang-Di Liu, Le Zhang. Robust Needle Localization and Enhancement Algorithm for Ultrasound by Deep Learning and Beam Steering Methods[J]. Journal of Computer Science and Technology, 2021, 36(2): 334-346. DOI: 10.1007/s11390-021-0861-7
    [7]Wei Du, Yu Sun, Hui-Min Bao, Liang Chen, Ying Li, Yan-Chun Liang. DeepHBSP: A Deep Learning Framework for Predicting Human Blood-Secretory Proteins Using Transfer Learning[J]. Journal of Computer Science and Technology, 2021, 36(2): 234-247. DOI: 10.1007/s11390-021-0851-9
    [8]Andrea Caroppo, Alessandro Leone, Pietro Siciliano. Comparison Between Deep Learning Models and Traditional Machine Learning Approaches for Facial Expression Recognition in Ageing Adults[J]. Journal of Computer Science and Technology, 2020, 35(5): 1127-1146. DOI: 10.1007/s11390-020-9665-4
    [9]Nuo Qun, Hang Yan, Xi-Peng Qiu, Xuan-Jing Huang. Chinese Word Segmentation via BiLSTM+Semi-CRF with Relay Node[J]. Journal of Computer Science and Technology, 2020, 35(5): 1115-1126. DOI: 10.1007/s11390-020-9576-4
    [10]Hui-Ying Lan, Lin-Yang Wu, Xiao Zhang, Jin-Hua Tao, Xun-Yu Chen, Bing-Rui Wang, Yu-Qing Wang, Qi Guo, Yun-Ji Chen. DLPlib: A Library for Deep Learning Processor[J]. Journal of Computer Science and Technology, 2017, 32(2): 286-296. DOI: 10.1007/s11390-017-1722-2
    • Supplements (3)

  • Others

Catalog

    Get Citation
    PDF
    XML
    Read Online
    Article views (202) PDF downloads (71) Cited by()
    Related

    Copyright ©2025 JCST, All Rights Reserved     京ICP备05002829号-1

    Institute of Computing Technology, Chinese Academy of Sciences
    P.O. Box 2704, Beijing 100190, P.R. China

    Tel.: 86-10-62610746 E-mail: jcst@ict.ac.cn

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return