We use cookies to improve your experience with our site.

Indexed in:

SCIE, EI, Scopus, INSPEC, DBLP, CSCD, etc.

Submission System
(Author / Reviewer / Editor)
Advanced Search
Volume 40 Issue 3
July  2025
Turn off MathJax
Article Contents
Abstract
Conflict of Interest
References
Supplements
Chen X, Sha LT, Xiao F et al. AB-DHD: An attention mechanism and bi-directional gated recurrent unit based model for dynamic link library hijacking vulnerability discovery. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY, 40(3): 887−903, May 2025. DOI: 10.1007/s11390-025-4497-x
Citation: Chen X, Sha LT, Xiao F et al. AB-DHD: An attention mechanism and bi-directional gated recurrent unit based model for dynamic link library hijacking vulnerability discovery. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY, 40(3): 887−903, May 2025. DOI: 10.1007/s11390-025-4497-x
shu

AB-DHD: An Attention Mechanism and Bi-Directional Gated Recurrent Unit Based Model for Dynamic Link Library Hijacking Vulnerability Discovery

  • School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023, China

Funds: 

This work was supported by the National Natural Science Foundation of China under Grant Nos. 62072253, 62172258, 62302238, and 62372245, the CCF-Tencent Rhino-Bird Open Research Fund, the Major Science and Technology Demonstration Project of Jiangsu Provincial Key Research and Development Program under Grant No. BE2022798, and the Postgraduate Research and Practice Innovation Program of Jiangsu Province of China under Grant No. KYCX20_0829.

More Information
  • Author Bio:

    Xiao Chen received his B.E. degree from the Nanjing University of Posts and Telecommunications (NJUPT), Nanjing, in 2018. He is currently a Ph.D. candidate in information security of NJUPT, Nanjing. His main research interests include IoT security, software and system security, and machine learning

    Le-Tian Sha received his B.E. degree from the Southwest Jiaotong University, Chengdu, in 2007, and his Ph.D. degree from Wuhan University, Wuhan, in 2014. He is currently a professor in the School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing. His main research interests include software security, cyber security, and defense in Internet of Things

    Fu Xiao received his Ph.D. degree in computer science and technology from the Nanjing University of Science and Technology, Nanjing, in 2007. He is currently a professor and a Ph.D. supervisor with the School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing. His research interest includes wireless sensor networks

    Jia-Ye Pan received his Ph.D. degree from Nanjing University of Aeronautics and Astronautics, Nanjing, in 2020. He is currently an associate professor with the School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing. His research interests include system security, software security, and network security

    Jian-Kuo Dong received his B.E. degree from the Xi’an Jiaotong University, Xi’an, in 2014, and his Ph.D. degree from the University of Chinese Academy of Sciences, Beijing, in 2019. He is currently an associate professor with the School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing. His research interests include cryptographic engineering, public key cryptography, and applied cryptography

  • Corresponding author:

    ltsha@njupt.edu.cn

  • Received Date: May 31, 2024
  • Accepted Date: February 17, 2025
    Abstract

  • With the rapid development of operating systems, attacks on system vulnerabilities are increasing. Dynamic link library (DLL) hijacking is prevalent in installers on freeware platforms and is highly susceptible to exploitation by malware attackers. However, existing studies are based solely on the load paths of DLLs, ignoring the attributes of installers and invocation modes, resulting in low accuracy and weak generality of vulnerability detection. In this paper, we propose a novel model, AB-DHD, which is based on an attention mechanism and a bi-directional gated recurrent unit (BiGRU) neural network for DLL hijacking vulnerability discovery. While BiGRU is an enhancement of GRU and has been widely applied in sequence data processing, a double-layer BiGRU network is introduced to analyze the internal features of installers with DLL hijacking vulnerabilities. Additionally, an attention mechanism is incorporated to dynamically adjust feature weights, significantly enhancing the ability of our model to detect vulnerabilities in new installers. A comprehensive “List of Easily Hijacked DLLs” is developed to serve a reference for future studies. We construct an EXEFul dataset and a DLLVul dataset, using data from two publicly available authoritative vulnerability databases, Common Vulnerabilities & Exposures (CVE) and China National Vulnerability Database (CNVD), and mainstream installer distribution platforms. Experimental results show that our model outperforms popular automated tools like Rattler and DLLHSC, achieving an accuracy of 97.79% and a recall of 94.72%. Moreover, 17 previously unknown vulnerabilities have been identified, and corresponding vulnerability certifications have been assigned.

    • FullText(HTML)
    • References (28)
  • [1]
    Han X, Yu X, Pasquier T, Li D, Rhee J, Mickens J, Seltzer M, Chen H. SIGL: Securing software installations through deep graph learning. In Proc. the 30th USENIX Security Symposium, Aug. 2021, pp.2345–2362.
    [2]
    Park J, Yoo D, Yun N, Lee J, Kim D. A thread chaining attack for bypassing a DLL injection monitoring system. In Proc. the 2024 IEEE International Conference on Consumer Electronics (ICCE), Jan. 2024. DOI: 10.1109/ICCE59016.2024.10444377.
    [3]
    Ashawa M, Owoh N P, Riley J, Osamor J, Hosseinzadeh S. An exploration of shared code execution for malware analysis. In Proc. the 2024 International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA), Feb. 2024. DOI: 10.1109/ACDSA59508.2024.10467679.
    [4]
    Yu C, Xiao Y, Lu J, Li Y, Li Y, Li L, Dong Y, Wang J, Shi J, Bo D, Huo W. File hijacking vulnerability: The elephant in the room. In Proc. the 2024 Network and Distributed System Security Symposium, Feb. 26 -Mar. 1, 2024. DOI: 10.14722/ndss.2024.23038.
    [5]
    Stewart A. DLL side-loading: A thorn in the side of the anti-virus (AV) industry. Technical Report, FireEye Inc, 2014. https://www.mandiant.com/sites/default/files/2021-09/rpt-dll-sideloading.pdf, May 2025.
    [6]
    Dick S, Volmar D. DLL hell: Software dependencies, failure, and the maintenance of Microsoft Windows. IEEE Annals of the History of Computing, 2018, 40(4): 28–51. DOI: 10.1109/MAHC.2018.2877913.
    [7]
    Dora J R, Hluchy L. Exploitation of thick client application vulnerabilities and a synopsis of mitigation: *How to conduct attacks to abuse weaknesses present in a Windows executable file. In Proc. the 18th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI), May 2024, pp.431–436. DOI: 10.1109/SACI60582.2024.10619849.
    [8]
    Cheng B, Ming J, Fu J, Peng G, Chen T, Zhang X, Marion J Y. Towards paving the way for large-scale Windows malware analysis: Generic binary unpacking with orders-of-magnitude performance boost. In Proc. the 2018 ACM SIGSAC Conference on Computer and Communications Security, Oct. 2018, pp.395–411. DOI: 10.1145/3243734.3243771.
    [9]
    Gittins Z, Soltys M. Malware persistence mechanisms. Procedia Computer Science, 2020, 176: 88–97. DOI: 10.1016/j.procs.2020.08.010.
    [10]
    Kwon T, Su Z. Automatic detection of unsafe component loadings. In Proc. the 19th International Symposium on Software Testing and Analysis, Jul. 2010, pp.107–118. DOI: 10.1145/1831708.1831722.
    [11]
    Kwon T, Su Z. Automatic detection of unsafe dynamic component loadings. IEEE Trans. Software Engineering, 2012, 38(2): 293–313. DOI: 10.1109/TSE.2011.108.
    [12]
    Kwon T, Su Z. Static detection of unsafe component loadings. In Proc. the 21st International Conference on Compiler Construction, Apr. 2012, pp.122–143. DOI: 10.1007/978-3-642-28652-0_7.
    [13]
    Min B, Varadharajan V. Secure dynamic software loading and execution using cross component verification. In Proc. the 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, Jun. 2015, pp.113–124. DOI: 10.1109/DSN.2015.17.
    [14]
    Min B, Varadharajan V. Rethinking software component security: Software component level integrity and cross verification. The Computer Journal, 2016, 59(11): 1735–1748. DOI: 10.1093/comjnl/bxw047.
    [15]
    Fernández-Álvarez P, Rodríguez R J. Module extraction and DLL hijacking detection via single or multiple memory dumps. Forensic Science International: Digital Investigation, 2023, 44 Suppl 1: 301505. DOI: 10.1016/j.fsidi.2023.301505.
    [16]
    Verdier A, Laborde R, Kandi M A, Benzekri A. A SLAHP in the face of DLL search order hijacking. In Proc. the 3rd International Conference on Ubiquitous Security, Nov. 2023, pp.177–190. DOI: 10.1007/978-981-97-1274-8_12.
    [17]
    Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation, 1997, 9(8): 1735–1780. DOI: 10.1162/neco.1997.9.8.1735.
    [18]
    Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct. 2014, pp.1724–1734. DOI: 10.3115/v1/D14-1179.
    [19]
    Kasongo S M. A deep learning technique for intrusion detection system using a recurrent neural networks based framework. Computer Communications, 2023, 199: 113–125. DOI: 10.1016/j.comcom.2022.12.010.
    [20]
    Imrana Y, Xiang Y, Ali L, Noor A, Sarpong K, Abdullah M A. CNN-GRU-FF: A double-layer feature fusion-based network intrusion detection system using convolutional neural network and gated recurrent units. Complex & Intelligent Systems, 2024, 10(3): 3353–3370. DOI: 10.1007/s40747-023-01313-y.
    [21]
    Lei X, Xia Y, Wang A, Jian X, Zhong H, Sun L. Mutual information based anomaly detection of monitoring data with attention mechanism and residual learning. Mechanical Systems and Signal Processing, 2023, 182: 109607. DOI: 10.1016/j.ymssp.2022.109607.
    [22]
    Chen Y, Xia R, Yang K, Zou K. DNNAM: Image inpainting algorithm via deep neural networks and attention mechanism. Applied Soft Computing, 2024, 154: 111392. DOI: 10.1016/j.asoc.2024.111392.
    [23]
    Zhao J, Guo S, Mu D. DouBiGRU-A: Software defect detection algorithm based on attention mechanism and double BiGRU. Computers & Security, 2021, 111: 102459. DOI: 10.1016/j.cose.2021.102459.
    [24]
    Vishnu P R, Vinod P, Yerima S Y. A deep learning approach for classifying vulnerability descriptions using self attention based neural network. Journal of Network and Systems Management, 2022, 30(1): Article No. 9. DOI: 10.1007/s10922-021-09624-6.
    [25]
    Han J, Huang C, Sun S, Liu Z, Liu J. bjXnet: An improved bug localization model based on code property graph and attention mechanism. Automated Software Engineering, 2023, 30(1): Article No. 12. DOI: 10.1007/s10515-023-00379-9.
    [26]
    Roodschild M, Sardiñas J G, Will A. A new approach for the vanishing gradient problem on sigmoid activation. Progress in Artificial Intelligence, 2020, 9(4): 351–360. DOI: 10.1007/s13748-020-00218-y.
    [27]
    Dubey S R, Singh S K, Chaudhuri B B. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 2022, 503: 92–108. DOI: 10.1016/j.neucom.2022.06.111.
    [28]
    Marcot B G, Hanea A M. What is an optimal value of k in k-fold cross-validation in discrete Bayesian network analysis? Computational Statistics, 2021, 36(3): 2009–2031. DOI: 10.1007/s00180-020-00999-9.
    • Relative Articles
  • Supplements (3)

  • Others

Catalog

    Get Citation
    PDF
    XML
    Read Online
    Article views (90) PDF downloads (11) Cited by()
    Related

    Copyright ©2025 JCST, All Rights Reserved     京ICP备05002829号-1

    Institute of Computing Technology, Chinese Academy of Sciences
    P.O. Box 2704, Beijing 100190, P.R. China

    Tel.: 86-10-62610746 E-mail: jcst@ict.ac.cn

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return