SCIE, EI, Scopus, INSPEC, DBLP, CSCD, etc.
Citation: | Chen X, Sha LT, Xiao F et al. AB-DHD: An attention mechanism and bi-directional gated recurrent unit based model for dynamic link library hijacking vulnerability discovery. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY, 40(3): 887−903, May 2025. DOI: 10.1007/s11390-025-4497-x |
With the rapid development of operating systems, attacks on system vulnerabilities are increasing. Dynamic link library (DLL) hijacking is prevalent in installers on freeware platforms and is highly susceptible to exploitation by malware attackers. However, existing studies are based solely on the load paths of DLLs, ignoring the attributes of installers and invocation modes, resulting in low accuracy and weak generality of vulnerability detection. In this paper, we propose a novel model, AB-DHD, which is based on an attention mechanism and a bi-directional gated recurrent unit (BiGRU) neural network for DLL hijacking vulnerability discovery. While BiGRU is an enhancement of GRU and has been widely applied in sequence data processing, a double-layer BiGRU network is introduced to analyze the internal features of installers with DLL hijacking vulnerabilities. Additionally, an attention mechanism is incorporated to dynamically adjust feature weights, significantly enhancing the ability of our model to detect vulnerabilities in new installers. A comprehensive “List of Easily Hijacked DLLs” is developed to serve a reference for future studies. We construct an EXEFul dataset and a DLLVul dataset, using data from two publicly available authoritative vulnerability databases, Common Vulnerabilities & Exposures (CVE) and China National Vulnerability Database (CNVD), and mainstream installer distribution platforms. Experimental results show that our model outperforms popular automated tools like Rattler and DLLHSC, achieving an accuracy of 97.79% and a recall of 94.72%. Moreover, 17 previously unknown vulnerabilities have been identified, and corresponding vulnerability certifications have been assigned.
[1] |
Han X, Yu X, Pasquier T, Li D, Rhee J, Mickens J, Seltzer M, Chen H. SIGL: Securing software installations through deep graph learning. In Proc. the 30th USENIX Security Symposium, Aug. 2021, pp.2345–2362.
|
[2] |
Park J, Yoo D, Yun N, Lee J, Kim D. A thread chaining attack for bypassing a DLL injection monitoring system. In Proc. the 2024 IEEE International Conference on Consumer Electronics (ICCE), Jan. 2024. DOI: 10.1109/ICCE59016.2024.10444377.
|
[3] |
Ashawa M, Owoh N P, Riley J, Osamor J, Hosseinzadeh S. An exploration of shared code execution for malware analysis. In Proc. the 2024 International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA), Feb. 2024. DOI: 10.1109/ACDSA59508.2024.10467679.
|
[4] |
Yu C, Xiao Y, Lu J, Li Y, Li Y, Li L, Dong Y, Wang J, Shi J, Bo D, Huo W. File hijacking vulnerability: The elephant in the room. In Proc. the 2024 Network and Distributed System Security Symposium, Feb. 26 -Mar. 1, 2024. DOI: 10.14722/ndss.2024.23038.
|
[5] |
Stewart A. DLL side-loading: A thorn in the side of the anti-virus (AV) industry. Technical Report, FireEye Inc, 2014. https://www.mandiant.com/sites/default/files/2021-09/rpt-dll-sideloading.pdf, May 2025.
|
[6] |
Dick S, Volmar D. DLL hell: Software dependencies, failure, and the maintenance of Microsoft Windows. IEEE Annals of the History of Computing, 2018, 40(4): 28–51. DOI: 10.1109/MAHC.2018.2877913.
|
[7] |
Dora J R, Hluchy L. Exploitation of thick client application vulnerabilities and a synopsis of mitigation: *How to conduct attacks to abuse weaknesses present in a Windows executable file. In Proc. the 18th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI), May 2024, pp.431–436. DOI: 10.1109/SACI60582.2024.10619849.
|
[8] |
Cheng B, Ming J, Fu J, Peng G, Chen T, Zhang X, Marion J Y. Towards paving the way for large-scale Windows malware analysis: Generic binary unpacking with orders-of-magnitude performance boost. In Proc. the 2018 ACM SIGSAC Conference on Computer and Communications Security, Oct. 2018, pp.395–411. DOI: 10.1145/3243734.3243771.
|
[9] |
Gittins Z, Soltys M. Malware persistence mechanisms. Procedia Computer Science, 2020, 176: 88–97. DOI: 10.1016/j.procs.2020.08.010.
|
[10] |
Kwon T, Su Z. Automatic detection of unsafe component loadings. In Proc. the 19th International Symposium on Software Testing and Analysis, Jul. 2010, pp.107–118. DOI: 10.1145/1831708.1831722.
|
[11] |
Kwon T, Su Z. Automatic detection of unsafe dynamic component loadings. IEEE Trans. Software Engineering, 2012, 38(2): 293–313. DOI: 10.1109/TSE.2011.108.
|
[12] |
Kwon T, Su Z. Static detection of unsafe component loadings. In Proc. the 21st International Conference on Compiler Construction, Apr. 2012, pp.122–143. DOI: 10.1007/978-3-642-28652-0_7.
|
[13] |
Min B, Varadharajan V. Secure dynamic software loading and execution using cross component verification. In Proc. the 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, Jun. 2015, pp.113–124. DOI: 10.1109/DSN.2015.17.
|
[14] |
Min B, Varadharajan V. Rethinking software component security: Software component level integrity and cross verification. The Computer Journal, 2016, 59(11): 1735–1748. DOI: 10.1093/comjnl/bxw047.
|
[15] |
Fernández-Álvarez P, Rodríguez R J. Module extraction and DLL hijacking detection via single or multiple memory dumps. Forensic Science International: Digital Investigation, 2023, 44 Suppl 1: 301505. DOI: 10.1016/j.fsidi.2023.301505.
|
[16] |
Verdier A, Laborde R, Kandi M A, Benzekri A. A SLAHP in the face of DLL search order hijacking. In Proc. the 3rd International Conference on Ubiquitous Security, Nov. 2023, pp.177–190. DOI: 10.1007/978-981-97-1274-8_12.
|
[17] |
Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation, 1997, 9(8): 1735–1780. DOI: 10.1162/neco.1997.9.8.1735.
|
[18] |
Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct. 2014, pp.1724–1734. DOI: 10.3115/v1/D14-1179.
|
[19] |
Kasongo S M. A deep learning technique for intrusion detection system using a recurrent neural networks based framework. Computer Communications, 2023, 199: 113–125. DOI: 10.1016/j.comcom.2022.12.010.
|
[20] |
Imrana Y, Xiang Y, Ali L, Noor A, Sarpong K, Abdullah M A. CNN-GRU-FF: A double-layer feature fusion-based network intrusion detection system using convolutional neural network and gated recurrent units. Complex & Intelligent Systems, 2024, 10(3): 3353–3370. DOI: 10.1007/s40747-023-01313-y.
|
[21] |
Lei X, Xia Y, Wang A, Jian X, Zhong H, Sun L. Mutual information based anomaly detection of monitoring data with attention mechanism and residual learning. Mechanical Systems and Signal Processing, 2023, 182: 109607. DOI: 10.1016/j.ymssp.2022.109607.
|
[22] |
Chen Y, Xia R, Yang K, Zou K. DNNAM: Image inpainting algorithm via deep neural networks and attention mechanism. Applied Soft Computing, 2024, 154: 111392. DOI: 10.1016/j.asoc.2024.111392.
|
[23] |
Zhao J, Guo S, Mu D. DouBiGRU-A: Software defect detection algorithm based on attention mechanism and double BiGRU. Computers & Security, 2021, 111: 102459. DOI: 10.1016/j.cose.2021.102459.
|
[24] |
Vishnu P R, Vinod P, Yerima S Y. A deep learning approach for classifying vulnerability descriptions using self attention based neural network. Journal of Network and Systems Management, 2022, 30(1): Article No. 9. DOI: 10.1007/s10922-021-09624-6.
|
[25] |
Han J, Huang C, Sun S, Liu Z, Liu J. bjXnet: An improved bug localization model based on code property graph and attention mechanism. Automated Software Engineering, 2023, 30(1): Article No. 12. DOI: 10.1007/s10515-023-00379-9.
|
[26] |
Roodschild M, Sardiñas J G, Will A. A new approach for the vanishing gradient problem on sigmoid activation. Progress in Artificial Intelligence, 2020, 9(4): 351–360. DOI: 10.1007/s13748-020-00218-y.
|
[27] |
Dubey S R, Singh S K, Chaudhuri B B. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 2022, 503: 92–108. DOI: 10.1016/j.neucom.2022.06.111.
|
[28] |
Marcot B G, Hanea A M. What is an optimal value of k in k-fold cross-validation in discrete Bayesian network analysis? Computational Statistics, 2021, 36(3): 2009–2031. DOI: 10.1007/s00180-020-00999-9.
|