Journal of Computer Science and Technology ›› 2021, Vol. 36 ›› Issue (2): 334-346.doi: 10.1007/s11390-021-0861-7

Special Issue: Emerging Areas

• Special Section on AI and Big Data Analytics in Biology and Medicine • Previous Articles     Next Articles

Robust Needle Localization and Enhancement Algorithm for Ultrasound by Deep Learning and Beam Steering Methods

Jun Gao1,2, Paul Liu2, Guang-Di Liu3, and Le Zhang1,4,5,*, Senior Member, CCF, Member, ACM        

  1. 1 College of Computer Science, Sichuan University, Chengdu 610065, China;
    2 Stork Healthcare, Chengdu 610000, China;
    3 College of Computer and Information Science, Southwest University, Chongqing 400715, China;
    4 West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu 610065, China;
    5 Pera Corporation Ltd., Beijing 100025, China
  • Received:2020-07-29 Revised:2021-02-25 Online:2021-03-05 Published:2021-04-01
  • Contact: Le Zhang
  • About author:Jun Gao received his B.S. and M.S. degrees in computer science from Sichuan University, Chengdu, in 2005 and 2008 respectively. Currently, he is a Ph.D. candidate at the College of Computer Science, Sichuan University, Chengdu. His research interests involve medical image processing, computer vision, and machine learning.
  • Supported by:
    This work was supported by the National Science and Technology Major Project of China under Grant No. 2018ZX10201002.

Ultrasound (US) imaging is clinically used to guide needle insertions because it is safe, real-time, and low cost. The localization of the needle in the ultrasound image, however, remains a challenging problem due to specular reflection off the smooth surface of the needle, speckle noise, and similar line-like anatomical features. This study presents a novel robust needle localization and enhancement algorithm based on deep learning and beam steering methods with three key innovations. First, we employ beam steering to maximize the reflection intensity of the needle, which can help us to detect and locate the needle precisely. Second, we modify the U-Net which is an end-to-end network commonly used in biomedical segmentation by using two branches instead of one in the last up-sampling layer and adding three layers after the last down-sample layer. Thus, the modified U-Net can real-time segment the needle shaft region, detect the needle tip landmark location and determine whether an image frame contains the needle by one shot. Third, we develop a needle fusion framework that employs the outputs of the multi-task deep learning (MTL) framework to precisely locate the needle tip and enhance needle shaft visualization. Thus, the proposed algorithm can not only greatly reduce the processing time, but also significantly increase the needle localization accuracy and enhance the needle visualization for real-time clinical intervention applications.

Key words: ultrasound; deep learning; segmentation; classification; optimization;

[1] Gulsen F, Kantarci F. Ultrasound-guided intervention around the hip joint. American Journal of Roentgenology, 2012, 198(1):W95. DOI:10.2214/AJR.11.7627.
[2] Chin K J, Perlas A, Chan V W S, Brull R. Needle visualization in ultrasound-guided regional anesthesia:Challenges and solutions. Regional Anesthesia and Pain Medicine, 2008, 33(6):532-544. DOI:10.1016/j.rapm.2008.06.002.
[3] Fevre M C, Vincent C, Picard J, Vighetti A, Chapuis C, Detavernier M, Allenet B, Payen J F, Bosson J L, Albaladejo P. Reduced variability and execution time to reach a target with a needle GPS system:Comparison between physicians, residents and nurse anaesthetists. Anaesthesia Critical Care & Pain Medicine, 2018, 37(1):55-60. DOI:10.1016/j.accpm.2016.05.008.
[4] Stolka P J, Foroughi P, Rendina M, Weiss C R, Hager G D, Boctor E M. Needle guidance using handheld stereo vision and projection for ultrasound-based interventions. In Proc. the 17th International Conference on Medical Image Computing and Computer-Assisted Intervention, September 2014, pp.684-691. DOI:10.1007/978-3-319-10470-685.
[5] Lu H, Li J, Lu Q, Bharat S, Erkamp R, Chen B, Drysdale J, Vignon F, Jain A. A new sensor technology for 2D ultrasound-guided needle tracking. In Proc. the 17th International Conference on Medical Image Computing and Computer-Assisted Intervention, September 2014, pp.389-396. DOI:10.1007/978-3-319-10470-649.
[6] Xia W, West S J, Finlay M C, Mari J M, Ourselin S, David A L, Desjardins A E. Looking beyond the imaging plane:3D needle tracking with a linear array ultrasound probe. Sci. Rep., 2017, 7(1):Article No. 3674. DOI:10.1038/s41598-017-03886-4.
[7] Ding M, Fenster A. Projection-based needle segmentation in 3D ultrasound images. In Proc. the 6th International Conference on Medical Image Computing and ComputerAssisted Intervention, November 2003, pp.319-327. DOI:10.3109/10929080500079321.
[8] Qiu W, Yuchi M, Ding M. Phase grouping-based needle segmentation in 3-D trans-rectal ultrasoundguided prostate trans-perineal therapy. Ultrasound in Medicine & Biology, 2014, 40(4):804-816. DOI:10.1016/j.ultrasmedbio.2013.11.004.
[9] Beigi P, Rohling R, Salcudean T, Lessoway V A, Ng G C. Needle trajectory and tip localization in realtime 3-D ultrasound using a moving stylus. Ultrasound in Medicine & Biology, 2015, 41(7):2057-2070. DOI:10.1016/j.ultrasmedbio.2015.03.013.
[10] Zhao Y, Shen Y, Bernard A, Cachard C, Liebgott H. Evaluation and comparison of current biopsy needle localization and tracking methods using 3D ultrasound. Ultrasonics, 2017, 73:206-220. DOI:10.1016/j.ultras.2016.09.006.
[11] Ayvali E, Desai J P. Optical flow-based tracking of needles and needle-tip localization using circular hough transform in ultrasound images. Annals of Biomedical Engineering, 2015, 43(8):1828-1840. DOI:10.1007/s10439-014-1208-0.
[12] Uhercik M, Kybic J, Liebgott H, Cachard C. Model fitting using RANSAC for surgical tool localization in 3D ultrasound images. IEEE Transactions on Biomedical Engineering, 2010, 57(8):1907-1916. DOI:10.1109/TBME.2010.2046416.
[13] Zhao Y, Cachard C, Liebgott H. Automatic needle detection and tracking in 3D ultrasound using an ROI-based RANSAC and Kalman method. Ultrasonic Imaging, 2013, 35(4):283-306. DOI:10.1177/0161734613502004.
[14] Mwikirize C, Nosher J L, Hacihaliloglu I. Learning needle tip localization from digital subtraction in 2D ultrasound. International Journal of Computer Assisted Radiology and Surgery, 2019, 14(6):1017-1026. DOI:10.1007/s11548-019-01951-z.
[15] Mwikirize C, Nosher J L, Hacihaliloglu I. Convolution neural networks for real-time needle detection and localization in 2D ultrasound. International Journal of Computer Assisted Radiology and Surgery, 2018, 13(5):647-657. DOI:10.1007/s11548-018-1721-y.
[16] Beigi P, Rohling R, Salcudean S E, Ng G C. CASPER:Computer-aided segmentation of imperceptible motion-A learning-based tracking of an invisible needle in ultrasound. International Journal of Computer Assisted Radiology and Surgery, 2017, 12(11):1857-1866. DOI:10.1007/s11548-017-1631-4.
[17] Pourtaherian A, Ghazvinian Zanjani F, Zinger S, Mihajlovic N, Ng G C, Korsten H H M, De With P H N. Robust and semantic needle detection in 3D ultrasound using orthogonal-plane convolutional neural networks. International Journal of Computer Assisted Radiology and Surgery, 2018, 13(9):1321-1333. DOI:10.1007/s11548-018-1798-3.
[18] Arif M, Moelker A, Van Walsum T. Automatic needle detection and real-time Bi-planar needle visualization during 3D ultrasound scanning of the liver. Medical Image Analysis, 2019, 53:104-110. DOI:10.1016/
[19] Mwikirize C, Nosher J L, Hacihaliloglu I. Single shot needle tip localization in 2D ultrasound. In Proc. the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, October 2019, pp.637-645. DOI:10.1007/978-3-030-32254-071.
[20] Hatt C R, Ng G, Parthasarathy V. Enhanced needle localization in ultrasound using beam steering and learning-based segmentation. Computerized Medical Imaging and Graphics, 2015, 41:46-54. DOI:10.1016/j.compmedimag.2014.06.016.
[21] Ronneberger O, Fischer P, Brox T. U-Net:Convolutional networks for biomedical image segmentation. In Proc. the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, October 2015, pp. 234-241. DOI:10.1007/978-3-319-24574-428.
[22] Gao J, Liu P, Liu G D, Zhang L. Supplementary material. Technical Report, Sichuan University, 2020. Material.pdf, Dec. 2020.
[23] Jiang B, Struthers A, Sun Z, Feng Z, Zhao X, Zhao K, Dai W, Zhou X, Berens M E, Zhang L. Employing graphics processing unit technology, alternating direction implicit method and domain decomposition to speed up the numerical diffusion solver for the biomedical engineering research. International Journal for Numerical Methods in Biomedical Engineering, 2011, 27(11):1829-1849. DOI:10.1002/cnm.1444.
[24] Jiang B N, Dai W Z, Khaliq A, Carey M, Zhou X B, Zhang L. Novel 3D GPU based numerical parallel diffusion algorithms in cylindrical coordinates for health care simulation. Mathematics and Computers in Simulation, 2015, 109:1-19. DOI:10.1016/j.matcom.2014.07.003.
[25] Zhang L, Jiang B, Wu Y, Strouthos C, Sun P Z, Su J, Zhou X. Developing a multiscale, multi-resolution agentbased brain tumor model by graphics processing units. Theoretical Biology and Medical Modelling, 2011, 8(1):Article No. 46. DOI:10.1186/1742-4682-8-46.
[26] Isola P, Zhu J Y, Zhou T, Efros A. Image-to-image translation with conditional adversarial networks. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.5967-5976. DOI:10.1109/CVPR.2017.632.
[27] Wu W, Song L, Yang Y, Wang J, Liu H, Zhang L. Exploring the dynamics and interplay of human papillomavirus and cervical tumorigenesis by integrating biological data into a mathematical model. BMC Bioinformatics, 2020, 21(Suppl 7):Article No. 152. DOI:10.1186/s12859-020-3454-5.
[28] Ren S, He K, Girshick R, Sun J. Faster R-CNN:Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149. DOI:10.1109/TPAMI.2016.2577031.
[29] Xu Z, Huo Y, Park J, Landman B, Milkowski A, Grbic S, Zhou S. Less is more:Simultaneous view classification and landmark detection for abdominal ultrasound images. In Proc. the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention, September 2018, pp.711-719. DOI:10.1007/978-3-030-00934-279.
[30] Powers D. Evaluation:From precision, recall and Ffactor to ROC, informedness, markedness & correlation. arXiv:2010.16061, 2008., Jan. 2021.
[31] Xia Y, Yang C, Hu N, Yang Z, He X, Li T, Zhang L. Exploring the key genes and signaling transduction pathways related to the survival time of glioblastoma multiforme patients by a novel survival analysis model. BMC Genomics, 2017, 18(Suppl 1):Article No. 950. DOI:10.1186/s12864-016-3256-3.
[32] Zhang L, Bai W, Yuan N, Du Z. Comprehensively benchmarking applications for detecting copy number variation. PLoS Computational Biology, 2019, 15(5):Article No. e1007069. DOI:10.1371/journal.pcbi.1007069.
[33] Zhang L, Li J, Yin K, Jiang Z, Li T, Hu R, Yu Z, Feng H, Chen Y. Computed tomography angiography-based analysis of high-risk intracerebral haemorrhage patients by employing a mathematical model. BMC Bioinformatics, 2019, 20(Suppl 7):Article No. 193. DOI:10.1186/s12859-019-2741-5.
[34] Zhang L, Liu Y, Wang M, Wu Z, Li N, Zhang J, Yang C. EZH2-, CHD4-, and IDH-linked epigenetic perturbation and its association with survival in glioma patients. J. Mol. Cell Biol., 2017, 9(6):477-488. DOI:10.1093/jmcb/mjx056.
[35] Kremkau F W, Taylor K J. Artifacts in ultrasound imaging. Journal of Ultrasound in Medicine, 1986, 5(4):227-237. DOI:10.7863/jum.1986.5.4.227.
[36] Paul Y, Barthez D, Léveillé R, Peter V, Scrivani D. Side lobes and grating lobes artifacts in ultrasound imaging. Veterinary Radiology & Ultrasound, 1997, 38(5):387-393. DOI:10.1111/j.1740-8261.1997.tb02104.x.
[37] Matalon T A, Silver B. US guidance of interventional procedures. Radiology, 1990, 174(1):43-47. DOI:10.1148/radiology.174.1.2403684.
[38] Sofka C, Collins A J, Adler R. Use of ultrasonographic guidance in interventional musculoskeletal procedures:A review from a single institution. Journal of Ultrasound in Medicine:Official Journal of the American Institute of Ultrasound in Medicine, 2001, 20(1):21-26. DOI:10.7863/jum.2001.20.1.21.
[1] Xin Zhang, Siyuan Lu, Shui-Hua Wang, Xiang Yu, Su-Jing Wang, Lun Yao, Yi Pan, and Yu-Dong Zhang. Diagnosis of COVID-19 Pneumonia via a Novel Deep Learning Architecture [J]. Journal of Computer Science and Technology, 2022, 37(2): 330-343.
[2] Que-Ping Kong, Zi-Yan Wang, Yuan Huang, Xiang-Ping Chen, Xiao-Cong Zhou, Zi-Bin Zheng, and Gang Huang. Characterizing and Detecting Gas-Inefficient Patterns in Smart Contracts [J]. Journal of Computer Science and Technology, 2022, 37(1): 67-82.
[3] Yu-Wei Wu, Qing-Gang Wang, Long Zheng, Xiao-Fei Liao, Hai Jin, Wen-Bin Jiang, Ran Zheng, Kan Hu. FDGLib: A Communication Library for Efficient Large-Scale Graph Processing in FPGA-Accelerated Data Centers [J]. Journal of Computer Science and Technology, 2021, 36(5): 1051-1070.
[4] Xiao-Jing Zha, Yin-Shui Xia, Shang-Luan Xie, Zhu-Fei Chu. Defect-Tolerant Mapping of CMOL Circuit Targeting Delay Optimization [J]. Journal of Computer Science and Technology, 2021, 36(5): 1118-1132.
[5] Hui-Ming Tian, Zhu-Fei Chu. Inversion Optimization Strategy Based on Primitives with Complement Attributes [J]. Journal of Computer Science and Technology, 2021, 36(5): 1145-1154.
[6] Songjie Niu, Shimin Chen. TransGPerf: Exploiting Transfer Learning for Modeling Distributed Graph Computation Performance [J]. Journal of Computer Science and Technology, 2021, 36(4): 778-791.
[7] Zhi-Xin Qi, Hong-Zhi Wang, An-Jie Wang. Impacts of Dirty Data on Classification and Clustering Models: An Experimental Evaluation [J]. Journal of Computer Science and Technology, 2021, 36(4): 806-821.
[8] Jing-Xuan Zhang, Chuan-Qi Tao, Zhi-Qiu Huang, Xin Chen. Discovering API Directives from API Specifications with Text Classification [J]. Journal of Computer Science and Technology, 2021, 36(4): 922-943.
[9] Wen-Jun Yang, Bei-Ji Zou, Kai-Wen Li, Shu Liu. A Character Flow Framework for Multi-Oriented Scene Text Detection [J]. Journal of Computer Science and Technology, 2021, 36(3): 465-477.
[10] Lan Chen, Juntao Ye, Xiaopeng Zhang. Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis [J]. Journal of Computer Science and Technology, 2021, 36(3): 478-493.
[11] Yu-Jie Yuan, Yukun Lai, Tong Wu, Lin Gao, Li-Gang Liu. A Revisit of Shape Editing Techniques: From the Geometric to the Neural Viewpoint [J]. Journal of Computer Science and Technology, 2021, 36(3): 520-554.
[12] Sheng-Luan Hou, Xi-Kun Huang, Chao-Qun Fei, Shu-Han Zhang, Yang-Yang Li, Qi-Lin Sun, Chuan-Qing Wang. A Survey of Text Summarization Approaches Based on Deep Learning [J]. Journal of Computer Science and Technology, 2021, 36(3): 633-663.
[13] Wei Du, Yu Sun, Hui-Min Bao, Liang Chen, Ying Li, Yan-Chun Liang. DeepHBSP: A Deep Learning Framework for Predicting Human Blood-Secretory Proteins Using Transfer Learning [J]. Journal of Computer Science and Technology, 2021, 36(2): 234-247.
[14] Yang-Jie Cao, Shuang Wu, Chang Liu, Nan Lin, Yuan Wang, Cong Yang, Jie Li. Seg-CapNet: A Capsule-Based Neural Network for the Segmentation of Left Ventricle from Cardiac Magnetic Resonance Imaging [J]. Journal of Computer Science and Technology, 2021, 36(2): 323-333.
[15] Hua Chen, Juan Liu, Qing-Man Wen, Zhi-Qun Zuo, Jia-Sheng Liu, Jing Feng, Bao-Chuan Pang, Di Xiao. CytoBrain: Cervical Cancer Screening System Based on Deep Learning Technology [J]. Journal of Computer Science and Technology, 2021, 36(2): 347-360.
Full text



[1] Feng Yulin;. Recursive Implementation of VLSI Circuits[J]. , 1986, 1(2): 72 -82 .
[2] Sun Zhongxiu; Shang Lujun;. DMODULA:A Distributed Programming Language[J]. , 1986, 1(2): 25 -31 .
[3] Wang Jianchao; Wei Daozheng;. An Effective Test Generation Algorithm for Combinational Circuits[J]. , 1986, 1(4): 1 -16 .
[4] Qiao Xiangzhen;. An Efficient Parallel Algorithm for FFT[J]. , 1987, 2(3): 174 -190 .
[5] Huang Guoxiang; Liu Jian;. A Key-Lock Access Control[J]. , 1987, 2(3): 236 -243 .
[6] Shi Weigeng; StephenY.H.Su;. An Online Diagnosable Fault-Tolerant Redundancy System[J]. , 1987, 2(4): 310 -321 .
[7] Li Renwei;. Soundness and Completeness of Kung s Reasoning Procedure[J]. , 1988, 3(1): 7 -15 .
[8] Wang Jianchao; Wei Daozheng;. Reconvergent-Fanout-Oriented Testability Measure[J]. , 1988, 3(1): 16 -28 .
[9] Jin Lingzi; Zhu Hong;. Systems Programming in the Functional Language FP[J]. , 1988, 3(1): 40 -55 .
[10] Zhang Yan; He Jichao;. Data Dependencies in Database with Incomplete Information[J]. , 1988, 3(2): 131 -138 .

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
  Copyright ©2015 JCST, All Rights Reserved