Journal of Computer Science and Technology ›› 2022, Vol. 37 ›› Issue (3): 666-679.doi: 10.1007/s11390-022-2173-y

Special Issue: Artificial Intelligence and Pattern Recognition; Computer Graphics and Multimedia

• Special Section of CVM 2022 • Previous Articles     Next Articles

ARSlice: Head-Mounted Display Augmented with Dynamic Tracking and Projection

Yu-Ping Wang1 (王瑀屏), Member, CCF, IEEE, Sen-Wei Xie1 (解森炜), Li-Hui Wang2,3,* (王立辉), Member, IEEE, Hongjin Xu4 (徐鸿金), Satoshi Tabata3, Member, ACM, and Masatoshi Ishikawa3,5        

  1. 1Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
    2Institute of Semiconductors, Guangdong Academy of Sciences, Guangzhou 510651, China
    3Data Science Research Division, Information Technology Center, The University of Tokyo, Tokyo 113-8656, Japan
    4Core Network Department, Advanced Operations Development Division, KDDI Cooperation, Tokyo 206-0034, Japan
    5Tokyo University of Science, Tokyo 113-8601, Japan
  • Received:2022-01-20 Revised:2022-04-08 Accepted:2022-04-24 Online:2022-05-30 Published:2022-05-30
  • Contact: Li-Hui Wang E-mail:wanglihui@gdisit.com
  • About author:Li-Hui Wang received his B.E. degree from Shenyang University of Technology, Shenyang, in 2007, and his M.E. degree in engineering from Northeastern University, Shenyang, in 2009, and his Ph.D. degree in information science and technology from the University of Tokyo, Tokyo, in 2014. He was a project assistant professor and project researcher at the University of Tokyo, Tokyo, till 2019. He is currently a processor in engineering at the Institute of Semiconductors, Guangdong Academy of Sciences, Guangzhou. His current research interests include adaptive optics system, high speed vision, and dynamic interaction.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China under Grant No. 61872210, the Guangdong Basic and Applied Basic Research Foundation under Grant Nos. 2021A1515012596 and 2021B1515120064, and the Guangdong Academy of Sciences Special Foundation under Grant No. 2021GDASYL-20210102006.

PDF (PC)

39

Supplement

1

Computed tomography (CT) generates cross-sectional images of the body. Visualizing CT images has been a challenging problem. The emergence of the augmented and virtual reality technology has provided promising solutions. However, existing solutions suffer from tethered display or wireless transmission latency. In this paper, we present ARSlice, a proof-of-concept prototype that can visualize CT images in an untethered manner without wireless transmission latency. Our ARSlice prototype consists of two parts, the user end and the projector end. By employing dynamic tracking and projection, the projector end can track the user-end equipment and project CT images onto it in real time. The user-end equipment is responsible for displaying these CT images into the 3D space. Its main feature is that the user-end equipment is a pure optical device with light weight, low cost, and no energy consumption. Our experiments demonstrate that our ARSlice prototype provides part of six degrees of freedom for the user, and a high frame rate. By interactively visualizing CT images into the 3D space, our ARSlice prototype can help untrained users better understand that CT images are slices of a body.

Key words: augmented and virtual reality; computed tomography (CT) image visualization; interactive visualization;

[1] Raskar R, Welch G, Low K, Bandyopadhyay D. Shader lamps: Animating real objects with image-based illumination. In Proc. the 12th Eurographics Workshop on Rendering Techniques, June 2001, pp.89-102. DOI: 10.2312/EGWR/EGWR01/089-101.

[2] Benko H, Ofek E, Zheng F, Wilson A D. FoveAR: Combining an optically see-through near-eye display with projector-based spatial augmented reality. In Proc. the 28th Annual ACM Symposium on User Interface Software &] Technology, November 2015, pp.129-135. DOI: 10.1145/2807442.2807493.

[3] Lincoln P, Welch G, Nashel A, State A, Ilie A, Fuchs H. Animatronic shader lamps avatars. Virtual Reality, 2011, 15(2/3): 225-238. DOI: 10.1007/s10055-010-0175-5.

[4] Lincoln P,Welch G, Nashel A, Ilie A, State A, Fuchs H. Animatronic shader lamps avatars. In Proc. the 8th IEEE International Symposium on Mixed and Augmented Reality, October 2009, pp.27-33. DOI: 10.1109/ISMAR.2009.5336503.

[5] Narita G, Watanabe Y, Ishikawa M. Dynamic projection mapping onto deforming non-rigid surface using deformable dot cluster marker. IEEE Trans. Vis. Comput. Graph., 2017, 23(3): 1235-1248. DOI: 10.1109/TVCG.2016.2592910.

[6] Wang L, Xu H, Tabata S, Hu Y, Watanabe Y, Ishikawa M. High-speed focal tracking projection based on liquid lens. In Proc. the 2020 ACM SIGGRAPH, August 2020, Article No. 15. DOI: 10.1145/3388534.3408333.

[7] Garcı́a-Berná J A, Sanchez-Gomez J M, Hermanns J, Garcı́a-Mateos G, Alemán J L F. Calcification detection of abdominal aorta in CT images and 3D visualization in VR devices. In Proc. the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, August 2016, pp.4157-4160. DOI: 10.1109/EMBC.2016.7591642.

[8] Azuma R. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 1997, 6(4): 355-385. DOI: 10.1162/pres.1997.6.4.355.

[9] Ibrahim A, Huynh B, Downey J, Höllerer T, Chun D, O’Donovan J. ARbis pictus: A study of vocabulary learning with augmented reality. IEEE Trans. Vis. Comput. Graph., 2018, 24(11): 2867-2874. DOI: 10.1109/TVCG.2018.2868568.

[10] Buttussi F, Chittaro L. Effects of different types of virtual reality display on presence and learning in a safety training scenario. IEEE Trans. Vis. Comput. Graph., 2018, 24(2): 1063-1076. DOI: 10.1109/TVCG.2017.2653117.

[11] Bai Z, Blackwell A F, Coulouris G. Using augmented reality to elicit pretend play for children with autism. IEEE Trans. Vis. Comput. Graph., 2015, 21(5): 598-610. DOI: 10.1109/TVCG.2014.2385092.

[12] Menk C, Koch R. Truthful color reproduction in spatial augmented reality applications. IEEE Trans. Vis. Comput. Graph., 2013, 19(2): 236-248. DOI: 10.1109/TVCG.2012.146.

[13] Liu X, Vlachou C, Qian F, Wang C, Kim K. Firefly: Untethered multi-user VR for commodity mobile devices. In Proc. the 2020 USENIX Annual Technical Conference, July 2020, pp.943-957.

[14] Stauffert J, Niebling F, Latoschik M E. Simultaneous run-time measurement of motion-to-photon latency and latency jitter. In Proc. the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, March 2020, pp.636-644. DOI: 10.1109/VR46266.2020.00086.

[15] Ehnes J, Hirota K, Hirose M. Projected augmentation—Augmented reality using rotatable video projectors. In Proc. the 3rd IEEE and ACM International Symposium on Mixed and Augmented Reality, November 2004, pp.26-35. DOI: 10.1109/ISMAR.2004.47.

[16] Schwerdtfeger B, Pustka D, Hofhauser A, Klinker G. Using laser projectors for augmented reality. In Proc. the 2008 ACM Symposium on Virtual Reality Software and Technology, October 2008, pp.134-137. DOI: 10.1145/1450579.1450608.

[17] Jones B R, Sodhi R, Campbell R H, Garnett G, Bailey B P. Build your world and play in it: Interacting with surface particles on complex objects. In Proc. the 9th IEEE International Symposium on Mixed and Augmented Reality, October 2010, pp.165-174. DOI: 10.1109/ISMAR.2010.5643566.

[18] Hua H, Gao C, Brown L D, Ahuja N, Rolland J P. Using a head-mounted projective display in interactive augmented environments. In Proc. the 4th International Symposium on Augmented Reality, October 2001, pp.217-223. DOI: 10.1109/ISAR.2001.970540.

[19] Hamasaki T, Itoh Y, Hiroi Y, Iwai D, Sugimoto M. HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection. IEEE Trans. Vis. Comput. Graph., 2018, 24(4): 1457-1466. DOI: 10.1109/TVCG.2018.2793659.

[20] Hiroi Y, Itoh Y, Hamasaki T, Iwai D, Sugimoto M. HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection. In Proc. the 2017 IEEE Virtual Reality, March 2017, pp.211-212. DOI: 10.1109/VR.2017.7892251.

[21] Kim K, Billinghurst M, Bruder G, Duh H B, Welch G F. Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008–2017). IEEE Trans. Vis. Comput. Graph., 2018, 24(11): 2947-2962. DOI: 10.1109/TVCG.2018.2868591.

[22] Zhou F, Duh H B, Billinghurst M. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proc. the 7th IEEE and ACM International Symposium on Mixed and Augmented Reality, September 2008, pp.193-202. DOI: 10.1109/ISMAR.2008.4637362.

[23] Liu S, Cheng D, Hua H. An optical see-through head mounted display with addressable focal planes. In Proc. the 7th IEEE and ACM International Symposium on Mixed and Augmented Reality, September 2008, pp.33-42. DOI: 10.1109/ISMAR.2008.4637321.

[24] Chakravarthula P, Dunn D, Aksit K, Fuchs H. FocusAR: Autofocus augmented reality eyeglasses for both real world and virtual imagery. IEEE Trans. Vis. Comput. Graph., 2018, 24(11): 2906-2916. DOI: 10.1109/TVCG.2018.2868532.

[25] Maimone A, Georgiou A, Kollin J S. Holographic near-eye displays for virtual and augmented reality. ACM Trans. Graph., 2017, 36(4): Article No. 85. DOI: 10.1145/3072959.3073624.

[26] Jang C, Bang K, Moon S, Kim J, Lee S, Lee B. Retinal 3D: Augmented reality near-eye display via pupil-tracked light field projection on retina. ACM Trans. Graph., 2017, 36(6): Article No. 190. DOI: 10.1145/3130800.3130889.

[27] Kikinis R, Shenton M E, Iosifescu D V, McCarley R W, Saiviroonporn P, Hokama H H, Robatino A, Metcalf D, Wible C, Portas C M, Donnino R M, Jolesz F A. A digital brain atlas for surgical planning, model-driven segmentation, and teaching. IEEE Trans. Vis. Comput. Graph., 1996, 2(3): 232-241. DOI: 10.1109/2945.537306.

[28] Wu J, Belle A, Hargraves R H, Cockrell C, Tang Y, Najarian K. Bone segmentation and 3D visualization of CT images for traumatic pelvic injuries. Int. J. Imaging Syst. Technol., 2014, 24(1): 29-38. DOI: 10.1002/ima.22076.

[29] Hu Z, Zou J, Gui J, Rong J, Li Y, Xi D, Zheng H. Real-time visualization and interaction of three-dimensional human CT images. J. Comput., 2010, 5(9): 1335-1342. DOI: 10.4304/jcp.5.9.1335-1342.

[30] Jung Y, Kim J, Feng D D. Dual-modal visibility metrics for interactive PET-CT visualization. In Proc. the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, August 28-September 1, 2012, pp.2696-2699. DOI: 10.1109/EMBC.2012.6346520.

[31] Javan R, Rao A, Jeun B S, Herur-Raman A, Singh N, Heidari P. From CT to 3D printed models, serious gaming, and virtual reality: Framework for educational 3D visualization of complex anatomical spaces from within—The pterygopalatine fossa. J. Digit. Imaging, 2020, 33(3): 776-791. DOI: 10.1007/s10278-019-00315-y.

[32] Xing J, Ai H, Lao S. Multi-object tracking through occlusions by local tracklets filtering and global tracklets association with detection responses. In Proc. the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2009, pp.1200-1207. DOI: 10.1109/CVPR.2009.5206745.

[33] Pirsiavash H, Ramanan D, Fowlkes C C. Globally-optimal greedy algorithms for tracking a variable number of objects. In Proc. the 24th IEEE Conference on Computer Vision and Pattern Recognition, June 2011, pp.1201-1208. DOI: 10.1109/CVPR.2011.5995604.

[34] Zhang Z. Flexible camera calibration by viewing a plane from unknown orientations. In Proc. the 7th IEEE International Conference on Computer Vision, September 1999, pp.666-673. DOI: 10.1109/ICCV.1999.791289.

[35] Kato H, Billinghurst M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In Proc. the 2nd IEEE and ACM International Workshop on Augmented Reality, October 1999, pp.85-94. DOI: 10.1109/IWAR.1999.803809.

[36] Gupta S, Jaynes C O. Active pursuit tracking in a projector-camera system with application to augmented reality. In Proc. the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, September 2005, Article No. 111. DOI: 10.1109/CVPR.2005.403.

[37] Wilson A, Benko H, Izadi S, Hilliges O. Steerable augmented reality with the beamatron. In Proc. the 25th Annual ACM Symposium on User Interface Software and Technology, October 2012, pp.413-422. DOI: 10.1145/2380116.2380169.

No related articles found!
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] Chen Shihua;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[2] Qu Yanwen;. AGDL: A Definition Language for Attribute Grammars[J]. , 1986, 1(3): 80 -91 .
[3] Chen Zhaoxiong; Gao Qingshi;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[4] Min Yinghua; Han Zhide;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[5] Huang Guoxiang; Liu Jian;. A Key-Lock Access Control[J]. , 1987, 2(3): 236 -243 .
[6] S. T. Chanson; L. Liang; A. Kumar. Throughput Models of CSMA Network with Stations Uniformly Distributed along the Bus[J]. , 1987, 2(4): 243 -264 .
[7] Bao Feng;. On the Condition for FSM Being a Scrambler[J]. , 1988, 3(1): 70 -74 .
[8] Chen Haibo; Ci Yungui;. Optimization for the Parallel Execution of Non-DO Loops under Leading Iteration Model[J]. , 1988, 3(4): 263 -272 .
[9] Feng Yin; Wang Kaizhu; Chang Yadong; Li Zhongrong;. CQAES,a Chinese Question Answer Experimental System[J]. , 1988, 3(4): 317 -319 .
[10] Zhang Bo; Zhang Tian; Zhang Jianwei; Zhang Ling;. Motion Planning for Robots with Topological Dimension Reduction Method[J]. , 1990, 5(1): 1 -16 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

 Home
Editorial Board
Author Guidelines
Subscription		 Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved