计算机科学技术学报 ›› 2022,Vol. 37 ›› Issue (3): 666-679.doi: 10.1007/s11390-022-2173-y

所属专题： Artificial Intelligence and Pattern Recognition； Computer Graphics and Multimedia

• • 上一篇    下一篇

ARSlice：动态追踪与投影增强的头戴显示器

  

  • 收稿日期:2022-01-20 修回日期:2022-04-08 接受日期:2022-04-24 出版日期:2022-05-30 发布日期:2022-05-30

ARSlice: Head-Mounted Display Augmented with Dynamic Tracking and Projection

Yu-Ping Wang1 (王瑀屏), Member, CCF, IEEE, Sen-Wei Xie1 (解森炜), Li-Hui Wang2,3,* (王立辉), Member, IEEE, Hongjin Xu4 (徐鸿金), Satoshi Tabata3, Member, ACM, and Masatoshi Ishikawa3,5        

  1. 1Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
    2Institute of Semiconductors, Guangdong Academy of Sciences, Guangzhou 510651, China
    3Data Science Research Division, Information Technology Center, The University of Tokyo, Tokyo 113-8656, Japan
    4Core Network Department, Advanced Operations Development Division, KDDI Cooperation, Tokyo 206-0034, Japan
    5Tokyo University of Science, Tokyo 113-8601, Japan
  • Received:2022-01-20 Revised:2022-04-08 Accepted:2022-04-24 Online:2022-05-30 Published:2022-05-30
  • Contact: Li-Hui Wang E-mail:wanglihui@gdisit.com
  • About author:Li-Hui Wang received his B.E. degree from Shenyang University of Technology, Shenyang, in 2007, and his M.E. degree in engineering from Northeastern University, Shenyang, in 2009, and his Ph.D. degree in information science and technology from the University of Tokyo, Tokyo, in 2014. He was a project assistant professor and project researcher at the University of Tokyo, Tokyo, till 2019. He is currently a processor in engineering at the Institute of Semiconductors, Guangdong Academy of Sciences, Guangzhou. His current research interests include adaptive optics system, high speed vision, and dynamic interaction.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China under Grant No. 61872210, the Guangdong Basic and Applied Basic Research Foundation under Grant Nos. 2021A1515012596 and 2021B1515120064, and the Guangdong Academy of Sciences Special Foundation under Grant No. 2021GDASYL-20210102006.

PDF (PC)

39

补充材料

1

1、研究背景：CT图像是利用X射线对人体某一平面进行扫描得到的图像。在一组相互平行的平面扫描可得到一组CT图像，其表达的三维信息可以帮助人们了解人体内部状态。但要理解一组CT图像，要求医生受到专业的训练，未受专业训练的人（如，患者及其家属）很难理解其所表达的三维信息。三维重建技术能够帮助更好的分析和存储三维信息，但并不能帮助人们理解CT图像，因为我们往往关心的不是从外部看到人体三维模型，而是其包含的人体内部信息。虚拟现实/增强现实（VR/AR）技术提供了具有潜力的解决方案，但现有的头戴式显示器受到显示信号线或无线传输延迟的限制，影响用户体验。
2、目的：我们的研究工作旨在设计一种交互式可视化方案，帮助人们理解CT图像。同时，不受传统VR/AR的头戴显示器的显示信号线限制，同时具有用户端重量轻、无能耗、低延迟等优势。
3、方法：我们的解决方案ARSlice，将空间增强现实（SAR）技术与头戴显示器相结合，分为投影端和用户端两部分。投影端使用动态追踪技术，定位用户头部位置和姿态，根据姿态生成投影图像，并根据位置高速调整焦距，使得投影图像能够清晰地投影到用户端的投影板上；用户端是纯光学设备，负责让用户在三维空间中看到投影板上的图像。用户端通过光学投影获取信息，无需显示信号线或无线传输；也无需关心数据生成问题，因此用户端不产生能耗；另外，通过高效的动态追踪算法，使得系统整体延迟符合人眼要求。
4、结果：我们设计制作了设备原型，用户可以在三维空间中看到CT图像切面，图像内容随着用户头部运动而变化，如同用户眼前有一个平面对人体进行切割显示。设备提供了近似的6自由度，用户端可以在距离投影端0.5-2米范围内移动；用户端可以在上下左右各方向约45°的幅度内转动；用户端的视角范围约75°；整体刷新率在50Hz以上。通过用户体验调研，大多数用户认为ARSlice系统可以帮助理解CT图像。
5、结论（Conclusions）：我们设计了一种帮助理解CT图像的交互式系统方案ARSlice。能够在用户端无显示信号线、无能耗的条件下实现低延迟的虚拟现实/增强现实体验，可有效帮助用户理解CT图像。 论文对系统的一些技术指标的改进方案及可能的其他应用场景进行了讨论。后续工作将进一步提供给专业医生使用。

关键词: 虚拟现实/增强现实, CT图像可视化, 交互式可视化

Abstract: Computed tomography (CT) generates cross-sectional images of the body. Visualizing CT images has been a challenging problem. The emergence of the augmented and virtual reality technology has provided promising solutions. However, existing solutions suffer from tethered display or wireless transmission latency. In this paper, we present ARSlice, a proof-of-concept prototype that can visualize CT images in an untethered manner without wireless transmission latency. Our ARSlice prototype consists of two parts, the user end and the projector end. By employing dynamic tracking and projection, the projector end can track the user-end equipment and project CT images onto it in real time. The user-end equipment is responsible for displaying these CT images into the 3D space. Its main feature is that the user-end equipment is a pure optical device with light weight, low cost, and no energy consumption. Our experiments demonstrate that our ARSlice prototype provides part of six degrees of freedom for the user, and a high frame rate. By interactively visualizing CT images into the 3D space, our ARSlice prototype can help untrained users better understand that CT images are slices of a body.

Key words: augmented and virtual reality, computed tomography (CT) image visualization, interactive visualization

[1] Raskar R, Welch G, Low K, Bandyopadhyay D. Shader lamps: Animating real objects with image-based illumination. In Proc. the 12th Eurographics Workshop on Rendering Techniques, June 2001, pp.89-102. DOI: 10.2312/EGWR/EGWR01/089-101.

[2] Benko H, Ofek E, Zheng F, Wilson A D. FoveAR: Combining an optically see-through near-eye display with projector-based spatial augmented reality. In Proc. the 28th Annual ACM Symposium on User Interface Software &] Technology, November 2015, pp.129-135. DOI: 10.1145/2807442.2807493.

[3] Lincoln P, Welch G, Nashel A, State A, Ilie A, Fuchs H. Animatronic shader lamps avatars. Virtual Reality, 2011, 15(2/3): 225-238. DOI: 10.1007/s10055-010-0175-5.

[4] Lincoln P,Welch G, Nashel A, Ilie A, State A, Fuchs H. Animatronic shader lamps avatars. In Proc. the 8th IEEE International Symposium on Mixed and Augmented Reality, October 2009, pp.27-33. DOI: 10.1109/ISMAR.2009.5336503.

[5] Narita G, Watanabe Y, Ishikawa M. Dynamic projection mapping onto deforming non-rigid surface using deformable dot cluster marker. IEEE Trans. Vis. Comput. Graph., 2017, 23(3): 1235-1248. DOI: 10.1109/TVCG.2016.2592910.

[6] Wang L, Xu H, Tabata S, Hu Y, Watanabe Y, Ishikawa M. High-speed focal tracking projection based on liquid lens. In Proc. the 2020 ACM SIGGRAPH, August 2020, Article No. 15. DOI: 10.1145/3388534.3408333.

[7] Garcı́a-Berná J A, Sanchez-Gomez J M, Hermanns J, Garcı́a-Mateos G, Alemán J L F. Calcification detection of abdominal aorta in CT images and 3D visualization in VR devices. In Proc. the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, August 2016, pp.4157-4160. DOI: 10.1109/EMBC.2016.7591642.

[8] Azuma R. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 1997, 6(4): 355-385. DOI: 10.1162/pres.1997.6.4.355.

[9] Ibrahim A, Huynh B, Downey J, Höllerer T, Chun D, O’Donovan J. ARbis pictus: A study of vocabulary learning with augmented reality. IEEE Trans. Vis. Comput. Graph., 2018, 24(11): 2867-2874. DOI: 10.1109/TVCG.2018.2868568.

[10] Buttussi F, Chittaro L. Effects of different types of virtual reality display on presence and learning in a safety training scenario. IEEE Trans. Vis. Comput. Graph., 2018, 24(2): 1063-1076. DOI: 10.1109/TVCG.2017.2653117.

[11] Bai Z, Blackwell A F, Coulouris G. Using augmented reality to elicit pretend play for children with autism. IEEE Trans. Vis. Comput. Graph., 2015, 21(5): 598-610. DOI: 10.1109/TVCG.2014.2385092.

[12] Menk C, Koch R. Truthful color reproduction in spatial augmented reality applications. IEEE Trans. Vis. Comput. Graph., 2013, 19(2): 236-248. DOI: 10.1109/TVCG.2012.146.

[13] Liu X, Vlachou C, Qian F, Wang C, Kim K. Firefly: Untethered multi-user VR for commodity mobile devices. In Proc. the 2020 USENIX Annual Technical Conference, July 2020, pp.943-957.

[14] Stauffert J, Niebling F, Latoschik M E. Simultaneous run-time measurement of motion-to-photon latency and latency jitter. In Proc. the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, March 2020, pp.636-644. DOI: 10.1109/VR46266.2020.00086.

[15] Ehnes J, Hirota K, Hirose M. Projected augmentation—Augmented reality using rotatable video projectors. In Proc. the 3rd IEEE and ACM International Symposium on Mixed and Augmented Reality, November 2004, pp.26-35. DOI: 10.1109/ISMAR.2004.47.

[16] Schwerdtfeger B, Pustka D, Hofhauser A, Klinker G. Using laser projectors for augmented reality. In Proc. the 2008 ACM Symposium on Virtual Reality Software and Technology, October 2008, pp.134-137. DOI: 10.1145/1450579.1450608.

[17] Jones B R, Sodhi R, Campbell R H, Garnett G, Bailey B P. Build your world and play in it: Interacting with surface particles on complex objects. In Proc. the 9th IEEE International Symposium on Mixed and Augmented Reality, October 2010, pp.165-174. DOI: 10.1109/ISMAR.2010.5643566.

[18] Hua H, Gao C, Brown L D, Ahuja N, Rolland J P. Using a head-mounted projective display in interactive augmented environments. In Proc. the 4th International Symposium on Augmented Reality, October 2001, pp.217-223. DOI: 10.1109/ISAR.2001.970540.

[19] Hamasaki T, Itoh Y, Hiroi Y, Iwai D, Sugimoto M. HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection. IEEE Trans. Vis. Comput. Graph., 2018, 24(4): 1457-1466. DOI: 10.1109/TVCG.2018.2793659.

[20] Hiroi Y, Itoh Y, Hamasaki T, Iwai D, Sugimoto M. HySAR: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection. In Proc. the 2017 IEEE Virtual Reality, March 2017, pp.211-212. DOI: 10.1109/VR.2017.7892251.

[21] Kim K, Billinghurst M, Bruder G, Duh H B, Welch G F. Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008–2017). IEEE Trans. Vis. Comput. Graph., 2018, 24(11): 2947-2962. DOI: 10.1109/TVCG.2018.2868591.

[22] Zhou F, Duh H B, Billinghurst M. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proc. the 7th IEEE and ACM International Symposium on Mixed and Augmented Reality, September 2008, pp.193-202. DOI: 10.1109/ISMAR.2008.4637362.

[23] Liu S, Cheng D, Hua H. An optical see-through head mounted display with addressable focal planes. In Proc. the 7th IEEE and ACM International Symposium on Mixed and Augmented Reality, September 2008, pp.33-42. DOI: 10.1109/ISMAR.2008.4637321.

[24] Chakravarthula P, Dunn D, Aksit K, Fuchs H. FocusAR: Autofocus augmented reality eyeglasses for both real world and virtual imagery. IEEE Trans. Vis. Comput. Graph., 2018, 24(11): 2906-2916. DOI: 10.1109/TVCG.2018.2868532.

[25] Maimone A, Georgiou A, Kollin J S. Holographic near-eye displays for virtual and augmented reality. ACM Trans. Graph., 2017, 36(4): Article No. 85. DOI: 10.1145/3072959.3073624.

[26] Jang C, Bang K, Moon S, Kim J, Lee S, Lee B. Retinal 3D: Augmented reality near-eye display via pupil-tracked light field projection on retina. ACM Trans. Graph., 2017, 36(6): Article No. 190. DOI: 10.1145/3130800.3130889.

[27] Kikinis R, Shenton M E, Iosifescu D V, McCarley R W, Saiviroonporn P, Hokama H H, Robatino A, Metcalf D, Wible C, Portas C M, Donnino R M, Jolesz F A. A digital brain atlas for surgical planning, model-driven segmentation, and teaching. IEEE Trans. Vis. Comput. Graph., 1996, 2(3): 232-241. DOI: 10.1109/2945.537306.

[28] Wu J, Belle A, Hargraves R H, Cockrell C, Tang Y, Najarian K. Bone segmentation and 3D visualization of CT images for traumatic pelvic injuries. Int. J. Imaging Syst. Technol., 2014, 24(1): 29-38. DOI: 10.1002/ima.22076.

[29] Hu Z, Zou J, Gui J, Rong J, Li Y, Xi D, Zheng H. Real-time visualization and interaction of three-dimensional human CT images. J. Comput., 2010, 5(9): 1335-1342. DOI: 10.4304/jcp.5.9.1335-1342.

[30] Jung Y, Kim J, Feng D D. Dual-modal visibility metrics for interactive PET-CT visualization. In Proc. the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, August 28-September 1, 2012, pp.2696-2699. DOI: 10.1109/EMBC.2012.6346520.

[31] Javan R, Rao A, Jeun B S, Herur-Raman A, Singh N, Heidari P. From CT to 3D printed models, serious gaming, and virtual reality: Framework for educational 3D visualization of complex anatomical spaces from within—The pterygopalatine fossa. J. Digit. Imaging, 2020, 33(3): 776-791. DOI: 10.1007/s10278-019-00315-y.

[32] Xing J, Ai H, Lao S. Multi-object tracking through occlusions by local tracklets filtering and global tracklets association with detection responses. In Proc. the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2009, pp.1200-1207. DOI: 10.1109/CVPR.2009.5206745.

[33] Pirsiavash H, Ramanan D, Fowlkes C C. Globally-optimal greedy algorithms for tracking a variable number of objects. In Proc. the 24th IEEE Conference on Computer Vision and Pattern Recognition, June 2011, pp.1201-1208. DOI: 10.1109/CVPR.2011.5995604.

[34] Zhang Z. Flexible camera calibration by viewing a plane from unknown orientations. In Proc. the 7th IEEE International Conference on Computer Vision, September 1999, pp.666-673. DOI: 10.1109/ICCV.1999.791289.

[35] Kato H, Billinghurst M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In Proc. the 2nd IEEE and ACM International Workshop on Augmented Reality, October 1999, pp.85-94. DOI: 10.1109/IWAR.1999.803809.

[36] Gupta S, Jaynes C O. Active pursuit tracking in a projector-camera system with application to augmented reality. In Proc. the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, September 2005, Article No. 111. DOI: 10.1109/CVPR.2005.403.

[37] Wilson A, Benko H, Izadi S, Hilliges O. Steerable augmented reality with the beamatron. In Proc. the 25th Annual ACM Symposium on User Interface Software and Technology, October 2012, pp.413-422. DOI: 10.1145/2380116.2380169.

No related articles found!
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] 陈世华;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[2] 屈延文;. AGDL: A Definition Language for Attribute Grammars[J]. , 1986, 1(3): 80 -91 .
[3] 陈肇雄; 高庆狮;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[4] 闵应骅; 韩智德;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[5] 黄国祥; 刘健;. A Key-Lock Access Control[J]. , 1987, 2(3): 236 -243 .
[6] S.T.Chanson; 梁路平; A.Kumar;. Throughput Models of CSMA Network with Stations Uniformly Distributed along the Bus[J]. , 1987, 2(4): 243 -264 .
[7] 包丰;. On the Condition for FSM Being a Scrambler[J]. , 1988, 3(1): 70 -74 .
[8] 陈海波; 慈云桂;. Optimization for the Parallel Execution of Non-DO Loops under Leading Iteration Model[J]. , 1988, 3(4): 263 -272 .
[9] 冯寅; 王开铸; 常雅冬; 李仲荣;. CQAES,a Chinese Question Answer Experimental System[J]. , 1988, 3(4): 317 -319 .
[10] 张钹; 张恬; 张建伟; 张铃;. Motion Planning for Robots with Topological Dimension Reduction Method[J]. , 1990, 5(1): 1 -16 .
版权所有 © 《计算机科学技术学报》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持：support@magtech.com.cn
总访问量：