Journal of Computer Science and Technology ›› 2018, Vol. 33 ›› Issue (6): 1164-1177.doi: 10.1007/s11390-018-1879-3

Special Issue: Computer Graphics and Multimedia

• Computer Graphics and Multimedia • Previous Articles     Next Articles

A Geometry-Based Point Cloud Reduction Method for Mobile Augmented Reality System

Hao-Ren Wang, Juan Lei, Ao Li, Yi-Hong Wu*, Member, CCF   

  1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China University of Chinese Academy of Sciences, Beijing 100049, China
  • Received:2017-10-27 Revised:2018-09-10 Online:2018-11-15 Published:2018-11-15
  • Contact: Yi-Hong Wu,E-mail:yhwu@nlpr.ia.ac.cn E-mail:yhwu@nlpr.ia.ac.cn
  • About author:Hao-Ren Wang is now a Ph.D. candidate in the Robert Vision Group of National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, and University of Chinese Academy of Sciences, Beijing. He received his B.S. degree in automation from Beijing Institute of Technology, Beijing, in 2004. He received his M.S. degree in pattern recognition from Chinese Aviation University of China, Tianjin, in 2010. His research focuses on computer vision, camera localization, and vision geometry.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China under Grant Nos. 61836015, 61572499, and 61421004.

In this paper, a geometry-based point cloud reduction method is proposed, and a real-time mobile augmented reality system is explored for applications in urban environments. We formulate a new objective function which combines the point reconstruction errors and constraints on spatial point distribution. Based on this formulation, a mixed integer programming scheme is utilized to solve the points reduction problem. The mobile augmented reality system explored in this paper is composed of the offline and online stages. At the offline stage, we build up the localization database using structure from motion and compress the point cloud by the proposed point cloud reduction method. While at the online stage, we compute the camera pose in real time by combining an image-based localization algorithm and a continuous pose tracking algorithm. Experimental results on benchmark and real data show that compared with the existing methods, this geometry-based point cloud reduction method selects a point cloud subset which helps the image-based localization method to achieve higher success rate. Also, the experiments conducted on a mobile platform show that the reduced point cloud not only reduces the time consuming for initialization and re-initialization, but also makes the memory footprint small, resulting a scalable and real-time mobile augmented reality system.

Key words: mobile platform; augmented reality; point cloud reduction; structure from motion;

[1] Agarwal S, Snavely N, Simon I, Seitz S M, Szeliski R. Building Rome in a day. In Proc. the 12th IEEE International Conference on Computer Vision, September 2009, pp.72-79.
[2] Arth C, Wagner D, Klopschitz M, Irschara A, Schmalstieg D. Wide area localization on mobile phones. In Proc. the 8th IEEE International Symposium on Mixed and Augmented Reality, October 2009, pp.73-82.
[3] Arth C, Mulloni A, Schmalstieg D. Exploiting sensors on mobile phones to improve wide-area localization. In Proc. the 21st International Conference on Pattern Recognition, November 2012, pp.2152-2156.
[4] Middelberg S, Sattler T, Untzelmann O, Kobbelt L. Scalable 6-DOF localization on mobile devices. In Proc. the 13th European Conference on Computer Vision, September 2014, pp.268-283.
[5] Jeong Y, Bok Y, Kim J S, Kweon I S. Complementation of cameras and lasers for accurate 6D SLAM:From correspondences to bundle adjustment. In Proc. the 2011 IEEE International Conference on Robotics and Automation, May 2011, pp.3581-3588.
[6] Miraldo P, Araujo H. Pose estimation for non-central cameras using planes. Journal of Intelligent & Robotic Systems, 2015, 80(3/4):595-608.
[7] Miraldo P, Araujo H, Gonçalves N. Pose estimation for general cameras using lines. IEEE Transactions on Cybernetics, 2015, 45(10):2156-2164.
[8] Liu H, Zhang G, Bao H. Robust keyframe-based monocular SLAM for augmented reality. In Proc. the 2016 IEEE International Symposium on Mixed and Augmented Reality, September 2016.
[9] Sattler T, Leibe B, Kobbelt L. Fast image-based localization using direct 2D-to-3D matching. In Proc. the 2011 IEEE International Conference on Computer Vision, November 2011, pp.667-674.
[10] Sattler T, Weyand T, Leibe B, Kobbelt L. Image retrieval for image-based localization revisited. In Proc. the 2012 British Machine Vision Conference, Sept. 2012.
[11] Choudhary S, Indelman V, Christensen H I, Dellaert F. Information-based reduced landmark SLAM. In Proc. the 2015 IEEE International Conference on Robotics and Automation, May 2015, pp.4620-4627.
[12] Torii A, Arandjelović R, Sivic J, Okutomi M, Pajdla T. 24/7 place recognition by view synthesis. In Proc. the 2015 IEEE Conference on Computer Vision and Pattern Recognition, June 2015, pp.1808-1817.
[13] Zeisl B, Sattler T, Pollefeys M. Camera pose voting for large-scale image-based localization. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.2704-2712.
[14] Feng Y, Fan L, Wu Y. Fast localization in large-scale environments using supervised indexing of binary features. IEEE Transactions on Image Processing, 2016, 25(1):343-358.
[15] Campbell D, Petersson L, Kneip L, Li H. Globally-optimal inlier set maximisation for simultaneous camera pose and feature correspondence. In Proc. the 2017 IEEE International Conference on Computer Vision, October 2017.
[16] Yang J, Li H, Campbell D, Jia Y. Go-ICP:A globally optimal solution to 3D ICP point-set registration. IEEE Transactions on Pattern Analysis and Machine Intelligence, November 2016, 38(11):2241-2254.
[17] Li Y, Snavely N, Huttenlocher D P. Location recognition using prioritized feature matching. In Proc. the 11th European Conference on Computer Vision, September 2010, pp.791-804.
[18] Irschara A, Zach C, Frahm J M, Bischof H. From structurefrom-motion point clouds to fast location recognition. In Proc. the 2009 IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp.2599-2606.
[19] Dong Z, Zhang G, Jia J, Bao H. Efficient keyframe-based real-time camera tracking. Computer Vision and Image Understanding, 2014, 118:97-110.
[20] Cao S, Snavely N. Minimal scene descriptions from structure from motion models. In Proc. the 2014 IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp.461-468.
[21] Park H S, Wang Y, Nurvitadhi E, Hoe J C, Sheikh Y, Chen M. 3D point cloud reduction using mixed-integer quadratic programming. In Proc. the 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, June 2013, pp.229-236.
[22] Leutenegger S, Chli M, Siegwart R. BRISK:Binary robust invariant scalable keypoints. In Proc. the 2011 IEEE International Conference on Computer Vision, November 2011, pp.2548-2555.
[23] Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2):91-110.
[24] Rosten E, Drummond T. Machine learning for high-speed corner detection. In Proc. the 9th European Conference on Computer Vision, May 2006, pp.430-443.
[25] Gionis A, Indyk P, Motwani R. Similarity search in high dimensions via hashing. In Proc. the 25th International Conference on Very Large Data Bases, September 1999, pp.518-529.
[26] Chum O, Matas J. Matching with PROSAC-Progressive sample consensus. In Proc. the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Volume 1, June 2005, pp.220-226.
[27] Lepetit V, Moreno-Noguer F, Fua P. EPnP:An accurate O(n) solution to the PnP problem. International Journal of Computer Vision, 2009, 81(2):155-166.
[28] Takacs G, Chandrasekhar V, Tsai S, Chen D, Grzeszczuk R, Girod B. Unified real-time tracking and recognition with rotation-invariant fast features. In Proc. the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2010, pp.934-941.
[29] Stewart C V. Robust parameter estimation in computer vision. SIAM Review, 1999, 41(3):513-537.
[30] Hartley R, Zisserman A. Multiple View Geometry in Computer Vision (2nd edition). Cambridge University Press, 2004.
[31] Fischler M A, Bolles R C. Random sample consensus:A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6):381-395.
[1] Mo-Han Zhang, Jin-Hui Yu, Kang Zhang, Jun-Song Zhang. Artistic Augmentation of Photographs with Droplets [J]. Journal of Computer Science and Technology, 2019, 34(6): 1294-1306.
[2] Bin Liu, Kun Xu, Ralph R. Martin. Static Scene Illumination Estimation from Video with Applications [J]. , 2017, 32(3): 430-442.
[3] Xin Liu (刘鑫), Wei Gao (高伟), and Zhan-Yi Hu (胡占义). Hybrid Parallel Bundle Adjustment for 3D Scene Reconstruction with Massive Points [J]. , 2012, 27(6): 1269-1280.
[4] Yu Li, Yong-Tian Wang, and Yue Liu. Fiducial Marker Based on Projective Invariant for Augmented Reality [J]. , 2007, 22(6): 890-897 .
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] Feng Yulin;. Hierarchical Protocol Analysis by Temporal Logic[J]. , 1988, 3(1): 56 -69 .
[2] Zhang Fuyan; Cai Shijie; Wang Shu; Ge Ruding;. The Human-Computer Dialogue Management of FCAD System[J]. , 1988, 3(3): 221 -227 .
[3] Zheng Chongxun; Zhang Kenong;. Orthogonal Algorithm of Logic Probability and Syndrome-Testable Analysis[J]. , 1990, 5(2): 203 -209 .
[4] Xu Meirui; Liu Xiaolin;. A VLSI Algorithm for Calculating the Tree to Tree Distance[J]. , 1993, 8(1): 68 -76 .
[5] Xiang Dong; Wei Daozheng;. GLOBAL: A Design for Random Testability Algorithm[J]. , 1994, 9(2): 182 -192 .
[6] Tan Jianrong; Zheng Jianmin; Peng Qunsheng;. A Unified Algorithm for Finding the Intersection Curve of Surfaces[J]. , 1994, 9(2): 107 -116 .
[7] Liao Xianzhi; Jin Lan;. Rendezvous Facilities in a Distributed Computer System[J]. , 1995, 10(2): 188 -192 .
[8] Gao Qingshi; Liu Zhiyong;. K-Dimensional Optimal Parallel Algorithm for the Solution of a General Class of Recurrence Equations[J]. , 1995, 10(5): 417 -424 .
[9] Tao Xuehong; Sun Wei; Ma Shaohan;. A Practical Propositional Knowledge Base Revision Algorithm[J]. , 1997, 12(2): 154 -159 .
[10] Zhang Yongyue; Peng Zhenyun; You Suya; Xu Guangyou;. A Multi-View Face Recognition System[J]. , 1997, 12(5): 400 -407 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

Home
Editorial Board
Author Guidelines
Subscription
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved