Journal of Computer Science and Technology ›› 2019, Vol. 34 ›› Issue (2): 256-271.doi: 10.1007/s11390-019-1909-9

Special Issue: Artificial Intelligence and Pattern Recognition; Computer Graphics and Multimedia

• Special Section of Advances in Computer Science and Technology—Current Advances in the NSFC Joint Research Fund for Overseas Chinese Scholars and Scholars in Hong Kong and Macao 2014-2017 (Part 2) • Previous Articles     Next Articles

Real-Time Avatar Pose Transfer and Motion Generation Using Locally Encoded Laplacian Offsets

Masoud Zadghorban Lifkooee1, Celong Liu1, Yongqing Liang1, Yimin Zhu2, Xin Li1,*   

  1. 1 Department of Electrical and Computer Engineering, Louisiana State University, Baton Rouge 70803, U.S.A.;
    2 Department of Construction Management, Louisiana State University, Baton Rouge 70803, U.S.A.
  • Received:2018-07-15 Revised:2018-12-21 Online:2019-03-05 Published:2019-03-16
  • Contact: Xin Li
  • About author:Masoud Zadghorban Lifkooee received his M.Sc. degree in electrical engineering from University of Guilan, Rasht, in 2013 where he worked on image processing and pattern recognition projects such as sign language recognition. Then, he moved to Louisiana State University, Baton Rouge, to continue his research in 2015 where he worked on some machine learning and machine vision projects such as vehicle classification and facial expression recognition. Now he is currently working with Dr. Xin Li in Department of Electrical and Computer Engineering, Louisiana State University, Baton Rouge, on human body modeling for virtual reality environments and especially to be used in the cave automatic virtual environment.
  • Supported by:
    This work was partly supported by the National Science Foundation of USA under Grant No. IIS-1320959 and the National Natural Science Foundation of China under Grant No. 61728206.

We propose a human avatar representation scheme based on intrinsic coordinates, which are invariant to isometry and insensitive to human pose changes, and an efficient pose transfer algorithm that can utilize this representation to reconstruct a human body geometry following a given pose. Such a pose transfer algorithm can be used to control the movement of an avatar model in virtual reality environments following a user's motion in real time. Our proposed algorithm consists of three main steps. First, we recognize the user's pose and select a template model from the database who has a similar pose; then, the intrinsic Laplacian offsets encoded in local coordinates are used to reconstruct the human body geometry following the template pose; finally, the morphing between the two poses is generated using a linear interpolation. We perform experiments to evaluate the accuracy and efficiency of our algorithm. We believe our proposed system is a promising human modeling tool that can be used in general virtual reality applications.

Key words: human body pose transfer; local intrinsic coordinates; avatar control in virtual reality;

[1] Robitaille N, Jackson P L, Hébert L J, Mercier C et al. A virtual reality avatar interaction (VRAI) platform to assess residual executive dysfunction in active military personnel with previous mild traumatic brain injury: Proof of concept. Disability and Rehabilitation: Assistive Technology, 2017, 12(7): 758-764.
[2] Lifkooee M Z, Liu C L, Li M Q, Li X. Image-based human character modeling and reconstruction for virtual reality exposure therapy. In Proc. the 13th International Conference on Computer Science & Education, Aug. 2018, Article No. 149.
[3] Urella N, Hughes J, Conrad E, Zhang J S, Li X. A VR scene modelling platform for PTSD treatment. In Proc. the 12th International Conference on Computer Science and Education, Aug. 2017, pp.257-262.
[4] Pishchulin L, Wuhrer S, Helten T, Theobalt C, Schiele B. Building statistical shape spaces for 3D human modeling. Pattern Recognition, 2017, 67: 276-286.
[5] Orts-Escolano S, Rhemann C, Fanello S et al. Holoportation: Virtual 3D teleportation in real-time. In Proc. the 29th Annual Symposium on User Interface Software and Technology, Oct. 2016, pp.741-754.
[6] Sorkine O, Cohen-Or D, Lipman Y, Alexa M, Rössl C, Seidel H P. Laplacian surface editing. In Proc. the 2nd Eurographics Symposium on Geometry processing, Jul. 2004, pp.175-184.
[7] Shi L, Yu Y Z, Bell N, Feng W W. A fast multigrid algorithm for mesh deformation. ACM Trans. Graph., 2006, 25(3): 1108-1117.
[8] Huang J, Shi X H, Liu X U, Zhou K, Wei L Y, Teng S H, Bao H J, Guo B N, Shum H Y. Subspace gradient domain mesh deformation. In Proc. ACM SIGGRAPH 2006 Papers, Jul. 2006, pp.1126-1134.
[9] Le B H, Hodgins J K. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics, 2016, 35(4): Article No. 37.
[10] De Aguiar E, Theobalt C, Thrun S, Seidel H P. Automatic conversion of mesh animations into skeleton-based animations. Computer Graphics Forum, 2008, 27(2): 389-397.
[11] Kim M, Pons-Moll G, Pujades S, Bang S, Kim J, Black M J, Lee S H. Data-driven physics for human soft tissue animation. ACM Transactions on Graphics, 2017, 36(4): Article No. 54.
[12] Shi X H, Zhou K, Tong Y Y, Desbrun M, Bao H J, Guo B N. Mesh puppetry: Cascading optimization of mesh deformation with inverse kinematics. ACM Trans. Graph., 2007, 26(3): Article No. 81.
[13] Yasin H, Iqbal U, Kruger B, Weber A, Gall J. A dual-source approach for 3D pose estimation from a single image. In Proc. the 2006 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.4948-4956.
[14] Liu Z, Zhu J K, Bu J J, Chen C. A survey of human pose estimation: The body parts parsing based methods. Journal of Visual Communication and Image Representation, 2015, 32: 10-19.
[15] Martinez J, Hossain R, Romero J, Little J J. A simple yet effective baseline for 3D human pose estimation. In Proc. the 2007 IEEE International Conference on Computer Vision, Oct. 2017, pp.2659-2668.
[16] Moreno-Noguer F. 3D human pose estimation from a single image via distance matrix regression. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.1561-1570.
[17] Mehta D, Sridhar S, Sotnychenko O, Rhodin H, Shafiei M, Seidel H P, Xu W P, Casas D, Theobalt C. VNect: Realtime 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics, 2017, 36(4): Article No. 44.
[18] Remondino F. 3-D reconstruction of static human body shape from image sequence. Computer Vision and Image Understanding, 2004, 93(1): 65-85.
[19] Seo H, Yeo Y I, Wohn K. 3D body reconstruction from photos based on range scan. In Proc. the 1st International Conference on Technologies for E-Learning and Digital Entertainment, Apr. 2006, pp.849-860.
[20] Cheng K L, Tong R F, Tang M, Qian J Y, Sarkis M. Parametric human body reconstruction based on sparse key points. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2467-2479.
[21] Lifkooee M Z, Soysal Ö M, Sekeroglu K. Video min- ing for facial action unit classification using statistical spatial-temporal feature image and LoG deep convolutional neural network. Machine Vision and Applications, 2018.,Jan.2019.
[22] Tagliasacchi A, Delame T, Spagnuolo M, Amenta N, Telea A. 3D skeletons: A state-of-the-art report. Comput. Graph. Forum, 2016, 35(2): 573-597.
[23] Saha P K, Borgefors G, di Baja G S. A survey on skeletonization algorithms and their applications. Pattern Recognition Letters, 2016, 76(1): 3-12.
[24] Cornea N D, Silver D, Min P. Curve-skeleton properties, applications, and algorithms. IEEE Transactions on Visualization and Computer Graphics, 2007, 13(3): 530-548.
[25] Tagliasacchi A, Zhang H, Cohen-Or D. Curve skeleton extraction from incomplete point cloud. ACM Trans. Graph., 2009, 28(3): Article No. 71.
[26] Zheng Q, Sharf A, Tagliasacchi A, Chen B Q, Zhang H, Sheffer A, Cohen-Or D. Consensus skeleton for non-rigid space-time registration. Computer Graphics Forum, 2010, 29(2): 635-644.
[27] James D L, Twigg C D. Skinning mesh animations. ACM Transactions on Graphics, 2005, 24(3): 399-407.
[28] Colaianni M, Zöllhoefer M, Süßmuth J, Seider B, Greiner G. A pose invariant statistical shape model for human bodies. In Proc. the 5th International Conference on 3D Body Scanning Technologies, Oct. 2014, pp.327-336.
[29] Wuhrer S, Shu C, Xi P C. Posture-invariant statistical shape analysis using Laplace operator. Computers & Graphics, 2012, 36(5): 410-416.
[30] Davis T A, Hager W W. Dynamic supernodes in sparse cholesky update/downdate and triangular solves. ACM Trans. Math. Software, 2009, 35(4): Article No. 27.
[31] Bogo F, Romero J, Loper M, Black M J. FAUST: Dataset and evaluation for 3D mesh registration. In Proc. the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3794-3801.
[32] Xu Z, Zhang Q, Cheng S. Multilevel active registration for Kinect human body scans: From low quality to high quality. Multimedia Systems, 2018, 24(3): 257-270.
[33] Li X, Iyengar S. On computing mapping of 3D objects: A survey. ACM Computing Surveys, 2015, 47(2): Article No. 34.
[34] van Kaick O, Zhang H, Hamarneh G, Cohen-Or D. A survey on shape correspondence. Computer Graphics Forum, 2011, 30(6): 1681-1707.
[35] Loper M, Mahmood N, Romero J, Pons-Moll G, Black M J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics, 2015, 34(6): Article No. 248.
[36] Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black M J. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Proc. the 14th European Conference on Computer Vision, Oct. 2016, pp.561- 578.
[37] Alexa M. Recent advances in mesh morphing. Computer Graphics Forum, 2002, 21(2): 173-198.
[38] Liu Z G, Zhou L Y, Leung H, Multon F, Shum H P. High quality compatible triangulations for planar shape animation. In Proc. the 2017 ACM SIGGRAPH ASIA Workshop on Data-Driven Animation Techniques, Nov. 2017, pp.1-8.
No related articles found!
Full text



No Suggested Reading articles found!

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
  Copyright ©2015 JCST, All Rights Reserved