Journal of Computer Science and Technology ›› 2021, Vol. 36 ›› Issue (3): 478-493.doi: 10.1007/s11390-021-1331-y

Special Issue: Computer Graphics and Multimedia

• Special Section of CVM 2021 • Previous Articles     Next Articles

Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis

Lan Chen1,2, Member, CCF, Juntao Ye1, Member, CCF, and Xiaopeng Zhang1,3,*, Member, CCF, ACM, IEEE        

  1. 1 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China;
    2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China;
    3 Zhejiang Lab, Hangzhou 311121, China
  • Received:2021-01-29 Revised:2021-04-27 Online:2021-05-05 Published:2021-05-31
  • Contact: Xiaopeng Zhang E-mail:xiaopeng.zhang@ia.ac.cn
  • About author:Lan Chen received her Bachelor's degree in mathematics from China University of Petroleum, Beijing, in 2016. She is currently a Ph.D. candidate of Institute of Automation, Chinese Academy of Sciences, Beijing. Her research interests include computer graphics, geometry processing and image processing, particularly synthesis of cloth animation.
  • Supported by:
    This work is supported by the National Key Research and Development Program of China under Grant No. 2018YFB2100602, the National Natural Science Foundation of China under Grant Nos. 61972459, 61971418 and 62071157, and Open Research Projects of Zhejiang Lab under Grant No. 2021KE0AB07.

Existing physical cloth simulators suffer from expensive computation and difficulties in tuning mechanical parameters to get desired wrinkling behaviors. Data-driven methods provide an alternative solution. They typically synthesize cloth animation at a much lower computational cost, and also create wrinkling effects that are similar to the training data. In this paper we propose a deep learning based method for synthesizing cloth animation with high resolution meshes. To do this we first create a dataset for training:a pair of low and high resolution meshes are simulated and their motions are synchronized. As a result the two meshes exhibit similar large-scale deformation but different small wrinkles. Each simulated mesh pair is then converted into a pair of low- and high-resolution "images" (a 2D array of samples), with each image pixel being interpreted as any of three descriptors:the displacement, the normal and the velocity. With these image pairs, we design a multi-feature super-resolution (MFSR) network that jointly trains an upsampling synthesizer for the three descriptors. The MFSR architecture consists of shared and task-specific layers to learn multi-level features when super-resolving three descriptors simultaneously. Frame-to-frame consistency is well maintained thanks to the proposed kinematics-based loss function. Our method achieves realistic results at high frame rates:12-14 times faster than traditional physical simulation. We demonstrate the performance of our method with various experimental scenes, including a dressed character with sophisticated collisions.

Key words: cloth animation; deep learning; multi-feature; super-resolution; wrinkle synthesis;

[1] Liang J, Lin M C. Machine learning for digital try-on:Challenges and progress. Computational Visual Media, 2021, 7(2):159-167. DOI:10.1007/s41095-020-0189-1.
[2] Wang M, Lyu X Q, Li Y J, Zhang F L. VR content creation and exploration with deep learning:A survey. Computational Visual Media, 2020, 6(1):3-28. DOI:10.1007/s41095-020-0162-z.
[3] Terzopoulos D, Platt J, Barr A, Fleischer K. Elastically deformable models. In Proc. the 14th Annual Conference on Computer Graphics and Interactive Techniques, August 1987, pp.205-214. DOI:10.1145/37401.37427.
[4] Provot X. Collision and self-collision handling in cloth model dedicated to design garments. In Proc. the Eurographics Workshop on Computer Animation and Simulation, September 1997, pp.177-189. DOI:10.1007/978-3-7091-6874-5_13.
[5] Baraff D, Witkin A. Large steps in cloth simulation. In Proc. the 25th Annual Conference on Computer Graphics and Interactive Techniques, July 1998, pp.43-54. DOI:10.1145/280814.280821.
[6] Bridson R, Marino S, Fedkiw R. Simulation of clothing with folds and wrinkles. In Proc. the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, July 2003, pp.28-36. DOI:10.1145/1198555.1198573.
[7] Wang H, Hecht F, Ramamoorthi R, O'Brien J. Examplebased wrinkle synthesis for clothing animation. ACM Trans. Graph., 2010, 29(4):Article No. 107. DOI:10.1145/1778765.1778844.
[8] Zurdo J S, Brito J P, Otaduy M A. Animating wrinkles by example on non-skinned cloth. IEEE Trans. Visual. Comput. Graph., 2013, 19(1):149-158. DOI:10.1109/TVCG.2012.79.
[9] Santesteban I, Otaduy M A, Casas D. Learning-based animation of clothing for virtual try-on. Computer Graphics Forum, 2019, 38(2):355-366. DOI:10.1111/cgf.13643.
[10] Feng W W, Yu Y, Kim B U. A deformation transformer for real-time cloth animation. ACM Trans. Graph., 2010, 29(4):Article No. 108. DOI:10.1145/1778765.1778845.
[11] De Aguiar E, Sigal L, Treuille A, Hodgins J K. Stable spaces for real-time clothing. ACM Trans. Graph., 2010, 29(3):Article No. 106. DOI:10.1145/1833351.1778843.
[12] Kavan L, Gerszewski D, Bargteil A W, Sloan P P. Physics-inspired upsampling for cloth simulation in games. ACM Trans. Graph., 2011, 30(4):Article No. 93. DOI:10.1145/2010324.1964988.
[13] Chen L, Ye J, Jiang L, Ma C, Cheng Z, Zhang X. Synthesizing cloth wrinkles by CNN-based geometry image superresolution. Computer Animation and Virtual Worlds, 2018, 29(3/4):Article No. e1810. DOI:10.1002/cav.1810.
[14] Oh Y J, Lee T M, Lee I K. Hierarchical cloth simulation using deep neural networks. In Proc. the 2018 Computer Graphics International, June 2018, pp.139-146. DOI:10.1145/3208159.3208162.
[15] Lähner Z, Cremers D, Tung T. DeepWrinkles:Accurate and realistic clothing modeling. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.698-715. DOI:10.1007/978-3-030-01225-0_41.
[16] Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z. Photorealistic single image super-resolution using a generative adversarial network. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.105-114. DOI:10.1109/CVPR.2017.19.
[17] Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y. Residual dense network for image super-resolution. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.2472-2481. DOI:10.1109/CVPR.2018.00262.
[18] Gu X, Gortler S J, Hoppe H. Geometry images. ACM Trans. Graph., 2002, 21(3):355-361. DOI:10.1145/566654.566589.
[19] Narain R, Samii A, O'Brien J F. Adaptive anisotropic remeshing for cloth simulation. ACM Trans. Graph., 2012, 31(6):Article No. 152. DOI:10.1145/2366145.2366171.
[20] Liu T, Bargteil A W, O'Brien J F, Kavan L. Fast simulation of mass-spring systems. ACM Trans. Graph., 2013, 32(6):Article No. 124. DOI:10.1145/2508363.2508406.
[21] Guan P, Reiss L, Hirshberg D A, Weiss A, Black M J. DRAPE:Dressing any person. ACM Trans. Graph., 2012, 31(4):Article No. 35. DOI:10.1145/2185520.2185531.
[22] Kim D, Koh W, Narain R, Fatahalian K, Treuille A, O'Brien J F. Near-exhaustive precomputation of secondary cloth effects. ACM Trans. Graph., 2013, 32(4):Article No. 87. DOI:10.1145/2461912.2462020.
[23] Gundogdu E, Constantin V, Seifoddini A, Dang M, Salzmann M, Fua P. GarNet:A two-stream network for fast and accurate 3D cloth draping. In Proc. the 2019 IEEE/CVF International Conference on Computer Vision, Oct. 27-Nov 2, 2019, pp.8738-8747. DOI:10.1109/ICCV.2019.00883.
[24] Wang T Y, Ceylan D, Popovic J, Mitra N J. Learning a shared shape space for multimodal garment design. ACM Trans. Graph., 2018, 37(6):Article No. 203. DOI:10.1145/3272127.3275074.
[25] Wang T Y, Shao T, Fu K, Mitra N J. Learning an intrinsic garment space for interactive authoring of garment animation. ACM Transactions on Graphics, 2019, 38(6):Article No. 220. DOI:10.1145/3355089.3356512.
[26] Hahn F, Thomaszewski B, Coros S, Sumner R W, Cole F, Meyer M, DeRose T, Gross M. Subspace clothing simulation using adaptive bases. ACM Trans. Graph., 2014, 33(4):Article No. 105. DOI:10.1145/2601097.2601160.
[27] Xiao Y P, Lai Y K, Zhang F L, Li C P, Gao L. A survey on deep geometry learning:From a representation perspective. Computational Visual Media, 2020, 6(2):113-133. DOI:10.1007/s41095-020-0174-8.
[28] Yuan Y J, Lai Y K, Wu T, Gao L, Liu L. A revisit of shape editing techniques:From the geometric to the neural viewpoint. arXiv:2103.01694, 2021. https://arxiv.org/abs/2103.01694, Jan. 2021.
[29] Wang P S, Liu Y, Guo Y X, Sun C Y, Tong X. O-CNN:Octree-based convolutional neural networks for 3D shape analysis. ACM Transactions on Graphics, 2017, 36(4):Article No. 72. DOI:10.1145/3072959.3073608.
[30] Su H, Maji S, Kalogerakis E, Learned-Miller E G. Multiview convolutional neural networks for 3D shape recognition. In Proc. the 2015 IEEE International Conference on Computer Vision, Dec. 2015, pp.945-953. DOI:10.1109/ICCV.2015.114.
[31] Sinha A, Bai J, Ramani K. Deep learning 3D shape surfaces using geometry images. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.223-240. DOI:10.1007/978-3-319-46466-4_14.
[32] Tan Q, Gao L, Lai Y, Yang J, Xia S. Mesh-based autoencoders for localized deformation component analysis. In Proc. the 32nd Conference on Artificial Intelligence, Feb. 2018, pp.2452-2459.
[33] Tan Q, Gao L, Lai Y, Yang J, Xia S. Variational autoencoders for deforming 3D mesh models. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.5841-5850. DOI:10.1109/CVPR.2018.00612.
[34] Gao L, Lai Y K, Liang D, Chen S Y, Xia S. Efficient and flexible deformation representation for data-driven surface modeling. ACM Transactions on Graphics, 2016, 35(5):Article No. 158. DOI:10.1145/2908736.
[35] Gao L, Lai Y K, Yang J, Zhang L X, Xia S, Kobbelt L. Sparse data driven mesh deformation. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(3):2085-2100. DOI:10.1109/TVCG.2019.2941200.
[36] Zhang M, Wang T, Ceylan D, Mitra N J. Deep detail enhancement for any garment. arXiv:2008.04367, 2020. https://arxiv.org/abs/2008.04367v1, Jan. 2021.
[37] Dong C, Loy C C, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307. DOI:10.1109/TPAMI.2015.2439281.
[38] Liu S, Gang R, Li C, Song R. Adaptive deep residual network for single image super-resolution. Computational Visual Media, 2019, 5(4):391-401. DOI:10.1007/s41095-019-0158-8.
[39] Yue H J, Shen S, Yang J Y, Hu H F, Chen Y F. Reference image guided super-resolution via progressive channel attention networks. Journal of Computer Science and Technology, 2020, 35(3):551-563. DOI:10.1007/s11390-020-0270-3.
[40] Dong C, Loy C C, Tang X. Accelerating the super-resolution convolutional neural network. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.391-407. DOI:10.1007/978-3-319-46475-6_25.
[41] Shi W, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1874-1883. DOI:10.1109/CVPR.2016.207.
[42] Haris M, Shakhnarovich G, Ukita N. Deep back projection networks for super-resolution. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.1664-1673. DOI:10.1109/CVPR.2018.00179.
[43] Kappeler A, Yoo S, Dai Q, Katsaggelos A K. Video superresolution with convolutional neural networks. IEEE Transactions on Computational Imaging, 2016, 2(2):109-122. DOI:10.1109/TCI.2016.2532323.
[44] Chu M, Xie Y, Leal-Taixé L, Thuerey N. Learning temporal coherence via self-supervision for GAN-based video generation. ACM Trans. Graph., 2020, 39(4):Article No. 75. DOI:10.1145/3386569.3392457.
[45] Bhattacharjee P, Das S. Directional attention based video frame prediction using graph convolutional networks. In Proc. the 2019 International Joint Conference on Neural Networks, July 2019, pp.4268-4277. DOI:10.1109/IJCNN.2019.8852090.
[46] Xie Y, Franz E, Chu M, Thuerey N. TempoGAN:A temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Transactions on Graphics, 2018, 37(4):Article No. 95. DOI:10.1145/3197517.3201304.
[47] Kabsch W. A discussion of the solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A:Foundations and Advances, 1978, 34(5):827-828. DOI:10.1107/S0567739478001680.
[48] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249-256.
[49] Bergou M, Mathur S, Wardetzky M, Grinspun E. TRACKS:Toward directable thin shells. ACM Trans. Graph., 2007, 26(3):Article No. 50. DOI:10.1145/1276377.1276439.
[50] Müller M, Gross M. Interactive virtual materials. In Proc. the 2004 Graphics Interface Conference, May 2004, pp.239-246.
[51] Caruana R. Multitask learning. Machine Learning, 1997, 28(1):41-75. DOI:10.1023/A:1007379606734.
[52] Burden R, Faires J. Numerical Analysis (9th Edition). Cengage Learning, 2010.
[53] Ye J, Ma G, Jiang L, Chen L, Li J, Xiong G, Zhang X, Tang M. A unified cloth untangling framework through discrete collision detection. Computer Graphics Forum, 2017, 36(7):217-228. DOI:10.1111/cgf.13287.
[54] Wang H, O'Brien J F, Ramamoorthi R. Data-driven elastic models for cloth:Modeling and measurement. ACM Trans. Graph., 2011, 30(4):Article No. 71. DOI:10.1145/2010324.1964966.
[55] Kingma D P, Ba J. Adam:A method for stochastic optimization. In Proc. the 3rd International Conference on Learning Representations, May 2015.
[1] Xin Zhang, Siyuan Lu, Shui-Hua Wang, Xiang Yu, Su-Jing Wang, Lun Yao, Yi Pan, and Yu-Dong Zhang. Diagnosis of COVID-19 Pneumonia via a Novel Deep Learning Architecture [J]. Journal of Computer Science and Technology, 2022, 37(2): 330-343.
[2] Songjie Niu, Shimin Chen. TransGPerf: Exploiting Transfer Learning for Modeling Distributed Graph Computation Performance [J]. Journal of Computer Science and Technology, 2021, 36(4): 778-791.
[3] Sheng-Luan Hou, Xi-Kun Huang, Chao-Qun Fei, Shu-Han Zhang, Yang-Yang Li, Qi-Lin Sun, Chuan-Qing Wang. A Survey of Text Summarization Approaches Based on Deep Learning [J]. Journal of Computer Science and Technology, 2021, 36(3): 633-663.
[4] Yu-Jie Yuan, Yukun Lai, Tong Wu, Lin Gao, Li-Gang Liu. A Revisit of Shape Editing Techniques: From the Geometric to the Neural Viewpoint [J]. Journal of Computer Science and Technology, 2021, 36(3): 520-554.
[5] Wei Du, Yu Sun, Hui-Min Bao, Liang Chen, Ying Li, Yan-Chun Liang. DeepHBSP: A Deep Learning Framework for Predicting Human Blood-Secretory Proteins Using Transfer Learning [J]. Journal of Computer Science and Technology, 2021, 36(2): 234-247.
[6] Jun Gao, Paul Liu, Guang-Di Liu, Le Zhang. Robust Needle Localization and Enhancement Algorithm for Ultrasound by Deep Learning and Beam Steering Methods [J]. Journal of Computer Science and Technology, 2021, 36(2): 334-346.
[7] Hua Chen, Juan Liu, Qing-Man Wen, Zhi-Qun Zuo, Jia-Sheng Liu, Jing Feng, Bao-Chuan Pang, Di Xiao. CytoBrain: Cervical Cancer Screening System Based on Deep Learning Technology [J]. Journal of Computer Science and Technology, 2021, 36(2): 347-360.
[8] Nuo Qun, Hang Yan, Xi-Peng Qiu, Xuan-Jing Huang. Chinese Word Segmentation via BiLSTM+Semi-CRF with Relay Node [J]. Journal of Computer Science and Technology, 2020, 35(5): 1115-1126.
[9] Andrea Caroppo, Alessandro Leone, Pietro Siciliano. Comparison Between Deep Learning Models and Traditional Machine Learning Approaches for Facial Expression Recognition in Ageing Adults [J]. Journal of Computer Science and Technology, 2020, 35(5): 1127-1146.
[10] Chuang-Ye Zhang, Yan Niu, Tie-Ru Wu, Xi-Ming Li. Color Image Super-Resolution and Enhancement with Inter-Channel Details at Trivial Cost [J]. Journal of Computer Science and Technology, 2020, 35(4): 889-899.
[11] Dun Liang, Yuan-Chen Guo, Shao-Kui Zhang, Tai-Jiang Mu, Xiaolei Huang. Lane Detection: A Survey with New Results [J]. Journal of Computer Science and Technology, 2020, 35(3): 493-505.
[12] Zheng Zeng, Lu Wang, Bei-Bei Wang, Chun-Meng Kang, Yan-Ning Xu. Denoising Stochastic Progressive Photon Mapping Renderings Using a Multi-Residual Network [J]. Journal of Computer Science and Technology, 2020, 35(3): 506-521.
[13] Shuai Li, Zheng Fang, Wen-Feng Song, Ai-Min Hao, Hong Qin. Bidirectional Optimization Coupled Lightweight Networks for Efficient and Robust Multi-Person 2D Pose Estimation [J]. Journal of Computer Science and Technology, 2019, 34(3): 522-536.
[14] Na Ding, Ye-Peng Liu, Lin-Wei Fan, Cai-Ming Zhang. Single Image Super-Resolution via Dynamic Lightweight Database with Local-Feature Based Interpolation [J]. Journal of Computer Science and Technology, 2019, 34(3): 537-549.
[15] Jin-Hua Tao, Zi-Dong Du, Qi Guo, Hui-Ying Lan, Lei Zhang, Sheng-Yuan Zhou, Ling-Jie Xu, Cong Liu, Hai-Feng Liu, Shan Tang, Allen Rush, Willian Chen, Shao-Li Liu, Yun-Ji Chen, Tian-Shi Chen. BENCHIP: Benchmarking Intelligence Processors [J]. , 2018, 33(1): 1-23.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] Zhou Di;. A Recovery Technique for Distributed Communicating Process Systems[J]. , 1986, 1(2): 34 -43 .
[2] Liu Mingye; Hong Enyu;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[3] Zhang Cui; Zhao Qinping; Xu Jiafu;. Kernel Language KLND[J]. , 1986, 1(3): 65 -79 .
[4] Zheng Guoliang; Li Hui;. The Design and Implementation of the Syntax-Directed Editor Generator(SEG)[J]. , 1986, 1(4): 39 -48 .
[5] Min Yinghua; Han Zhide;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[6] Gong Zhenhe;. On Conceptual Model Specification and Verification[J]. , 1987, 2(1): 35 -50 .
[7] Min Yinghua;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[8] Huang Guoxiang; Liu Jian;. A Key-Lock Access Control[J]. , 1987, 2(3): 236 -243 .
[9] Lin Qi; Xia Peisu;. The Design and Implementation of a Very Fast Experimental Pipelining Computer[J]. , 1988, 3(1): 1 -6 .
[10] Sun Chengzheng; Tzu Yungui;. A New Method for Describing the AND-OR-Parallel Execution of Logic Programs[J]. , 1988, 3(2): 102 -112 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

Home
Editorial Board
Author Guidelines
Subscription
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved