计算机科学技术学报 ›› 2021,Vol. 36 ›› Issue (3): 478-493.doi: 10.1007/s11390-021-1331-y

所属专题: Computer Graphics and Multimedia

• • 上一篇    下一篇

基于多特征超分网络的布料褶皱合成

Lan Chen1,2, Member, CCF, Juntao Ye1, Member, CCF, and Xiaopeng Zhang1,3,*, Member, CCF, ACM, IEEE   

  1. 1 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China;
    2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China;
    3 Zhejiang Lab, Hangzhou 311121, China
  • 收稿日期:2021-01-29 修回日期:2021-04-27 出版日期:2021-05-05 发布日期:2021-05-31
  • 通讯作者: Xiaopeng Zhang E-mail:xiaopeng.zhang@ia.ac.cn
  • 作者简介:Lan Chen received her Bachelor's degree in mathematics from China University of Petroleum, Beijing, in 2016. She is currently a Ph.D. candidate of Institute of Automation, Chinese Academy of Sciences, Beijing. Her research interests include computer graphics, geometry processing and image processing, particularly synthesis of cloth animation.
  • 基金资助:
    This work is supported by the National Key Research and Development Program of China under Grant No. 2018YFB2100602, the National Natural Science Foundation of China under Grant Nos. 61972459, 61971418 and 62071157, and Open Research Projects of Zhejiang Lab under Grant No. 2021KE0AB07.

Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis

Lan Chen1,2, Member, CCF, Juntao Ye1, Member, CCF, and Xiaopeng Zhang1,3,*, Member, CCF, ACM, IEEE        

  1. 1 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China;
    2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China;
    3 Zhejiang Lab, Hangzhou 311121, China
  • Received:2021-01-29 Revised:2021-04-27 Online:2021-05-05 Published:2021-05-31
  • Contact: Xiaopeng Zhang E-mail:xiaopeng.zhang@ia.ac.cn
  • About author:Lan Chen received her Bachelor's degree in mathematics from China University of Petroleum, Beijing, in 2016. She is currently a Ph.D. candidate of Institute of Automation, Chinese Academy of Sciences, Beijing. Her research interests include computer graphics, geometry processing and image processing, particularly synthesis of cloth animation.
  • Supported by:
    This work is supported by the National Key Research and Development Program of China under Grant No. 2018YFB2100602, the National Natural Science Foundation of China under Grant Nos. 61972459, 61971418 and 62071157, and Open Research Projects of Zhejiang Lab under Grant No. 2021KE0AB07.

1、研究背景(context):
布料仿真在众多应用中起着重要的作用,如电影,视频游戏,虚拟试衣。物理仿真技术的飞速发展,使具有逼真细节的服装动画成为现实。但这些方法需要很高分辨率的网格来表示精细的细节,因此需要大量的计算来求解方程、解决碰撞问题。此外还需要大量的人力来调整仿真参数从而获得想要的褶皱形态。近期数据驱动的方法提供了快速且逼真的褶皱生成效果。依靠预先计算的数据和数据驱动技术,可以直接合成高分辨率网格,也可以通过物理模拟的低分辨率网格对其进行褶皱叠加。尽管如此,现有的数据驱动方法或依赖于人体姿势不适用于宽松服装,或缺乏对自由度较大的通用布料进行动态的褶皱建模。
2、目的(Objective):该研究的目的是快速合成高质量且时序稳定的布料仿真动画,不仅能针对紧身衣,还能适应于更加宽松的服装或者通用布料,解决目前研究方法遇到的挑战。
3、方法(Method):本文提出了多特征超分网络来进行布料的褶皱合成。通过虚拟弹簧约束和多分辨率动态模型同步模拟一对高低分辨率网格,使其在细节上有差异。然后将模拟的网格转换为双分辨率几何图像,编码位移、法相和速度特征。基于这些特征,本文设计了空间和时间损失函数来训练多特征超分网络,从而获得时序一致的几何图像。再将几何图像转换成三维网格即可快速获得稳定且富含细节的布料动画。
4、结果(Result & Findings):
本文构建了三个高低分辨率数据集并在数据集上进行了定量和定性对比,取得了最优的数值指标和最佳的视觉效果。通过对比和消融实验证明了本文提出的多特征及损失函数的价值。本文提出的方法以高帧速率得到逼真的布料动画:比传统的物理模拟快12~14倍。
5、结论(Conclusions):
本文提出了多特征框架合成布料动画细节。将三维网格的时空特征嵌入到多张图像中,共同学习超分辨网络,设计了基于运动学的损失,保证时序一致性。在数据构造时设计了同步仿真方法,不仅可以生成配对的三维衣服网格用于细节增强,也可以用于其他应用,如不同形状结构的三维形状匹配。就非配对数据的细节增强以及将该方法泛化到不同材质的服装上是未来进一步研究的方向。

关键词: 布料仿真, 深度学习, 褶皱合成, 多特征, 超分辨率

Abstract: Existing physical cloth simulators suffer from expensive computation and difficulties in tuning mechanical parameters to get desired wrinkling behaviors. Data-driven methods provide an alternative solution. They typically synthesize cloth animation at a much lower computational cost, and also create wrinkling effects that are similar to the training data. In this paper we propose a deep learning based method for synthesizing cloth animation with high resolution meshes. To do this we first create a dataset for training:a pair of low and high resolution meshes are simulated and their motions are synchronized. As a result the two meshes exhibit similar large-scale deformation but different small wrinkles. Each simulated mesh pair is then converted into a pair of low- and high-resolution "images" (a 2D array of samples), with each image pixel being interpreted as any of three descriptors:the displacement, the normal and the velocity. With these image pairs, we design a multi-feature super-resolution (MFSR) network that jointly trains an upsampling synthesizer for the three descriptors. The MFSR architecture consists of shared and task-specific layers to learn multi-level features when super-resolving three descriptors simultaneously. Frame-to-frame consistency is well maintained thanks to the proposed kinematics-based loss function. Our method achieves realistic results at high frame rates:12-14 times faster than traditional physical simulation. We demonstrate the performance of our method with various experimental scenes, including a dressed character with sophisticated collisions.

Key words: cloth animation, deep learning, multi-feature, super-resolution, wrinkle synthesis

[1] Liang J, Lin M C. Machine learning for digital try-on:Challenges and progress. Computational Visual Media, 2021, 7(2):159-167. DOI:10.1007/s41095-020-0189-1.
[2] Wang M, Lyu X Q, Li Y J, Zhang F L. VR content creation and exploration with deep learning:A survey. Computational Visual Media, 2020, 6(1):3-28. DOI:10.1007/s41095-020-0162-z.
[3] Terzopoulos D, Platt J, Barr A, Fleischer K. Elastically deformable models. In Proc. the 14th Annual Conference on Computer Graphics and Interactive Techniques, August 1987, pp.205-214. DOI:10.1145/37401.37427.
[4] Provot X. Collision and self-collision handling in cloth model dedicated to design garments. In Proc. the Eurographics Workshop on Computer Animation and Simulation, September 1997, pp.177-189. DOI:10.1007/978-3-7091-6874-5_13.
[5] Baraff D, Witkin A. Large steps in cloth simulation. In Proc. the 25th Annual Conference on Computer Graphics and Interactive Techniques, July 1998, pp.43-54. DOI:10.1145/280814.280821.
[6] Bridson R, Marino S, Fedkiw R. Simulation of clothing with folds and wrinkles. In Proc. the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, July 2003, pp.28-36. DOI:10.1145/1198555.1198573.
[7] Wang H, Hecht F, Ramamoorthi R, O'Brien J. Examplebased wrinkle synthesis for clothing animation. ACM Trans. Graph., 2010, 29(4):Article No. 107. DOI:10.1145/1778765.1778844.
[8] Zurdo J S, Brito J P, Otaduy M A. Animating wrinkles by example on non-skinned cloth. IEEE Trans. Visual. Comput. Graph., 2013, 19(1):149-158. DOI:10.1109/TVCG.2012.79.
[9] Santesteban I, Otaduy M A, Casas D. Learning-based animation of clothing for virtual try-on. Computer Graphics Forum, 2019, 38(2):355-366. DOI:10.1111/cgf.13643.
[10] Feng W W, Yu Y, Kim B U. A deformation transformer for real-time cloth animation. ACM Trans. Graph., 2010, 29(4):Article No. 108. DOI:10.1145/1778765.1778845.
[11] De Aguiar E, Sigal L, Treuille A, Hodgins J K. Stable spaces for real-time clothing. ACM Trans. Graph., 2010, 29(3):Article No. 106. DOI:10.1145/1833351.1778843.
[12] Kavan L, Gerszewski D, Bargteil A W, Sloan P P. Physics-inspired upsampling for cloth simulation in games. ACM Trans. Graph., 2011, 30(4):Article No. 93. DOI:10.1145/2010324.1964988.
[13] Chen L, Ye J, Jiang L, Ma C, Cheng Z, Zhang X. Synthesizing cloth wrinkles by CNN-based geometry image superresolution. Computer Animation and Virtual Worlds, 2018, 29(3/4):Article No. e1810. DOI:10.1002/cav.1810.
[14] Oh Y J, Lee T M, Lee I K. Hierarchical cloth simulation using deep neural networks. In Proc. the 2018 Computer Graphics International, June 2018, pp.139-146. DOI:10.1145/3208159.3208162.
[15] Lähner Z, Cremers D, Tung T. DeepWrinkles:Accurate and realistic clothing modeling. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.698-715. DOI:10.1007/978-3-030-01225-0_41.
[16] Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z. Photorealistic single image super-resolution using a generative adversarial network. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.105-114. DOI:10.1109/CVPR.2017.19.
[17] Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y. Residual dense network for image super-resolution. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.2472-2481. DOI:10.1109/CVPR.2018.00262.
[18] Gu X, Gortler S J, Hoppe H. Geometry images. ACM Trans. Graph., 2002, 21(3):355-361. DOI:10.1145/566654.566589.
[19] Narain R, Samii A, O'Brien J F. Adaptive anisotropic remeshing for cloth simulation. ACM Trans. Graph., 2012, 31(6):Article No. 152. DOI:10.1145/2366145.2366171.
[20] Liu T, Bargteil A W, O'Brien J F, Kavan L. Fast simulation of mass-spring systems. ACM Trans. Graph., 2013, 32(6):Article No. 124. DOI:10.1145/2508363.2508406.
[21] Guan P, Reiss L, Hirshberg D A, Weiss A, Black M J. DRAPE:Dressing any person. ACM Trans. Graph., 2012, 31(4):Article No. 35. DOI:10.1145/2185520.2185531.
[22] Kim D, Koh W, Narain R, Fatahalian K, Treuille A, O'Brien J F. Near-exhaustive precomputation of secondary cloth effects. ACM Trans. Graph., 2013, 32(4):Article No. 87. DOI:10.1145/2461912.2462020.
[23] Gundogdu E, Constantin V, Seifoddini A, Dang M, Salzmann M, Fua P. GarNet:A two-stream network for fast and accurate 3D cloth draping. In Proc. the 2019 IEEE/CVF International Conference on Computer Vision, Oct. 27-Nov 2, 2019, pp.8738-8747. DOI:10.1109/ICCV.2019.00883.
[24] Wang T Y, Ceylan D, Popovic J, Mitra N J. Learning a shared shape space for multimodal garment design. ACM Trans. Graph., 2018, 37(6):Article No. 203. DOI:10.1145/3272127.3275074.
[25] Wang T Y, Shao T, Fu K, Mitra N J. Learning an intrinsic garment space for interactive authoring of garment animation. ACM Transactions on Graphics, 2019, 38(6):Article No. 220. DOI:10.1145/3355089.3356512.
[26] Hahn F, Thomaszewski B, Coros S, Sumner R W, Cole F, Meyer M, DeRose T, Gross M. Subspace clothing simulation using adaptive bases. ACM Trans. Graph., 2014, 33(4):Article No. 105. DOI:10.1145/2601097.2601160.
[27] Xiao Y P, Lai Y K, Zhang F L, Li C P, Gao L. A survey on deep geometry learning:From a representation perspective. Computational Visual Media, 2020, 6(2):113-133. DOI:10.1007/s41095-020-0174-8.
[28] Yuan Y J, Lai Y K, Wu T, Gao L, Liu L. A revisit of shape editing techniques:From the geometric to the neural viewpoint. arXiv:2103.01694, 2021. https://arxiv.org/abs/2103.01694, Jan. 2021.
[29] Wang P S, Liu Y, Guo Y X, Sun C Y, Tong X. O-CNN:Octree-based convolutional neural networks for 3D shape analysis. ACM Transactions on Graphics, 2017, 36(4):Article No. 72. DOI:10.1145/3072959.3073608.
[30] Su H, Maji S, Kalogerakis E, Learned-Miller E G. Multiview convolutional neural networks for 3D shape recognition. In Proc. the 2015 IEEE International Conference on Computer Vision, Dec. 2015, pp.945-953. DOI:10.1109/ICCV.2015.114.
[31] Sinha A, Bai J, Ramani K. Deep learning 3D shape surfaces using geometry images. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.223-240. DOI:10.1007/978-3-319-46466-4_14.
[32] Tan Q, Gao L, Lai Y, Yang J, Xia S. Mesh-based autoencoders for localized deformation component analysis. In Proc. the 32nd Conference on Artificial Intelligence, Feb. 2018, pp.2452-2459.
[33] Tan Q, Gao L, Lai Y, Yang J, Xia S. Variational autoencoders for deforming 3D mesh models. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.5841-5850. DOI:10.1109/CVPR.2018.00612.
[34] Gao L, Lai Y K, Liang D, Chen S Y, Xia S. Efficient and flexible deformation representation for data-driven surface modeling. ACM Transactions on Graphics, 2016, 35(5):Article No. 158. DOI:10.1145/2908736.
[35] Gao L, Lai Y K, Yang J, Zhang L X, Xia S, Kobbelt L. Sparse data driven mesh deformation. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(3):2085-2100. DOI:10.1109/TVCG.2019.2941200.
[36] Zhang M, Wang T, Ceylan D, Mitra N J. Deep detail enhancement for any garment. arXiv:2008.04367, 2020. https://arxiv.org/abs/2008.04367v1, Jan. 2021.
[37] Dong C, Loy C C, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307. DOI:10.1109/TPAMI.2015.2439281.
[38] Liu S, Gang R, Li C, Song R. Adaptive deep residual network for single image super-resolution. Computational Visual Media, 2019, 5(4):391-401. DOI:10.1007/s41095-019-0158-8.
[39] Yue H J, Shen S, Yang J Y, Hu H F, Chen Y F. Reference image guided super-resolution via progressive channel attention networks. Journal of Computer Science and Technology, 2020, 35(3):551-563. DOI:10.1007/s11390-020-0270-3.
[40] Dong C, Loy C C, Tang X. Accelerating the super-resolution convolutional neural network. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.391-407. DOI:10.1007/978-3-319-46475-6_25.
[41] Shi W, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1874-1883. DOI:10.1109/CVPR.2016.207.
[42] Haris M, Shakhnarovich G, Ukita N. Deep back projection networks for super-resolution. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.1664-1673. DOI:10.1109/CVPR.2018.00179.
[43] Kappeler A, Yoo S, Dai Q, Katsaggelos A K. Video superresolution with convolutional neural networks. IEEE Transactions on Computational Imaging, 2016, 2(2):109-122. DOI:10.1109/TCI.2016.2532323.
[44] Chu M, Xie Y, Leal-Taixé L, Thuerey N. Learning temporal coherence via self-supervision for GAN-based video generation. ACM Trans. Graph., 2020, 39(4):Article No. 75. DOI:10.1145/3386569.3392457.
[45] Bhattacharjee P, Das S. Directional attention based video frame prediction using graph convolutional networks. In Proc. the 2019 International Joint Conference on Neural Networks, July 2019, pp.4268-4277. DOI:10.1109/IJCNN.2019.8852090.
[46] Xie Y, Franz E, Chu M, Thuerey N. TempoGAN:A temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Transactions on Graphics, 2018, 37(4):Article No. 95. DOI:10.1145/3197517.3201304.
[47] Kabsch W. A discussion of the solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A:Foundations and Advances, 1978, 34(5):827-828. DOI:10.1107/S0567739478001680.
[48] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249-256.
[49] Bergou M, Mathur S, Wardetzky M, Grinspun E. TRACKS:Toward directable thin shells. ACM Trans. Graph., 2007, 26(3):Article No. 50. DOI:10.1145/1276377.1276439.
[50] Müller M, Gross M. Interactive virtual materials. In Proc. the 2004 Graphics Interface Conference, May 2004, pp.239-246.
[51] Caruana R. Multitask learning. Machine Learning, 1997, 28(1):41-75. DOI:10.1023/A:1007379606734.
[52] Burden R, Faires J. Numerical Analysis (9th Edition). Cengage Learning, 2010.
[53] Ye J, Ma G, Jiang L, Chen L, Li J, Xiong G, Zhang X, Tang M. A unified cloth untangling framework through discrete collision detection. Computer Graphics Forum, 2017, 36(7):217-228. DOI:10.1111/cgf.13287.
[54] Wang H, O'Brien J F, Ramamoorthi R. Data-driven elastic models for cloth:Modeling and measurement. ACM Trans. Graph., 2011, 30(4):Article No. 71. DOI:10.1145/2010324.1964966.
[55] Kingma D P, Ba J. Adam:A method for stochastic optimization. In Proc. the 3rd International Conference on Learning Representations, May 2015.
[1] 张鑫, 陆思源, 王水花, 余翔, 王甦菁, 姚仑, 潘毅, 张煜东. 通过新型深度学习架构诊断COVID-19肺炎[J]. 计算机科学技术学报, 2022, 37(2): 330-343.
[2] Songjie Niu, Shimin Chen. TransGPerf:利用迁移学习建模分布式图计算性能[J]. 计算机科学技术学报, 2021, 36(4): 778-791.
[3] Sheng-Luan Hou, Xi-Kun Huang, Chao-Qun Fei, Shu-Han Zhang, Yang-Yang Li, Qi-Lin Sun, Chuan-Qing Wang. 基于深度学习的文本摘要研究综述[J]. 计算机科学技术学报, 2021, 36(3): 633-663.
[4] Yu-Jie Yuan, Yukun Lai, Tong Wu, Lin Gao, Li-Gang Liu. 回顾形状编辑技术:从几何角度到神经网络方法[J]. 计算机科学技术学报, 2021, 36(3): 520-554.
[5] Wei Du, Yu Sun, Hui-Min Bao, Liang Chen, Ying Li, Yan-Chun Liang. 基于迁移学习与深度学习的人类血液分泌蛋白预测框架[J]. 计算机科学技术学报, 2021, 36(2): 234-247.
[6] Jun Gao, Paul Liu, Guang-Di Liu, Le Zhang. 基于深度学习与波束偏转的穿刺针定位与增强算法[J]. 计算机科学技术学报, 2021, 36(2): 334-346.
[7] Hua Chen, Juan Liu, Qing-Man Wen, Zhi-Qun Zuo, Jia-Sheng Liu, Jing Feng, Bao-Chuan Pang, Di Xiao. CytoBrain:基于深度学习技术的宫颈癌筛查系统[J]. 计算机科学技术学报, 2021, 36(2): 347-360.
[8] Andrea Caroppo, Alessandro Leone, Pietro Siciliano. 用于老年人面部表情识别的深度学习模型和传统机器学习方法的对比研究[J]. 计算机科学技术学报, 2020, 35(5): 1127-1146.
[9] Chuang-Ye Zhang, Yan Niu, Tie-Ru Wu, Xi-Ming Li. 基于跨通道细节的低代价的彩色图像超分辨率重建与增强[J]. 计算机科学技术学报, 2020, 35(4): 889-899.
[10] Huan-Jing Yue, Sheng Shen, Jing-Yu Yang, Hao-Feng Hu, Yan-Fang Chen. 基于渐进式通道注意力网络的参考图引导超分辨率研究[J]. 计算机科学技术学报, 2020, 35(3): 551-563.
[11] 梁盾, 郭元晨, 张少魁, 穆太江, 黄晓蕾. 车道检测-新结果和调查研究[J]. 计算机科学技术学报, 2020, 35(3): 493-505.
[12] Zheng Zeng, Lu Wang, Bei-Bei Wang, Chun-Meng Kang, Yan-Ning Xu. 一种基于多重残差网络的随机渐进式光子映射的降噪方法[J]. 计算机科学技术学报, 2020, 35(3): 506-521.
[13] Shuai Li, Zheng Fang, Wen-Feng Song, Ai-Min Hao, Hong Qin. 基于双向特征共享网络的多人姿态估计方法研究[J]. 计算机科学技术学报, 2019, 34(3): 522-536.
[14] Na Ding, Ye-Peng Liu, Lin-Wei Fan, Cai-Ming Zhang. 基于动态轻型数据库和局部特征插值的单幅超分辨率重建方法[J]. 计算机科学技术学报, 2019, 34(3): 537-549.
[15] Jin-Hua Tao, Zi-Dong Du, Qi Guo, Hui-Ying Lan, Lei Zhang, Sheng-Yuan Zhou, Ling-. 智能处理器的评测基准[J]. , 2018, 33(1): 1-23.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 周笛;. A Recovery Technique for Distributed Communicating Process Systems[J]. , 1986, 1(2): 34 -43 .
[2] 刘明业; 洪恩宇;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[3] 章萃; 赵沁平; 徐家福;. Kernel Language KLND[J]. , 1986, 1(3): 65 -79 .
[4] 郑国梁; 李辉;. The Design and Implementation of the Syntax-Directed Editor Generator(SEG)[J]. , 1986, 1(4): 39 -48 .
[5] 闵应骅; 韩智德;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[6] 龚振和;. On Conceptual Model Specification and Verification[J]. , 1987, 2(1): 35 -50 .
[7] 闵应骅;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[8] 黄国祥; 刘健;. A Key-Lock Access Control[J]. , 1987, 2(3): 236 -243 .
[9] 林琦; 夏培肃;. The Design and Implementation of a Very Fast Experimental Pipelining Computer[J]. , 1988, 3(1): 1 -6 .
[10] 孙成政; 慈云桂;. A New Method for Describing the AND-OR-Parallel Execution of Logic Programs[J]. , 1988, 3(2): 102 -112 .
版权所有 © 《计算机科学技术学报》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持:support@magtech.com.cn
总访问量: