Journal of Computer Science and Technology ›› 2022, Vol. 37 ›› Issue (3): 539-558.doi: 10.1007/s11390-022-2030-z

Special Issue: Artificial Intelligence and Pattern Recognition

• Special Section on Self-Learning with Deep Neural Networks • Previous Articles     Next Articles

Self-Supervised Music Motion Synchronization Learning for Music-Driven Conducting Motion Generation

Fan Liu1(刘凡), Member, CCF, IEEE, De-Long Chen1,*(陈德龙), Rui-Zhi Zhou1(周睿志), Sai Yang2(杨赛), and Feng Xu1(许峰), Member, CCF        

  1. 1College of Computer and Information, Hohai University, Nanjing 211100, China
    2School of Electrical Engineering, Nantong University, Nantong 226019, China
  • Received:2021-11-19 Revised:2022-02-28 Accepted:2022-03-10 Online:2022-05-30 Published:2022-05-30
  • Contact: De-Long Chen E-mail:chendelong@hhu.edu.cn
  • About author:De-Long Chen received his B.S. degree in computer science in Hohai University, Nanjing, in 2021. He is currently a research assistant in Hohai University, Nanjing, and a research intern at MEGVII Technology, Beijing. His research includes computer vision, music information retrieval, multimodal learning, unsupervised learning and self-supervised learning.
  • Supported by:
    This work was partially funded by the Natural Science Foundation of Jiangsu Province of China under Grant No. BK20191298 and the National Natural Science Foundation of China under Grant No. 61902110.

The correlation between music and human motion has attracted widespread research attention. Although recent studies have successfully generated motion for singers, dancers, and musicians, few have explored motion generation for orchestral conductors. The generation of music-driven conducting motion should consider not only the basic music beats, but also mid-level music structures, high-level music semantic expressions, and hints for different parts of orchestras (strings, woodwind, etc.). However, most existing conducting motion generation methods rely heavily on human-designed rules, which significantly limits the quality of generated motion. Therefore, we propose a novel Music Motion Synchronized Generative Adversarial Network (M2S-GAN), which generates motions according to the automatically learned music representations. More specifically, M2S-GAN is a cross-modal generative network comprising four components: 1) a music encoder that encodes the music signal; 2) a generator that generates conducting motion from the music codes; 3) a motion encoder that encodes the motion; 4) a discriminator that differentiates the real and generated motions. These four components respectively imitate four key aspects of human conductors: understanding music, interpreting music, precision and elegance. The music and motion encoders are first jointly trained by a self-supervised contrastive loss, and can thus help to facilitate the music motion synchronization during the following adversarial learning process. To verify the effectiveness of our method, we construct a large-scale dataset, named ConductorMotion100, which consists of unprecedented 100 hours of conducting motion data. Extensive experiments on ConductorMotion100 demonstrate the effectiveness of M2S-GAN. Our proposed approach outperforms various comparison methods both quantitatively and qualitatively. Through visualization, we show that our approach can generate plausible, diverse, and music-synchronized conducting motion.

Key words: self-supervised learning; generative adversarial network; human motion generation

;

[1] Ren X, Li H, Huang Z, Chen Q. Self-supervised dance video synthesis conditioned on music. In Proc. the 28th ACM International Conference on Multimedia, October 2020, pp.46-54. DOI: 10.1145/3394171.3413932.

[2] Lee H, Yang X, Liu M, Wang T, Lu Y, Yang M, Kautz J. Dancing to music. In Proc. the Annual Conference on Neural Information Processing Systems, December 2019, pp.3581-3591.

[3] Li B, Maezawa A, Duan Z. Skeleton plays piano: Online generation of pianist body movements from MIDI performance. In Proc. the 19th International Society for Music Information Retrieval Conference, September 2018, pp.218-224.

[4] Kao H, Su L. Temporally guided music-to-body-movement generation. In Proc. the 28th ACM International Conference on Multimedia, October 2020, pp.147-155. DOI: 10.1145/3394171.3413848.

[5] Ruttkay Z, Huang Z, Eliens A. The conductor: Gestures for embodied agents with logic programming. In Proc. the Joint Annual ERCIM/CoLogNet International Workshop on Constraint and Logic Programming, June 30-July 2, 2003, pp.9-16. DOI: 10.1007/978-3-540-24662-6.

[6] Bos P, Reidsma D, Ruttkay Z, Nijholt A. Interacting with a virtual conductor. In Proc. the 5th International Conference on Entertainment Computing, September 2006, pp.25-30. DOI: 10.1007/11872320.

[7] Nijholt A, Reidsma D, Ebbers R, Maat M. The virtual conductor: Learning and teaching about music, performing, and conducting. In Proc. the 8th IEEE International Conference on Advanced Learning Technologies, July 2008, pp.897-899. DOI: 10.1109/ICALT.2008.43.

[8] Maat M, Ebbers R, Reidsma D, Nijholt A. Beyond the beat: Modelling intentions in a virtual conductor. In Proc. the 2nd International Conference on Intelligent Technologies for Interactive Entertainment, January 2008, Article No. 12. DOI: 10.4108/ICST.INTETAIN2008.2489.

[9] Reidsma D, Nijholt A, Bos P. Temporal interaction between an artificial orchestra conductor and human musicians. Comput. Entertain., 2008, 6(4): Article No. 53. DOI: 10.1145/1461999.1462005.

[10] Takatsu R, Maki Y, Inoue T, Okada K, Shigeno H. Multiple virtual conductors allow amateur orchestra players to perform better and more easily. In Proc. the 20th IEEE International Conference on Computer Supported Cooperative Work in Design, May 2016, pp.486-491. DOI: 10.1109/CSCWD.2016.7566038.

[11] Katayama N, Takatsu R, Inoue T, Shigeno H, Okada K. Efficient generation of conductor avatars for the concert by multiple virtual conductors. In Proc. the 8th International Conference on Collaboration Technologies and Social Computing, Sept. 2016, pp.45-57. DOI: 10.1007/978-981-10-2618-8.

[12] Wang T, Zheng N, Li Y, Xu Y, Shum H. Learning kernel-based HMMs for dynamic sequence synthesis. Graph. Model., 2003, 65(4): 206-221. DOI: 10.1016/S1524-0703(03)00040-7.

[13] Shu X, Qi G, Tang J, Wang J. Weakly-shared deep transfer networks for heterogeneous-domain knowledge propagation. In Proc. the 23rd Annual ACM Conference on Multimedia, October 2015, pp.35-44. DOI: 10.1145/2733373.2806216.

[14] Tang J, Shu X, Qi G, Li Z, Wang M, Yan S, Jain R C. Tri-clustered tensor completion for social-aware image tag refinement. IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39(8): 1662-1674. DOI: 10.1109/TPAMI.2016.2608882.

[15] Tang J, Shu X, Li Z, Jiang Y, Tian Q. Social anchor-unit graph regularized tensor completion for large-scale image retagging. IEEE Trans. Pattern Anal. Mach. Intell., 2019, 41(8): 2027-2034. DOI: 10.1109/TPAMI.2019.2906603.

[16] Du X, Yang Y, Yang L, Shen F, Qin Z, Tang J. Captioning videos using large-scale image corpus. J. Comput. Sci. Technol., 2017, 32(3): 480-493. DOI: 10.1007/s11390-017-1738-7.

[17] Korbar B, Tran D, Torresani L. Cooperative learning of audio and video models from self-supervised synchronization. In Proc. the Annual Conference on Neural Information Processing Systems, December 2018, pp.7774-7785.

[18] Arjovsky M, Chintala S, Bottou L. Wasserstein GAN. arXiv:1701.07875, 2017. https://arxiv.org/pdf/1701.078 75.pdf, Dec. 2021.

[19] Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville A C. Improved training of Wasserstein GANs. In Proc. the Annual Conference on Neural Information Processing Systems, December 2017, pp.5767-5777.

[20] Redmon J, Farhadi A. YOLOv3: An incremental improvement. arXiv:1804.02767, 2018. https://arxiv.org/abs/ 1804.02767, Dec. 2021.

[21] Fang H, Xie S, Tai Y, Lu C. RMPE: Regional multi-person pose estimation. In Proc. the 2017 IEEE International Conference on Computer Vision, October 2017, pp.2353-2362. DOI: 10.1109/ICCV.2017.256.

[22] Geuther B, Breese A, Wang Y. A study on musical conducting robots and their users. In Proc. the 10th IEEE-RAS International Conference on Humanoid Robots, December 2010, pp.124-129. DOI: 10.1109/ICHR.2010.5686302.

[23] Salgian A, Ault C, Nakra T M, Wang Y, Stone M. Multidisciplinary computer science through conducting robots. In Proc. the 42nd ACM Technical Symposium on Computer Science Education, March 2011, pp.219-224. DOI: 10.1145/1953163.1953229.

[24] Salgian A, Ault C, Nakra T M, Wang Y, Stone M. A theory of ‘multiple creativities’: Outcomes from an undergraduate seminar in conducting robots. In Proc. the Music, Mind, and Invention Workshop, March 2012.

[25] Dansereau D G, Brock N, Cooperstock J R. Predicting an orchestral conductor’s baton movements using machine learning. Comput. Music. J., 2013, 37(2): 28-45. DOI: 10.1162/COMJa.

[26] Yalta N. Sequential deep learning for dancing motion generation. In Proc. the 46th AI Challenge Study Group, November 2016, pp.43-49.

[27] Li Z, Liu F, Yang W, Peng S, Zhou J. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems. DOI: 10.1109/TNNLS.2021.3084827.

[28] Yalta N, Watanabe S, Nakadai K, Ogata T. Weakly-supervised deep recurrent neural networks for basic dance step generation. In Proc. the 2019 International Joint Conference on Neural Networks, July 2019. DOI: 10.1109/IJCNN.2019.8851872.

[29] Tang T, Jia J, Mao H. Dance with melody: An LSTM-autoencoder approach to music-oriented dance synthesis. In Proc. the 2018 ACM Multimedia Conference on Multimedia, October 2018, pp.1598-1606. DOI: 10.1145/3240508.3240526.

[30] Bogaers A, Yumak Z, Volk A. Music-driven animation generation of expressive musical gestures. In Proc. the 2020 International Conference on Multimodal Interaction, October 2020, pp.22-26. DOI: 10.1145/3395035.3425244.

[31] Qi Y, Liu Y, Sun Q. Music-driven dance generation. IEEE Access, 2019, 7: 166540-166550. DOI: 10.1109/ACCESS.2019.2953698.

[32] Shlizerman E, Dery L M, Schoen H, Kemelmacher-Shlizerman I. Audio to body dynamics. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.7574-7583. DOI: 10.1109/CVPR.2018.00790.

[33] Haag K, Shimodaira H. Bidirectional LSTM networks employing stacked bottleneck features for expressive speech-driven head motion synthesis. In Proc. the 16th International Conference on Intelligent Virtual Agents, September 2016, pp.198-207. DOI: 10.1007/978-3-319-47665-0.

[34] Ferstl Y, McDonnell R. Investigating the use of recurrent motion modelling for speech gesture generation. In Proc. the 18th International Conference on Intelligent Virtual Agents, November 2018, pp.93-98. DOI: 10.1145/3267851.3267898.

[35] Sadoughi N, Busso C. Joint learning of speech-driven facial motion with bidirectional long-short term memory. In Proc. the 17th International Conference on Intelligent Virtual Agents, August 2017, pp.389-402. DOI: 10.1007/978-3-319-67401-8.

[36] Huang R, Hu H, Wu W, Sawada K, Zhang M, Jiang D. Dance revolution: Long-term dance generation with music via curriculum learning. In Proc. the 9th International Conference on Learning Representations, May 2021.

[37] Sun G, Wong Y, Cheng Z, Kankanhalli M S, Geng W, Li X. DeepDance: Music-to-dance motion choreography with adversarial learning. IEEE Trans. Multim., 2020, 23: 497-509. DOI: 10.1109/TMM.2020.2981989.

[38] Ahn H, Kim J, Kim K, Oh S. Generative autoregressive networks for 3D dancing move synthesis from music. IEEE Robotics and Automation Letters, 2020, 5(2): 3501-3508. DOI: 10.1109/LRA.2020.2977333.

[39] Lee J, Kim S, Lee K. Automatic choreography generation with convolutional encoder-decoder network. In Proc. the 20th International Society for Music Information Retrieval Conference, November 2019, pp.894-899. DOI: 10.5281/zenodo.3527958.

[40] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser L, Polosukhin I. Attention is all you need. In Proc. the Annual Conference on Neural Information Processing Systems, December 2017, pp.5998-6008.

[41] Li R, Yang S, Ross D A, Kanazawa A. Learn to dance with AIST + +: Music conditioned 3D dance generation. arXiv:2101.08779, 2021. https://arxiv.org/abs/2101.08779, Dec. 2021.

[42] Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A C, Bengio Y. Generative adversarial nets. In Proc. the Annual Conference on Neural Information Processing Systems, December 2014, pp.2672-2680.

[43] Ginosar S, Bar A, Kohavi G, Chan C, Owens A, Malik J. Learning individual styles of conversational gesture. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019, pp.3497-3506. DOI: 10.1109/CVPR.2019.00361.

[44] Eskimez S E, Maddox R K, Xu C, Duan Z. End-to-end generation of talking faces from noisy speech. In Proc. the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, May 2020, pp.1948-1952. DOI: 10.1109/ICASSP40776.2020.9054103.

[45] Song Y, Zhu J, Li D, Wang A, Qi H. Talking face generation by conditional recurrent adversarial network. In Proc. the 28th International Joint Conference on Artificial Intelligence, August 2019, pp.919-925.

[46] Sadoughi N, Busso C. Novel realizations of speech-driven head movements with generative adversarial networks. In Proc. the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, April 2018, pp.6169-6173. DOI: 10.1109/ICASSP.2018.8461967.

[47] Ferstl Y, Neff M, McDonnell R. Multi-objective adversarial gesture generation. In Proc. the Motion, Interaction and Games, October 2019, Article No. 3. DOI: 10.1145/3359566.3360053.

[48] Sarasúa Á. Context-aware gesture recognition in classical music conducting. In Proc. the 21st ACM International Conference on Multimedia, October 2013, pp.1059-1062. DOI: 10.1145/2502081.2502216.

[49] Sarasúa Á, Guaus E. Beat tracking from conducting gestural data: A multi-subject study. In Proc. the International Workshop on Movement and Computing, June 2014, pp.118-123. DOI: 10.1145/2617995.2618016.

[50] Karipidou K, Ahnlund J, Friberg A, Alexanderson S, Kjellström H. Computer analysis of sentiment interpretation in musical conducting. In Proc. the 12th IEEE International Conference on Automatic Face & Gesture Recognition, May 30-June 3, 2017, pp.400-405. DOI: 10.1109/FG.2017.57.

[51] Huang Y, Chen T, Moran N, Coleman S, Su L. Identifying expressive semantics in orchestral conducting kinematics. In Proc. the 20th International Society for Music Information Retrieval Conference, November 2019, pp.115-122. DOI: 10.5281/zenodo.3527753.

[52] Lemouton S, Borghesi R, Haapamäki S, Bevilacqua F, Fléty E. Following orchestra conductors: The IDEA open movement dataset. In Proc. the 6th International Conference on Movement and Computing, October 2019, Article No. 25. DOI: 10.1145/3347122.3359599.

[53] Yan S, Xiong Y, Lin D. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proc. the 32nd AAAI Conference on Artificial Intelligence, February 2018, pp.7444-7452.

[54] Bai S, Kolter J Z, Koltun V. Convolutional sequence modeling revisited. In Proc. the 6th International Conference on Learning Representations, April 30-May 3, 2018.

[55] Arandjelovic R, Zisserman A. Look, listen and learn. In Proc. the 2017 IEEE International Conference on Computer Vision, October 2017, pp.609-617. DOI: 10.1109/ICCV.2017.73.

[56] Chung J S, Zisserman A. Out of time: Automated lip sync in the wild. In Proc. the 2016 ACCV International Workshops on Computer Vision, November 2016, pp.251-263. DOI: 10.1007/978-3-319-54427-4.

[57] Chen L, Srivastava S, Duan Z, Xu C. Deep cross-modal audio-visual generation. In Proc. the Thematic Workshops of the 2017 ACM Multimedia, October 2017, pp.349-357. DOI: 10.1145/3126686.3126723.

[58] Hao W, Zhang Z, Guan H. CMCGAN: A uniform framework for cross-modal visual-audio mutual generation. In Proc. the 32nd AAAI Conference on Artificial Intelligence, February 2018, pp.6886-6893.

[59] Zhou H, Liu Z, Xu X, Luo P, Wang X. Vision-infused deep audio inpainting. In Proc. the 2019 IEEE/CVF International Conference on Computer Vision, October 27-November 2, 2019, pp.283-292. DOI: 10.1109/ICCV.2019.00037.

[60] Choi H, Park C, Lee K. From inference to generation: End-to-end fully self-supervised generation of human face from speech. In Proc. the 8th International Conference on Learning Representations, April 2020.

[61] Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.694-711. DOI: 10.1007/978-3-319-46475-6.

[62] Li M, Hsu W, Xie X, Cong J, Gao W. SACNN: Self-attention convolutional neural network for low-dose CT denoising with self-supervised perceptual loss network. IEEE Trans. Medical Imaging, 2020, 39(7): 2289-2301. DOI: 10.1109/TMI.2020.2968472.

[63] Akella R T, Halder S S, Shandeelya A P, Pankajakshan V. Enhancing perceptual loss with adversarial feature matching for super-resolution. In Proc. the 2020 International Joint Conference on Neural Networks, July 2020. DOI: 10.1109/IJCNN48605.2020.9207102.

[64] Tieleman T, Hinton G. Lecture 6.5-rmsprop, COURSERA: Neural networks for machine learning. Technical Report, University of Toronto, 2012.

[65] Diederik P K, Jimmy B. Adam: A method for stochastic optimization. In Proc. the 3rd International Conference on Learning Representations, May 2015.

[66] Sarasúa Á, Caramiaux B, Tanaka A. Machine learning of personal gesture variation in music conducting. In Proc. the 2016 CHI Conference on Human Factors in Computing Systems, May 2016, pp.3428-3432. DOI: 10.1145/2858036.2858328.

[67] Cosentino S, Petersen K, Lin Z, Bartolomeo L, Sessa S, Zecca M, Takanishi A. Natural human-robot musical interaction: Understanding the music conductor gestures by using the WB-4 inertial measurement system. Adv. Robotics, 2014, 28(11): 781-792. DOI: 10.1080/01691864.2014.889577.

[68] Lee K, Junokas M J, Amanzadeh M, Garnett G E. An analysis of basic expressive qualities in instrumental conducting. In Proc. the 2nd International Workshop on Movement and Computing, August 2015, pp.148-155. DOI: 10.1145/2790994.2791005.

[1] Peng-Fei Fang, Xian Li, Yang Yan, Shuai Zhang, Qi-Yue Kang, Xiao-Fei Li, and Zhen-Zhong Lan. Connecting the Dots in Self-Supervised Learning: A Brief Survey for Beginners [J]. Journal of Computer Science and Technology, 2022, 37(3): 507-526.
[2] Peng-Fei Sun, Ya-Wen Ouyang, Ding-Jie Song, and Xin-Yu Dai. Self-Supervised Task Augmentation for Few-Shot Intent Detection [J]. Journal of Computer Science and Technology, 2022, 37(3): 527-538.
[3] Chun-Yang Ruan, Ye Wang, Jiangang Ma, Yanchun Zhang, Xin-Tian Chen. Adversarial Heterogeneous Network Embedding with Metapath Attention Mechanism [J]. Journal of Computer Science and Technology, 2019, 34(6): 1217-1229.
[4] Yifan Wu, Fan Yang, Yong Xu, Haibin Ling. Privacy-Protective-GAN for Privacy Preserving Face De-Identification [J]. Journal of Computer Science and Technology, 2019, 34(1): 47-60.
[5] Huai-Yu Li, Wei-Ming Dong, Bao-Gang Hu. Facial Image Attributes Transformation via Conditional Recycle Generative Adversarial Networks [J]. , 2018, 33(3): 511-521.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] Zhou Di;. A Recovery Technique for Distributed Communicating Process Systems[J]. , 1986, 1(2): 34 -43 .
[2] Li Wei;. A Structural Operational Semantics for an Edison Like Language(2)[J]. , 1986, 1(2): 42 -53 .
[3] Pan Qijing;. A Routing Algorithm with Candidate Shortest Path[J]. , 1986, 1(3): 33 -52 .
[4] Chen Zhaoxiong; Gao Qingshi;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[5] Zhong Renbao; Xing Lin; Ren Zhaoyang;. An Interactive System SDI on Microcomputer[J]. , 1987, 2(1): 64 -71 .
[6] Li Renwei;. Soundness and Completeness of Kung s Reasoning Procedure[J]. , 1988, 3(1): 7 -15 .
[7] Bao Feng;. On the Condition for FSM Being a Scrambler[J]. , 1988, 3(1): 70 -74 .
[8] Xue Xing; Sun Zhongxiu; Zhou Jianqiang; Xu Xihao;. A Message-Based Distributed Kernel for a Full Heterogeneous Environment[J]. , 1990, 5(1): 47 -56 .
[9] A.Corradi; L.Leonardi;. PROM:A Support for Robust Replication in a Distributed Object Environment[J]. , 1990, 5(2): 139 -155 .
[10] Yu Xiangdong;. Some Hard Examples for the Resolution Method[J]. , 1990, 5(3): 302 -304 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

Home
Editorial Board
Author Guidelines
Subscription
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved