›› 2018, Vol. 33 ›› Issue (3): 487-501.doi: 10.1007/s11390-018-1833-4

Special Issue: Artificial Intelligence and Pattern Recognition; Computer Graphics and Multimedia

• Special Section of CVM 2018 • Previous Articles     Next Articles

Multi-exposure Motion Estimation based on Deep Convolutional Networks

Zhi-Feng Xie1,2, Yu-Chen Guo1, Shu-Han Zhang1, Wen-Jun Zhang1, Li-Zhuang Ma2,3, Member, CCF   

  1. 1 Department of Film and Television Engineering, Shanghai University, Shanghai 200072, China;
    2 Shanghai Engineering Research Center of Motion Picture Special Effects, Shanghai 200072, China;
    3 Department of Software Science and Technology, East China Normal University, Shanghai 200062, China
  • Received:2017-12-27 Revised:2018-03-06 Online:2018-05-05 Published:2018-05-05
  • About author:Zhi-Feng Xie received his Ph.D. degree in computer application technology from Shanghai Jiao Tong University, Shanghai, in 2013. He was a research assistant at the Department of Computer Science, City University of Hong Kong, Hong Kong, in 2011. He is now an assistant professor with Shanghai University, Shanghai. His research interests include image/video editing, computer graphics, and digital media technology.
  • Supported by:

    This work was supported by the National Natural Science Foundation of China under Grant Nos. 61303093, 61472245, and 61402278, the Innovation Program of the Science and Technology Commission of Shanghai Municipality of China under Grant No. 16511101300, and the Gaofeng Film Discipline Grant of Shanghai Municipal Education Commission of China.

PDF

661

In motion estimation, illumination change is always a troublesome obstacle, which often causes severely performance reduction of optical flow computation. The essential reason is that most of estimation methods fail to formalize a unified definition in color or gradient domain for diverse environmental changes. In this paper, we propose a new solution based on deep convolutional networks to solve the key issue. Our idea is to train deep convolutional networks to represent the complex motion features under illumination change, and further predict the final optical flow fields. To this end, we construct a training dataset of multi-exposure image pairs by performing a series of non-linear adjustments in the traditional datasets of optical flow estimation. Our end-to-end network model consists of three main components:low-level feature network, fusion feature network, and motion estimation network. The former two components belong to the contracting part of our model in order to extract and represent the multi-exposure motion features; the third component is the expanding part of our model in order to learn and predict the high-quality optical flow. Compared with many state-of-the-art methods, our motion estimation based on deep convolutional networks can eliminate the obstacle of illumination change and yield optical flow results with competitive accuracy and time efficiency. Moreover, the good performance of our model is also demonstrated in some multi-exposure video applications, like HDR (High Dynamic Range) composition and flicker removal.

[1] Bouguet J. Pyramidal implementation of the Lucas Kanade feature tracker description of the algorithm. http://robots.stanford.edu/cs223b04/algotracking.pdf, Mar. 2018.

[2] Liu C. Beyond pixels:Exploring new representations and applications for motion analysis[Ph.D. Thesis]. Massachusetts Institute of Technology, MA, USA, 2009.

[3] Sun D Q, Roth S, Black M J. Secrets of optical flow estimation and their principles. In Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, June 2010, pp.2432-2439.

[4] Brox T, Malik J. Large displacement optical flow:Descriptor matching in variational motion estimation. IEEE Trans. Pattern Analysis and Machine Intelligence, 2011, 33(3):500-513.

[5] Xu L, Jia J Y, Matsushita Y. Motion detail preserving optical flow estimation. IEEE Trans. Pattern Analysis and Machine Intelligence, 2012, 34(9):1744-1757.

[6] Brox T, Bruhn A, Papenberg N, Weickert J. High accuracy optical flow estimation based on a theory for warping. In Proc. the 8th European Conf. Computer Vision, May 2004, pp.25-36.

[7] Liu C, Yuen J, Torralba A. SIFT flow:Dense correspondence across scenes and its applications. IEEE Trans. Pattern Analysis and Machine Intelligence, 2011, 33(5):978-994.

[8] Dosovitskiy A, Fischer P, Ilg E, Häusser P, Hazirbas C, Golkov V, van der Smagt P, Cremers D, Brox T. FlowNet:Learning optical flow with convolutional networks. In Proc. IEEE Int. Conf. Computer Vision, December 2015, pp.2758-2766.

[9] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In Proc. the 25th Int. Conf. Neural Information Processing Systems, December 2012, pp.1097-1105.

[10] Girshick R. Fast R-CNN. In Proc. IEEE Int. Conf. Computer Vision, December 2015, pp.1440-1448.

[11] Farabet C, Couprie C, Najman L, LeCun Y. Learning hierarchical features for scene labeling. IEEE Trans. Pattern Analysis and Machine Intelligence, 2013, 35(8):1915-1929.

[12] Eigen D, Puhrsch C, Fergus R. Depth map prediction from a single image using a multi-scale deep network. In Proc. the 28th Annual Conf. Neural Information Processing Systems, January 2014, pp.2366-2374.

[13] Teney D, Hebert M. Learning to extract motion from videos in convolutional neural networks. In Proc.the 13th Asian Conf. Computer Vision, November 2016, pp.412-428.

[14] Horn B K P, Schunck B G. Determining optical flow. Artificial Intelligence, 1981, 17(1/2/3):185-203.

[15] Anandan P. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 1989, 2(3):283-310.

[16] Bergen J R, Anandan P, Hanna K J, Hingorani R. Hierarchical model-based motion estimation. In Proc. the 2nd European Conf. Computer Vision, May 1992, pp.237-252.

[17] Bruhn A, Weickert J. Towards ultimate motion estimation:Combining highest accuracy with real-time performance. In Proc. the 10th IEEE Int. Conf. Computer Vision, October 2005, pp.749-755.

[18] Bruhn A, Weickert J, Schnörr C. Lucas/Kanade meets Horn/Schunck:Combining local and global optic flow methods. International Journal of Computer Vision, 2005, 61(3):211-231.

[19] Lempitsky V, Roth S, Rother C. FusionFlow:Discretecontinuous optimization for optical flow estimation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2008.

[20] Wedel A, Cremers D, Pock T, Bischof H. Structure-and motion-adaptive regularization for high accuracy optic flow. In Proc. the 12th IEEE Int. Conf. Computer Vision, September 29-October 2, 2009, pp.1663-1668.

[21] Zimmer H, Bruhn A, Weickert J. Optic flow in harmony. International Journal of Computer Vision, 2011, 93(3):368-388.

[22] Mémin E, Pérez P. Hierarchical estimation and segmentation of dense motion fields. International Journal of Computer Vision, 2002, 46(2):129-155.

[23] Xu L, Chen J N, Jia J Y. A segmentation based variational model for accurate optical flow estimation. In Proc. the 10th European Conf. Computer Vision, October 2008, pp.671-684.

[24] Lei C, Yang Y H. Optical flow estimation on coarse-to-fine region-trees using discrete optimization. In Proc. the 12th IEEE Int. Conf. Computer Vision, September 29-October 2, 2009, pp.1562-1569.

[25] Werlberger M, Pock T, Bischof H. Motion estimation with non-local total variation regularization. In Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, June 2010, pp.2464-2471.

[26] Xiao J J, Cheng H, Sawhney H, Rao C, Isnardi M. Bilateral filtering-based optical flow estimation with occlusion detection. In Proc. the 9th European Conf. Computer Vision, May 2006, pp.211-224.

[27] Seitz S M, Baker S. Filter flow. In Proc. the 12th IEEE Int. Conf. Computer Vision, September 29-Octomber 2, 2009, pp.143-150.

[28] Brox T, Bregler C, Malik J. Large displacement optical flow. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2009, pp.41-48.

[29] Steinbrucker F, Pock T, Cremers D. Large displacement optical flow computation without warping. In Proc. the 12th IEEE Int. Conf. Computer Vision, September 29-October 2, 2009, pp.1609-1614.

[30] Sand P, Teller S. Particle video:Long-range motion estimation using point trajectories. In Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, June 2006, pp.2195-2202.

[31] Chen Z Y, Jin H L, Lin Z, Cohen S, Wu Y. Large displacement optical flow from nearest neighbor fields. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2013, pp.2443-2450.

[32] Revaud J, Weinzaepfel P, Harchaoui Z, Schmid C. EpicFlow:Edge-preserving interpolation of correspondences for optical flow. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2015, pp.1164-1172.

[33] Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C. DeepFlow:Large displacement optical flow with deep matching. In Proc. IEEE Int. Conf. Computer Vision, December 2013, pp.1385-1392.

[34] Bailer C, Taetz B, Stricker D. Flow fields:Dense correspondence fields for highly accurate large displacement optical flow estimation. In Proc. IEEE Int. Conf. Computer Vision, December 2015, pp.4015-4023.

[35] Black M J, Anandan P. The robust estimation of multiple motions:Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 1996, 63(1):75-104.

[36] Haussecker H W, Fleet D J Computing optical flow with physical models of brightness variation. IEEE Trans. Pattern Analysis and Machine Intelligence, 2001, 23(6):661-673.

[37] Shen X Y, Xu L, Zhang Q, Jia J Y. Multi-modal and multispectral registration for natural images. In Proc. the 13th European Conf. Computer Vision, September 2014, pp.309-324.

[38] Kumar A, Tung F, Wong A, Clausi D A. A decoupled approach to illumination-robust optical flow estimation. IEEE Trans. Image Processing, 2013, 22(10):4136-4147.

[39] Mohamed M A, Rashwan H A, Mertsching B, García M A, Puig D. Illumination-robust optical flow using a local directional pattern. IEEE Trans. Circuits and Systems for Video Technology, 2014, 24(9):1499-1508.

[40] Roth S, Black M J. On the spatial statistics of optical flow. In Proc. the 10th IEEE Int. Conf. Computer Vision, October 2005, pp.42-49.

[41] Sun D Q, Roth S, Lewis J P, Black M J. Learning optical flow. In Proc. the 10th European Conf. Computer Vision, October 2008, pp.83-97.

[42] Rosenbaum D, Zoran D, Weiss Y. Learning the local statistics of optical flow. In Proc. the 27th Annual Conf. Neural Information Processing Systems, December 2013, pp.2373-2381.

[43] Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T. FlowNet 2.0:Evolution of optical flow estimation with deep networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, July 2017, pp.1647-1655.

[44] Zhao W B. A concise tutorial on human motion tracking and recognition with Microsoft Kinect. Science China Information Sciences, 2016, 59(9):93101.

[45] Xia S H, Gao L, Lai Y K, Yuan M Z, Chai J X. A survey on human performance capture and animation. Journal of Computer Science and Technology, 2017, 32(3):536-554.

[46] Liu B, Xu K, Martin R P. Static scene illumination estimation from videos with applications. Journal of Computer Science and Technology, 2017, 32(3):430-442.

[47] Xie Z F, Tang S, Huang D J, Ding Y D, Ma L Z. Photographic appearance enhancement via detail-based dictionary learning. Journal of Computer Science and Technology, 2017, 32(3):417-429.
No related articles found!
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] Ju-Hum Kwon, Chee-Yang Song, Chang-Joo Moon, and Doo-Kwon Baik. Bridging Real World Semantics to Model World Semantics for Taxonomy Based Knowledge Representation System[J]. , 2005, 20(3): 296 -308 .
[2] Bo Yan, You-Xing Qu, Feng-Lou Mao, Victor N. Olman, and Ying Xu. PRIME: A Mass Spectrum Data Mining Tool for De Novo Sequencing and PTMs Identification[J]. , 2005, 20(4): 483 -490 .
[3] Li-Guo Yu and Srini Ramaswamy. Component Dependency in Object-Oriented Software[J]. , 2007, 22(3): 379 -386 .
[4] Ji-Gang Wu, Thambipillai Srikanthan, and Guang-Wei Zou. New Model and Algorithm for Hardware/Software Partitioning[J]. , 2008, 23(4 ): 644 -651 .
[5] Jian-Yun Liu (刘建芸), Yu-Hang Zhao (赵宇航), Member, CCF, Zhao-Xiang Zhang (张兆翔), Member, CCF, ACM, IEEE, Yun-Hong Wang (王蕴红), Member, CCF, ACM, IEEE, Xue-Mei Yuan (袁雪梅), Lei Hu (胡磊), Member, CCF, and Zhe. Spam Short Messages Detection via Mining Social Networks[J]. , 2012, 27(3): 506 -514 .
[6] Belal Al-Khateeb and Graham Kendall, Senior Member, IEEE. Effect of Look-Ahead Depth in Evolutionary Checkers[J]. , 2012, 27(5): 996 -1006 .
[7] Ren-Jie He and Zhen-Yu Yang, Member, CCF, ACM, IEEE. Differential Evolution with Adaptive Mutation and Parameter Control Using Lévy Probability Distribution[J]. , 2012, 27(5): 1035 -1055 .
[8] Jie Tang, Chen Liu, Shao-Shan Liu, Zhi-Min Gu, and Jean-Luc Gaudiot. Pinned OS/Services: A Case Study of XML Parsing on Intel SCC[J]. , 2013, 28(1): 3 -13 .
[9] Chong Cao, Hai-Zhou Ai. Facial similarity learning with humans in the loop[J]. , 2015, 30(3): 499 -510 .
[10] Huai-Yu Wan, Zhi-Wei Wang You-Fang Lin, Xu-Guang Jia, Yuan-Wei Zhou. Discovering Family Groups in Passenger Social Networks[J]. , 2015, 30(5): 1141 -1153 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

 Home
Editorial Board
Author Guidelines
Subscription		 Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved