? Temporally Consistent Depth Map Prediction Using Deep CNN and Spatial-temporal Conditional Random Field
Journal of Computer Science and Technology
Quick Search in JCST
 Advanced Search 
      Home | PrePrint | SiteMap | Contact Us | FAQ
 
Indexed by   SCIE, EI ...
Bimonthly    Since 1986
Journal of Computer Science and Technology 2017, Vol. 32 Issue (3) :443-456    DOI: 10.1007/s11390-017-1735-x
Special Section of CVM 2017 Current Issue | Archive | Adv Search << Previous Articles | Next Articles >>
Temporally Consistent Depth Map Prediction Using Deep CNN and Spatial-temporal Conditional Random Field
Xu-Ran Zhao, Xun Wang*, Senior Member, CCF, Member, ACM, IEEE, Qi-Chao Chen
School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou 310018, China

Abstract
Reference
Related Articles
Download: [PDF 3384KB]     Export: BibTeX or EndNote (RIS)  
Abstract Deep convolutional neural networks (DCNN) based methods recently keep setting new records on tasks of predicting depth maps from monocular images. When dealing with video-based applications such as 2D to 3D video conversion, however, these approaches tend to produce temporally inconsistent depth maps, since their CNN models are optimized over single frames. In this paper, we address this problem by introducing a novel spatial-temporal Conditional Random Fields (CRF) model into the DCNN architecture, which is able to enforce temporal consistency between depth map estimations over consecutive video frames. In our approach, temporally consistent superpixel (TSP) is first applied to an image sequence to establish correspondence of targets in consecutive frames. A DCNN network is then used to regress the depth value of each temporal superpixel, followed by a spatial-temporal CRF layer to model the relationship of the estimated depths in both spatial and temporal domain. The parameters in both DCNN and CRF models are jointly optimized with back propagation. Experimental results show that our approach not only is able to significantly enhance the temporal consistency of estimated depth maps over existing single-frame-based approaches, but also improves the depth estimation accuracy in terms of various evaluation metrics.
Articles by authors
Xu-Ran Zhao
Xun Wang
Qi-Chao Chen
Keywordsdepth estimation   temporal consistency   convolutional neural networks   conditional random fields     
Received 2016-12-23;
Fund:

This work is supported in part by the Natural Science Foundation of Zhejiang Province of China under Grant No.LQ17F030001,the National Natural Science Foundation of China under Grant No.U1609215,Qianjiang Talent Program of Zhejiang Province of China under Grant No.QJD1602021,the National Key Technology Research and Development Program of the Ministry of Science and Technology of China under Grant No.2014BAK14B01,and Beihang University Virtual Reality Technology and System National Key Laboratory Open Project under Grant No.BUAA-VR-16KF-17.

Corresponding Authors: Xun Wang     Email: wx@zjgsu.edu.cn
About author: Xu-Ran Zhao is currently an assistant professor at the School of Computer Science and Information Engineering, Zhejiang Gongshang University, Hangzhou. He received his B.S. degree in electronic and information technologies from Shanghai University, Shanghai, and M.S. degree in electrical and computer engineering from Georgia Institute of Technology, Atlanta, in 2006 and 2010 respectively. He received his Ph.D. degree from Telecom ParisTech, Paris, in 2013. During 2014~2016, he worked as a postdoctoral researcher on machine learning in School of Computer Science at Aalto University, Helsinki. His current research interests include pattern recognition, computer vision and biometric recognition.
Cite this article:   
Xu-Ran Zhao, Xun Wang, Qi-Chao Chen.Temporally Consistent Depth Map Prediction Using Deep CNN and Spatial-temporal Conditional Random Field[J]  Journal of Computer Science and Technology, 2017,V32(3): 443-456
URL:  
http://jcst.ict.ac.cn:8080/jcst/EN/10.1007/s11390-017-1735-x
Copyright 2010 by Journal of Computer Science and Technology