Journal of Computer Science and Technology ›› 2019, Vol. 34 ›› Issue (5): 1123-1135.doi: 10.1007/s11390-019-1964-2

Special Issue: Computer Graphics and Multimedia

• Computer Graphics and Multimedia • Previous Articles     Next Articles

DEMC: A Deep Dual-Encoder Network for Denoising Monte Carlo Rendering

Xin Yang1, Dawei Wang2, Wenbo Hu3, Li-Jing Zhao1, Bao-Cai Yin1, Qiang Zhang1, Xiao-Peng Wei1,*, Hongbo Fu4   

  1. 1 Department of Computer Science and Technology, Dalian University of Technology, Dalian 116024, China;
    2 Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong, China;
    3 Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong, China;
    4 School of Creative Media, City University of Hong Kong, Kowloon, Hong Kong, China
  • Received:2019-01-15 Revised:2019-05-28 Online:2019-08-31 Published:2019-08-31
  • Contact: Xiao-Peng Wei
  • About author:Xin Yang is an associate professor in the Department of Computer Science and Technology at Dalian University of Technology, Dalian. Xin received his B.S. degree in computer science from Jilin University, Changchun, in 2007. From 2007 to June 2012, he was a joint Ph.D. student in Zhejiang University, Hangzhou, and Davis University of California for Graphics, and received his Ph.D. degree in computer science from Zhejiang University, Hangzhou, in 2012. His research interests include computer graphics and robotic vision.
  • Supported by:
    This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 91748104, U1811463, 61632006, 61425002, and 61751203, the National Key Research and Development Program of China under Grant No. 2018YFC0910506, the Open Project Program of the State Key Laboratory of CAD&CG of Zhejiang University of China under Grant No. A1901, and the Open Research Fund of Beijing Key Laboratory of Big Data Technology for Food Safety Project under Grant No. BTBD-2018KF.

In this paper, we present DEMC, a deep dual-encoder network to remove Monte Carlo noise efficiently while preserving details. Denoising Monte Carlo rendering is different from natural image denoising since inexpensive by-products (feature buffers) can be extracted in the rendering stage. Most of them are noise-free and can provide sufficient details for image reconstruction. However, these feature buffers also contain redundant information. Hence, the main challenge of this topic is how to extract useful information and reconstruct clean images. To address this problem, we propose a novel network structure, dual-encoder network with a feature fusion sub-network, to fuse feature buffers firstly, then encode the fused feature buffers and a noisy image simultaneously, and finally reconstruct a clean image by a decoder network. Compared with the state-of-the-art methods, our model is more robust on a wide range of scenes, and is able to generate satisfactory results in a significantly faster way.

Key words: Monte Carlo rendering, Monte Carlo denoising, neural network

[1] Kajiya J T. The rendering equation. In Proc. the 13th Annual Conference on Computer Graphics and Interactive Techniques, August 1986, pp.143-150.
[2] Rousselle F, Manzi M, Zwicker M. Robust denoising using feature and color information. Computer Graphics Forum, 2013, 32(7):121-130.
[3] Kalantari N K, Bako S, Sen P. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph., 2015, 34(4):Article No. 122.
[4] Bitterli B, Rousselle F, Moon B, Guitián J A, Adler D, Mitchell K, Jarosz W, Novák J. Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. Computer Graphics Forum, 2016, 35(4):107-117.
[5] Bako S, Vogels T, McWilliams B, Meyer M, Novák J, Harvill A, Sen P, Derose T, Rousselle F. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph., 2017, 36(4):Article No. 97.
[6] Moon B, Carr N, Yoon S E. Adaptive rendering based on weighted local regression. ACM Transactions on Graphics, 2017, 33(5):Article No. 170.
[7] Cook R L, Porter T, Carpenter L. Distributed ray tracing. In Proc. the 11th Annual Conference on Computer Graphics and Interactive Techniques, July 1984, pp.137-145.
[8] Zwicker M, Jarosz W, Lehtinen J, Moon B, Ramamoorthi R, Rousselle F, Sen P, Soler C, Yoon S E. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Computer Graphics Forum, 2015, 34(2):667-681.
[9] Ramamoorthi R, Mahajan D, Belhumeur P. A first-order analysis of lighting, shading, and shadows. ACM Transactions on Graphics, 2007, 26(1):Article No. 2.
[10] Jarosz W, Schönefeld V, Kobbelt L, Jensen H W. Theory, analysis and applications of 2D global illumination. ACM Transactions on Graphics, 2012, 31(5):Article No. 125.
[11] Bauszat P, Eisemann M, Magnor M. Guided image filtering for interactive high-quality global illumination. Computer Graphics Forum, 2011, 30(4):1361-1368.
[12] Bauszat P, Eisemann M, Eisemann E, Magnor M. General and robust error estimation and reconstruction for Monte Carlo rendering. Computer Graphics Forum, 2015, 34(2):597-608.
[13] Yang X, Wang D, Hu W, Zhao L, Piao X, Zhou D, Zhang Q, Yin B, Cai Q, Wei X. Fast reconstruction for Monte Carlo rendering using deep convolutional networks. IEEE Access, 2019, 7:21177-21187.
[14] Chaitanya C R A, Kaplanyan A S, Schied C, Salvi M, Lefohn A, Nowrouzezahrai D, Aila T. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics, 2017, 36(4):Article No. 98.
[15] Zhang K, Zuo W, Chen Y, Meng D, Zhang L. Beyond a Gaussian denoiser:Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 2017, 26(7):3142-3155.
[16] Gharbi M, Chaurasia G, Paris S, Durand F. Deep joint demosaicking and denoising. ACM Transactions on Graphics, 2016, 35(6):Article No. 191.
[17] Mao X, Shen C, Yang Y B. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Proc. the 2016 Annual Conference on Neural Information Processing Systems, December 2016, pp.2802-2810.
[18] Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2016, 313(5786):504-507.
[19] Balduzzi D, McWilliams B, Butler-Yeoman T. Neural Taylor approximations:Convergence and exploration in rectifier networks. In Proc. the 34th International Conference on Machine Learning, August 2017, pp.351-360.
[20] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.
[21] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment:From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4):600-612.
[22] Pharr M, Jakob W, Humphreys G. Physically Based Rendering:From Theory to Implementation (3rd edition). Morgan Kaufmann, 2016.
[23] Eilertsen G, Kronander J, Denes G, Mantiuk R K, Unger J. HDR image reconstruction from a single exposure using deep CNNs. ACM Transactions on Graphics, 2107, 36(6):Article No. 178.
[24] Rousselle F, Knaus C, Zwicker. Adaptive sampling and reconstruction using greedy error minimization. ACM Transactions on Graphics, 2011, 30(6):Article No. 159.
[25] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249-256
[26] Abadi M, Barham P, Chen J et al. Tensorflow:A system for large-scale machine learning. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, November 2016, pp.265-283.
[27] Kingma D P, Ba J. Adam:A method for stochastic optimization. arXiv:1412.6980, 2014., May 2019.
[28] Boughida M, Boubekeur T. Bayesian collaborative denoising for Monte Carlo rendering. Computer Graphics Forum, 2017, 36(4):137-153.
[29] Lang M, Wang O, Aydin T, Smolic A, Gross M. Practical temporal consistency for image-based graphics applications. ACM Transactions on Graphics, 2012, 31(4):Article No. 34.
[30] Bonneel N, Tompkin J, Sunkavalli K, Sun D, Paris S, Pfister H. Blind video temporal consistency. ACM Transactions on Graphics, 2015, 34(6):Article No. 196.
[31] Lai W S, Huang J B, Wang O, Shechtman E, Yumer E, Yang M H. Learning blind video temporal consistency. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.179-185.
[1] Yun-Yun Wang, Jian-Min Gu, Chao Wang, Song-Can Chen, Hui Xue. Discrimination-Aware Domain Adversarial Neural Network [J]. Journal of Computer Science and Technology, 2020, 35(2): 259-267.
[2] Shu-Quan Wang, Lei Wang, Yu Deng, Zhi-Jie Yang, Sha-Sha Guo, Zi-Yang Kang, Yu-Feng Guo, Wei-Xia Xu. SIES: A Novel Implementation of Spiking Convolutional Neural Network Inference Engine on Field-Programmable Gate Array [J]. Journal of Computer Science and Technology, 2020, 35(2): 475-489.
[3] Sa Wang, Yan-Hai Zhu, Shan-Pei Chen, Tian-Ze Wu, Wen-Jie Li, Xu-Sheng Zhan, Hai-Yang Ding, Wei-Song Shi, Yun-Gang Bao. A Case for Adaptive Resource Management in Alibaba Datacenter Using Neural Networks [J]. Journal of Computer Science and Technology, 2020, 35(1): 209-220.
[4] Xing-Gang Wang, Jia-Si Wang, Peng Tang, Wen-Yu Liu. Weakly- and Semi-Supervised Fast Region-Based CNN for Object Detection [J]. Journal of Computer Science and Technology, 2019, 34(6): 1269-1278.
[5] Robail Yasrab. SRNET: A Shallow Skip Connection Based Convolutional Neural Network Design for Resolving Singularities [J]. Journal of Computer Science and Technology, 2019, 34(4): 924-938.
[6] Ri-Sheng Liu, Cai-Sheng Mao, Zhi-Hui Wang, Hao-Jie Li. Blind Image Deblurring via Adaptive Optimization with Flexible Sparse Structure Control [J]. Journal of Computer Science and Technology, 2019, 34(3): 609-621.
[7] Han Liu, Hang Du, Dan Zeng, Qi Tian. Cloud Detection Using Super Pixel Classification and Semantic Segmentation [J]. Journal of Computer Science and Technology, 2019, 34(3): 622-633.
[8] Dong-Di Zhao, Fan Li, Kashif Sharif, Guang-Min Xia, Yu Wang. Space Efficient Quantization for Deep Convolutional Neural Networks [J]. Journal of Computer Science and Technology, 2019, 34(2): 305-317.
[9] Feng Zhou, Hao-Min Zhou, Zhi-Hua Yang, Li-Hua Yang. A 2-Stage Strategy for Non-Stationary Signal Prediction and Recovery Using Iterative Filtering and Neural Network [J]. Journal of Computer Science and Technology, 2019, 34(2): 318-338.
[10] Tie-Ke He, Hao Lian, Ze-Min Qin, Zhen-Yu Chen, Bin Luo. PTM: A Topic Model for the Inferring of the Penalty [J]. , 2018, 33(4): 756-767.
[11] Bei-Ji Zou, Yun-Di Guo, Qi He, Ping-Bo Ouyang, Ke Liu, Zai-Liang Chen. 3D Filtering by Block Matching and Convolutional Neural Network for Image Denoising [J]. , 2018, 33(4): 838-848.
[12] Nai-Ming Yao, Hui Chen, Qing-Pei Guo, Hong-An Wang. Non-Frontal Facial Expression Recognition Using a Depth-Patch Based Deep Neural Network [J]. , 2017, 32(6): 1172-1185.
[13] Wei-Qing, Liu Jing Li. An Approach to Automatic Performance Prediction for Cloud-enhanced Mobile Applications with Sparse Data [J]. , 2017, 32(5): 936-956.
[14] Shu-Chang Zhou, Yu-Zhi Wang, He Wen, Qin-Yao He, Yu-Heng Zou. Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks [J]. , 2017, 32(4): 667-682.
[15] Xiang Bai, Zheng Zhang, Hong-Yang Wang, Wei Shen. Directional Edge Boxes: Exploiting Inner Normal Direction Cues for Effective Object Proposal Generation [J]. , 2017, 32(4): 701-713.
Full text



[1] Li Wei;. A Structural Operational Semantics for an Edison Like Language(2)[J]. , 1986, 1(2): 42 -53 .
[2] Sun Zhongxiu; Shang Lujun;. DMODULA:A Distributed Programming Language[J]. , 1986, 1(2): 25 -31 .
[3] Jin Lan; Yang Yuanyuan;. A Modified Version of Chordal Ring[J]. , 1986, 1(3): 15 -32 .
[4] Min Yinghua; Han Zhide;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[5] Huang Xuedong; Cai Lianhong; Fang Ditang; Chi Bianjin; Zhou Li; Jiang Li;. A Computer System for Chinese Character Speech Input[J]. , 1986, 1(4): 75 -83 .
[6] Xu Xiaoshu;. Simplification of Multivalued Sequential SULM Network by Using Cascade Decomposition[J]. , 1986, 1(4): 84 -95 .
[7] Shi Zhongzhi;. Knowledge-Based Decision Support System[J]. , 1987, 2(1): 22 -29 .
[8] Wu Yunzeng;. On the Development of Applications of Logic in Programming[J]. , 1987, 2(1): 30 -34 .
[9] Tang Tonggao; Zhao Zhaokeng;. Stack Method in Program Semantics[J]. , 1987, 2(1): 51 -63 .
[10] Zhong Renbao; Xing Lin; Ren Zhaoyang;. An Interactive System SDI on Microcomputer[J]. , 1987, 2(1): 64 -71 .

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
  Copyright ©2015 JCST, All Rights Reserved