Journal of Computer Science and Technology ›› 2019, Vol. 34 ›› Issue (5): 1123-1135.doi: 10.1007/s11390-019-1964-2

Special Issue: Computer Graphics and Multimedia

• Computer Graphics and Multimedia • Previous Articles     Next Articles

DEMC: A Deep Dual-Encoder Network for Denoising Monte Carlo Rendering

Xin Yang1, Dawei Wang2, Wenbo Hu3, Li-Jing Zhao1, Bao-Cai Yin1, Qiang Zhang1, Xiao-Peng Wei1,*, Hongbo Fu4   

  1. 1 Department of Computer Science and Technology, Dalian University of Technology, Dalian 116024, China;
    2 Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong, China;
    3 Department of Computer Science and Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong, China;
    4 School of Creative Media, City University of Hong Kong, Kowloon, Hong Kong, China
  • Received:2019-01-15 Revised:2019-05-28 Online:2019-08-31 Published:2019-08-31
  • Contact: Xiao-Peng Wei
  • About author:Xin Yang is an associate professor in the Department of Computer Science and Technology at Dalian University of Technology, Dalian. Xin received his B.S. degree in computer science from Jilin University, Changchun, in 2007. From 2007 to June 2012, he was a joint Ph.D. student in Zhejiang University, Hangzhou, and Davis University of California for Graphics, and received his Ph.D. degree in computer science from Zhejiang University, Hangzhou, in 2012. His research interests include computer graphics and robotic vision.
  • Supported by:
    This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 91748104, U1811463, 61632006, 61425002, and 61751203, the National Key Research and Development Program of China under Grant No. 2018YFC0910506, the Open Project Program of the State Key Laboratory of CAD&CG of Zhejiang University of China under Grant No. A1901, and the Open Research Fund of Beijing Key Laboratory of Big Data Technology for Food Safety Project under Grant No. BTBD-2018KF.

In this paper, we present DEMC, a deep dual-encoder network to remove Monte Carlo noise efficiently while preserving details. Denoising Monte Carlo rendering is different from natural image denoising since inexpensive by-products (feature buffers) can be extracted in the rendering stage. Most of them are noise-free and can provide sufficient details for image reconstruction. However, these feature buffers also contain redundant information. Hence, the main challenge of this topic is how to extract useful information and reconstruct clean images. To address this problem, we propose a novel network structure, dual-encoder network with a feature fusion sub-network, to fuse feature buffers firstly, then encode the fused feature buffers and a noisy image simultaneously, and finally reconstruct a clean image by a decoder network. Compared with the state-of-the-art methods, our model is more robust on a wide range of scenes, and is able to generate satisfactory results in a significantly faster way.

Key words: Monte Carlo rendering; Monte Carlo denoising; neural network;

[1] Kajiya J T. The rendering equation. In Proc. the 13th Annual Conference on Computer Graphics and Interactive Techniques, August 1986, pp.143-150.
[2] Rousselle F, Manzi M, Zwicker M. Robust denoising using feature and color information. Computer Graphics Forum, 2013, 32(7):121-130.
[3] Kalantari N K, Bako S, Sen P. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph., 2015, 34(4):Article No. 122.
[4] Bitterli B, Rousselle F, Moon B, Guitián J A, Adler D, Mitchell K, Jarosz W, Novák J. Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. Computer Graphics Forum, 2016, 35(4):107-117.
[5] Bako S, Vogels T, McWilliams B, Meyer M, Novák J, Harvill A, Sen P, Derose T, Rousselle F. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph., 2017, 36(4):Article No. 97.
[6] Moon B, Carr N, Yoon S E. Adaptive rendering based on weighted local regression. ACM Transactions on Graphics, 2017, 33(5):Article No. 170.
[7] Cook R L, Porter T, Carpenter L. Distributed ray tracing. In Proc. the 11th Annual Conference on Computer Graphics and Interactive Techniques, July 1984, pp.137-145.
[8] Zwicker M, Jarosz W, Lehtinen J, Moon B, Ramamoorthi R, Rousselle F, Sen P, Soler C, Yoon S E. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Computer Graphics Forum, 2015, 34(2):667-681.
[9] Ramamoorthi R, Mahajan D, Belhumeur P. A first-order analysis of lighting, shading, and shadows. ACM Transactions on Graphics, 2007, 26(1):Article No. 2.
[10] Jarosz W, Schönefeld V, Kobbelt L, Jensen H W. Theory, analysis and applications of 2D global illumination. ACM Transactions on Graphics, 2012, 31(5):Article No. 125.
[11] Bauszat P, Eisemann M, Magnor M. Guided image filtering for interactive high-quality global illumination. Computer Graphics Forum, 2011, 30(4):1361-1368.
[12] Bauszat P, Eisemann M, Eisemann E, Magnor M. General and robust error estimation and reconstruction for Monte Carlo rendering. Computer Graphics Forum, 2015, 34(2):597-608.
[13] Yang X, Wang D, Hu W, Zhao L, Piao X, Zhou D, Zhang Q, Yin B, Cai Q, Wei X. Fast reconstruction for Monte Carlo rendering using deep convolutional networks. IEEE Access, 2019, 7:21177-21187.
[14] Chaitanya C R A, Kaplanyan A S, Schied C, Salvi M, Lefohn A, Nowrouzezahrai D, Aila T. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics, 2017, 36(4):Article No. 98.
[15] Zhang K, Zuo W, Chen Y, Meng D, Zhang L. Beyond a Gaussian denoiser:Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 2017, 26(7):3142-3155.
[16] Gharbi M, Chaurasia G, Paris S, Durand F. Deep joint demosaicking and denoising. ACM Transactions on Graphics, 2016, 35(6):Article No. 191.
[17] Mao X, Shen C, Yang Y B. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Proc. the 2016 Annual Conference on Neural Information Processing Systems, December 2016, pp.2802-2810.
[18] Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2016, 313(5786):504-507.
[19] Balduzzi D, McWilliams B, Butler-Yeoman T. Neural Taylor approximations:Convergence and exploration in rectifier networks. In Proc. the 34th International Conference on Machine Learning, August 2017, pp.351-360.
[20] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.
[21] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment:From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4):600-612.
[22] Pharr M, Jakob W, Humphreys G. Physically Based Rendering:From Theory to Implementation (3rd edition). Morgan Kaufmann, 2016.
[23] Eilertsen G, Kronander J, Denes G, Mantiuk R K, Unger J. HDR image reconstruction from a single exposure using deep CNNs. ACM Transactions on Graphics, 2107, 36(6):Article No. 178.
[24] Rousselle F, Knaus C, Zwicker. Adaptive sampling and reconstruction using greedy error minimization. ACM Transactions on Graphics, 2011, 30(6):Article No. 159.
[25] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249-256
[26] Abadi M, Barham P, Chen J et al. Tensorflow:A system for large-scale machine learning. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, November 2016, pp.265-283.
[27] Kingma D P, Ba J. Adam:A method for stochastic optimization. arXiv:1412.6980, 2014., May 2019.
[28] Boughida M, Boubekeur T. Bayesian collaborative denoising for Monte Carlo rendering. Computer Graphics Forum, 2017, 36(4):137-153.
[29] Lang M, Wang O, Aydin T, Smolic A, Gross M. Practical temporal consistency for image-based graphics applications. ACM Transactions on Graphics, 2012, 31(4):Article No. 34.
[30] Bonneel N, Tompkin J, Sunkavalli K, Sun D, Paris S, Pfister H. Blind video temporal consistency. ACM Transactions on Graphics, 2015, 34(6):Article No. 196.
[31] Lai W S, Huang J B, Wang O, Shechtman E, Yumer E, Yang M H. Learning blind video temporal consistency. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.179-185.
[1] Xiao-Zheng Xie, Jian-Wei Niu, Xue-Feng Liu, Qing-Feng Li, Yong Wang, Jie Han, and Shaojie Tang. DG-CNN: Introducing Margin Information into Convolutional Neural Networks for Breast Cancer Diagnosis in Ultrasound Images [J]. Journal of Computer Science and Technology, 2022, 37(2): 277-294.
[2] Xin-Feng Wang, Xiang Zhou, Jia-Hua Rao, Zhu-Jin Zhang, and Yue-Dong Yang. Imputing DNA Methylation by Transferred Learning Based Neural Network [J]. Journal of Computer Science and Technology, 2022, 37(2): 320-329.
[3] Xin Zhang, Siyuan Lu, Shui-Hua Wang, Xiang Yu, Su-Jing Wang, Lun Yao, Yi Pan, and Yu-Dong Zhang. Diagnosis of COVID-19 Pneumonia via a Novel Deep Learning Architecture [J]. Journal of Computer Science and Technology, 2022, 37(2): 330-343.
[4] Dan-Hao Zhu, Xin-Yu Dai, Jia-Jun Chen. Pre-Train and Learn: Preserving Global Information for Graph Neural Networks [J]. Journal of Computer Science and Technology, 2021, 36(6): 1420-1430.
[5] Yi Zhong, Jian-Hua Feng, Xiao-Xin Cui, Xiao-Le Cui. Machine Learning Aided Key-Guessing Attack Paradigm Against Logic Block Encryption [J]. Journal of Computer Science and Technology, 2021, 36(5): 1102-1117.
[6] Feng Wang, Guo-Jie Luo, Guang-Yu Sun, Yu-Hao Wang, Di-Min Niu, Hong-Zhong Zheng. Area Efficient Pattern Representation of Binary Neural Networks on RRAM [J]. Journal of Computer Science and Technology, 2021, 36(5): 1155-1166.
[7] Shao-Jie Qiao, Guo-Ping Yang, Nan Han, Hao Chen, Fa-Liang Huang, Kun Yue, Yu-Gen Yi, Chang-An Yuan. Cardinality Estimator: Processing SQL with a Vertical Scanning Convolutional Neural Network [J]. Journal of Computer Science and Technology, 2021, 36(4): 762-777.
[8] Chen-Chen Sun, De-Rong Shen. Mixed Hierarchical Networks for Deep Entity Matching [J]. Journal of Computer Science and Technology, 2021, 36(4): 822-838.
[9] Yang Liu, Ruili He, Xiaoqian Lv, Wei Wang, Xin Sun, Shengping Zhang. Is It Easy to Recognize Baby's Age and Gender? [J]. Journal of Computer Science and Technology, 2021, 36(3): 508-519.
[10] Yang-Jie Cao, Shuang Wu, Chang Liu, Nan Lin, Yuan Wang, Cong Yang, Jie Li. Seg-CapNet: A Capsule-Based Neural Network for the Segmentation of Left Ventricle from Cardiac Magnetic Resonance Imaging [J]. Journal of Computer Science and Technology, 2021, 36(2): 323-333.
[11] Zhang-Jin Huang, Xiang-Xiang He, Fang-Jun Wang, Qing Shen. A Real-Time Multi-Stage Architecture for Pose Estimation of Zebrafish Head with Convolutional Neural Networks [J]. Journal of Computer Science and Technology, 2021, 36(2): 434-444.
[12] Bo-Wei Zou, Rong-Tao Huang, Zeng-Zhuang Xu, Yu Hong, Guo-Dong Zhou. Language Adaptation for Entity Relation Classification via Adversarial Neural Networks [J]. Journal of Computer Science and Technology, 2021, 36(1): 207-220.
[13] Yue-Huan Wang, Ze-Nan Li, Jing-Wei Xu, Ping Yu, Taolue Chen, Xiao-Xing Ma. Predicted Robustness as QoS for Deep Neural Network Models [J]. Journal of Computer Science and Technology, 2020, 35(5): 999-1015.
[14] Bi-Ying Yan, Chao Yang, Pan Deng, Qiao Sun, Feng Chen, Yang Yu. A Spatiotemporal Causality Based Governance Framework for Noisy Urban Sensory Data [J]. Journal of Computer Science and Technology, 2020, 35(5): 1084-1098.
[15] Dun Liang, Yuan-Chen Guo, Shao-Kui Zhang, Tai-Jiang Mu, Xiaolei Huang. Lane Detection: A Survey with New Results [J]. Journal of Computer Science and Technology, 2020, 35(3): 493-505.
Full text



[1] Liu Mingye; Hong Enyu;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[2] Chen Shihua;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[3] Gao Qingshi; Zhang Xiang; Yang Shufan; Chen Shuqing;. Vector Computer 757[J]. , 1986, 1(3): 1 -14 .
[4] Chen Zhaoxiong; Gao Qingshi;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[5] Huang Heyan;. A Parallel Implementation Model of HPARLOG[J]. , 1986, 1(4): 27 -38 .
[6] Min Yinghua; Han Zhide;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[7] Tang Tonggao; Zhao Zhaokeng;. Stack Method in Program Semantics[J]. , 1987, 2(1): 51 -63 .
[8] Min Yinghua;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[9] Zhu Hong;. Some Mathematical Properties of the Functional Programming Language FP[J]. , 1987, 2(3): 202 -216 .
[10] Li Minghui;. CAD System of Microprogrammed Digital Systems[J]. , 1987, 2(3): 226 -235 .

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
  Copyright ©2015 JCST, All Rights Reserved