计算机科学技术学报 ›› 2020,Vol. 35 ›› Issue (3): 506-521.doi: 10.1007/s11390-020-0264-1

所属专题: Artificial Intelligence and Pattern Recognition Computer Graphics and Multimedia

• Special Section of CVM 2020 • 上一篇    下一篇

一种基于多重残差网络的随机渐进式光子映射的降噪方法

Zheng Zeng1, Lu Wang1,*, Member, CCF, ACM, Bei-Bei Wang2,*, Member, CCF Chun-Meng Kang3, Member, CCF, IEEE, Yan-Ning Xu1, Member, CCF   

  1. 1 School of Software, Shandong University, Jinan 250101, China;
    2 School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China;
    3 School of Information Science and Engineering, Shandong Normal University, Jinan 250358, China
  • 收稿日期:2020-01-02 修回日期:2020-03-24 出版日期:2020-05-28 发布日期:2020-05-28
  • 通讯作者: Lu Wang, Bei-Bei Wang E-mail:luwang_hcivr@sdu.edu.cn;beibei.wang@njust.edu.cn
  • 作者简介:Zheng Zeng received his B.S. degree in digital media technology from Shandong University, Jinan, in 2018. He is currently a Master student in the School of Software at Shandong University, Jinan, supervised by Prof. Lu Wang. His research interests include photorealistic rendering and high-performance rendering.
  • 基金资助:
    This work was partially supported by the National Key Research and Development Program of China under Grant No. 2017YFB0203000, the National Natural Science Foundation of China under Grant Nos. 61802187, 61872223, and 61702311, and the Natural Science Foundation of Jiangsu Province of China under Grant No. BK20170857.

Denoising Stochastic Progressive Photon Mapping Renderings Using a Multi-Residual Network

Zheng Zeng1, Lu Wang1,*, Member, CCF, ACM, Bei-Bei Wang2,*, Member, CCF Chun-Meng Kang3, Member, CCF, IEEE, Yan-Ning Xu1, Member, CCF        

  1. 1 School of Software, Shandong University, Jinan 250101, China;
    2 School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China;
    3 School of Information Science and Engineering, Shandong Normal University, Jinan 250358, China
  • Received:2020-01-02 Revised:2020-03-24 Online:2020-05-28 Published:2020-05-28
  • Contact: Lu Wang, Bei-Bei Wang E-mail:luwang_hcivr@sdu.edu.cn;beibei.wang@njust.edu.cn
  • About author:Zheng Zeng received his B.S. degree in digital media technology from Shandong University, Jinan, in 2018. He is currently a Master student in the School of Software at Shandong University, Jinan, supervised by Prof. Lu Wang. His research interests include photorealistic rendering and high-performance rendering.
  • Supported by:
    This work was partially supported by the National Key Research and Development Program of China under Grant No. 2017YFB0203000, the National Natural Science Foundation of China under Grant Nos. 61802187, 61872223, and 61702311, and the Natural Science Foundation of Jiangsu Province of China under Grant No. BK20170857.

随机渐进式光子映射是一种被广泛使用的全局光照算法,作为光子映射的改进算法,它可以有效地计算焦散等复杂的光照效果。然而,作为一种有偏且一致的算法,随机渐进式光子映射在渲染参数不合适或渲染迭代次数不足时,其渲染结果常常被偏差和方差同时影响,具体表现为物体表面大小各异的光斑及噪声,这有别于其他无偏且一致的算法。最近,深度学习被广泛应用于降噪无偏且一致的基于蒙特卡洛的渲染算法,并有着出色的效果。但此类基于学习的方法还没有被应用于降噪有偏的算法。在本文中,我们提出了第一个基于深度学习的,在图像空间对随机渐进式光子映射这一有偏算法进行降噪的方法。在对随机渐进式光子映射进行降噪时,拥有更大的视野域的降噪网络可以更有效地处理多尺度的光斑和噪声,特别是位于低频表面的较大的光斑,但这类降噪网络往往不善于处理高频区域的噪声。而一个拥有小视野域的降噪网络则恰好相反。基于以上观察,为了更有效地处理随机渐进式光子映射的渲染结果中独特的大小各异的光斑及噪声,我们提出了一种全新的网络结构,它使用了一组特殊的残差格模块。使用这种残差格模块同时集合大视野域和小视野域的优点,可以有效处理随机渐进式光子映射的多尺度噪声。同时,为了进一步提高降噪质量并更好地保持光照细节,我们首先将渲染结果分为全局和焦散两个成分,再输入两个网络分别进行降噪。此外,我们还提出了一组光子相关的辅助特征向量,用于帮助降噪网络更高效地处理噪声,同时有效保持焦散等光照细节。我们使用了若干场景自行制作了大量的训练数据去训练提出的降噪网络。同时,我们还引入了其他几种在蒙特卡洛降噪领域表现优异的方法以作比较,并设计了若干实验验证其模块的有效性。实验结果显示,相比于其他方法,我们提出的方法可以有效处理随机渐进式光子映射中的多尺度噪声,特别是善于处理位于低频区域的较大光斑,同时又可以保留较多光照细节。其相较于其他的无偏算法,随机渐进式光子映射的有偏这一特性,使得其渲染结果中存在多种不同的误差。因此,为其设计降噪算法便颇具挑战性。本文提出了第一种基于深度学习的随机渐进式光子映射的降噪算法,并探索出了一条可行的解决思路。在这个科研领域中,仍有许多方向值得探索,例如本文尚未解决时序连贯性、极大光斑等棘手问题。

关键词: 降噪, 光子映射, 深度学习

Abstract: Stochastic progressive photon mapping (SPPM) is one of the important global illumination methods in computer graphics. It can simulate caustics and specular-diffuse-specular lighting effects efficiently. However, as a biased method, it always suffers from both bias and variance with limited iterations, and the bias and the variance bring multi-scale noises into SPPM renderings. Recent learning-based methods have shown great advantages on denoising unbiased Monte Carlo (MC) methods, but have not been leveraged for biased ones. In this paper, we present the first learning-based method specially designed for denoising-biased SPPM renderings. Firstly, to avoid conflicting denoising constraints, the radiance of final images is decomposed into two components: caustic and global. These two components are then denoised separately via a two-network framework. In each network, we employ a novel multi-residual block with two sizes of filters, which significantly improves the model’s capabilities, and makes it more suitable for multi-scale noises on both low-frequency and high-frequency areas. We also present a series of photon-related auxiliary features, to better handle noises while preserving illumination details, especially caustics. Compared with other state-of-the-art learning-based denoising methods that we apply to this problem, our method shows a higher denoising quality, which could efficiently denoise multi-scale noises while keeping sharp illuminations.

Key words: denoising, stochastic progressive photon mapping (SPPM), deep learning, residual neural network

[1] Hachisuka T, Jensen H W. Stochastic progressive photon mapping. ACM Transactions on Graphics, 2009, 28(5):Article No. 141.
[2] Jensen H W. Realistic Image Synthesis Using Photon Mapping (1st edition). Routledge, 2001.
[3] Hachisuka T, Ogaki S, Jensen H W. Progressive photon mapping. ACM Transactions on Graphics, 2008, 27(5):Article No. 130.
[4] Kang C, Wang L, Xu Y et al. A survey of photon mapping state-of-the-art research and future challenges. Frontiers of Information Technology&Electronic Engineering, 2016, 17(3):185-199.
[5] Ritschel T, Dachsbacher C, Grosch T et al. The state of the art in interactive global illumination. Computer Graphics Forum, 2012, 31(1):160-188.
[6] Bako S, Vogels T, McWilliams B et al. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics, 2017, 36(4):Article No. 97.
[7] Vogels T, Rousselle F, McWilliams B et al. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics, 2018, 37(4):Article No. 124.
[8] Wong K M, Wong T T. Robust deep residual denoising for Monte Carlo rendering. In Proc. the 2018 SIGGRAPH Asia Technical Briefs, December 2018, Article No. 14.
[9] Wong K M, Wong T T. Deep residual learning for denoising Monte Carlo renderings. Computational Visual Media, 2019, 5(3):239-255.
[10] Kalantari N K, Bako S, Sen P. A machine learning approach for filtering Monte Carlo noise. ACM Transactions on Graphics, 2015, 34(4):Article No. 122.
[11] Xu B, Zhang J, Wang R et al. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Transactions on Graphics, 2019, 38(6):Article No. 224.
[12] Yang X, Wang D, Hu W et al. DEMC:A deep dual-encoder network for denoising Monte Carlo rendering. Journal of Computer Science and Technology, 2019, 34(5):1123-1135.
[13] Gharbi M, Li T M, Aittala M et al. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics, 2019, 38(4):Article No. 125.
[14] Günther T, Grosch T. Distributed out-of-core stochastic progressive photon mapping. Computer Graphics Forum, 2014, 33(6):154-166.
[15] Havran V, Bittner J, Herzog R et al. Ray maps for global illumination. Rendering Techniques, 2005, 2005:43-54.
[16] Frisvad J R, Schjøth L, Erleben K et al. Photon differential splatting for rendering caustics. Computer Graphics Forum, 2014, 33(6):252-263.
[17] Spencer B, Jones M W. Into the blue:Better caustics through photon relaxation. Computer Graphics Forum, 2009, 28(2):319-328.
[18] Fu Z, Jensen H W. Noise reduction for progressive photon mapping. In Proc. the International Conference on Computer Graphics and Interactive Techniques, August 2012, Article No. 29.
[19] Kaplanyan A S, Dachsbacher C. Adaptive progressive photon mapping. ACM Transactions on Graphics, 2013, 32(2):Article No. 16.
[20] Bengio Y, Simard P, Frasconi P. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 1994, 5(2):157-166.
[21] He K, Zhang X, Ren S et al. Identity mappings in deep residual networks. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.630-645.
[22] He K, Zhang X, Ren S et al. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.
[23] Zagoruyko S, Komodakis N. Wide residual networks. arXiv:1605.07146, 2016. https://arxiv.org/abs/1605.07146, Feb. 2019.
[24] Abdi M, Nahavandi S. Multi-residual networks:Improving the speed and accuracy of residual networks. arXiv:1609.05672, 2016. https://arxiv.org/abs/1609.05672, Feb. 2019.
[25] Silverman B W. Density Estimation for Statistics and Data Analysis. Chapman and Hall, 1986.
[26] Pharr M, Jakob W, Humphreys G. Physically Based Rendering:From Theory to Implementation (3rd edition). Morgan Kaufmann, 2016.
[27] He K, Zhang X, Ren S et al. Delving deep into rectifiers:Surpassing human-level performance on imageNet classification. In Proc. the IEEE International Conference on Computer Vision, December 2015, pp.1026-1034.
[28] Schjøth L, Sporring J, Olsen F O. Diffusion based photon mapping. Computer Graphics Forum, 2008, 27(8):2114-2127.
[29] Abadi M, Barham P, Chen J et al. TensorFlow:A system for large-scale machine learning. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, November 2016, pp.265-283.
[30] Kingma D P, Ba J. Adam:A method for stochastic optimization. arXiv:1412.6980, 2014. https://arxiv.org/abs/1412.6980, Dec. 2019.
[31] Glorot X, Bengio Y. Understanding the difficulty of training deep feed forward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249-256.
[1] 张鑫, 陆思源, 王水花, 余翔, 王甦菁, 姚仑, 潘毅, 张煜东. 通过新型深度学习架构诊断COVID-19肺炎[J]. 计算机科学技术学报, 2022, 37(2): 330-343.
[2] Songjie Niu, Shimin Chen. TransGPerf:利用迁移学习建模分布式图计算性能[J]. 计算机科学技术学报, 2021, 36(4): 778-791.
[3] Sheng-Luan Hou, Xi-Kun Huang, Chao-Qun Fei, Shu-Han Zhang, Yang-Yang Li, Qi-Lin Sun, Chuan-Qing Wang. 基于深度学习的文本摘要研究综述[J]. 计算机科学技术学报, 2021, 36(3): 633-663.
[4] Lan Chen, Juntao Ye, Xiaopeng Zhang. 基于多特征超分网络的布料褶皱合成[J]. 计算机科学技术学报, 2021, 36(3): 478-493.
[5] Yu-Jie Yuan, Yukun Lai, Tong Wu, Lin Gao, Li-Gang Liu. 回顾形状编辑技术:从几何角度到神经网络方法[J]. 计算机科学技术学报, 2021, 36(3): 520-554.
[6] Wei Du, Yu Sun, Hui-Min Bao, Liang Chen, Ying Li, Yan-Chun Liang. 基于迁移学习与深度学习的人类血液分泌蛋白预测框架[J]. 计算机科学技术学报, 2021, 36(2): 234-247.
[7] Jun Gao, Paul Liu, Guang-Di Liu, Le Zhang. 基于深度学习与波束偏转的穿刺针定位与增强算法[J]. 计算机科学技术学报, 2021, 36(2): 334-346.
[8] Hua Chen, Juan Liu, Qing-Man Wen, Zhi-Qun Zuo, Jia-Sheng Liu, Jing Feng, Bao-Chuan Pang, Di Xiao. CytoBrain:基于深度学习技术的宫颈癌筛查系统[J]. 计算机科学技术学报, 2021, 36(2): 347-360.
[9] Andrea Caroppo, Alessandro Leone, Pietro Siciliano. 用于老年人面部表情识别的深度学习模型和传统机器学习方法的对比研究[J]. 计算机科学技术学报, 2020, 35(5): 1127-1146.
[10] 梁盾, 郭元晨, 张少魁, 穆太江, 黄晓蕾. 车道检测-新结果和调查研究[J]. 计算机科学技术学报, 2020, 35(3): 493-505.
[11] Shuai Li, Zheng Fang, Wen-Feng Song, Ai-Min Hao, Hong Qin. 基于双向特征共享网络的多人姿态估计方法研究[J]. 计算机科学技术学报, 2019, 34(3): 522-536.
[12] Lin Yue, Xiao-Xin Sun, Wen-Zhu Gao, Guo-Zhong Feng, Bang-Zuo Zhang. 基于多重辅助信息及深度模型的协同过滤[J]. , 2018, 33(4): 668-681.
[13] Jin-Hua Tao, Zi-Dong Du, Qi Guo, Hui-Ying Lan, Lei Zhang, Sheng-Yuan Zhou, Ling-. 智能处理器的评测基准[J]. , 2018, 33(1): 1-23.
[14] Fei Hu, Li Li, Zi-Li Zhang, Jing-Yuan Wang, Xiao-Fei Xu. 基于RNN的文本关键字强调模型用于情感分类[J]. , 2017, 32(4): 785-795.
[15] Wei Zhang, Chao-Wei Fang, Guan-Bin Li. 提升空间一致性与边缘定位的图像自动上色方法[J]. , 2017, 32(3): 494-506.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 周笛;. A Recovery Technique for Distributed Communicating Process Systems[J]. , 1986, 1(2): 34 -43 .
[2] 陈世华;. On the Structure of Finite Automata of Which M Is an(Weak)Inverse with Delay τ[J]. , 1986, 1(2): 54 -59 .
[3] 李万学;. Almost Optimal Dynamic 2-3 Trees[J]. , 1986, 1(2): 60 -71 .
[4] 王选; 吕之敏; 汤玉海; 向阳;. A High Resolution Chinese Character Generator[J]. , 1986, 1(2): 1 -14 .
[5] C.Y.Chung; 华宣仁;. A Chinese Information Processing System[J]. , 1986, 1(2): 15 -24 .
[6] 章萃; 赵沁平; 徐家福;. Kernel Language KLND[J]. , 1986, 1(3): 65 -79 .
[7] 王建潮; 魏道政;. An Effective Test Generation Algorithm for Combinational Circuits[J]. , 1986, 1(4): 1 -16 .
[8] 陈肇雄; 高庆狮;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[9] 黄河燕;. A Parallel Implementation Model of HPARLOG[J]. , 1986, 1(4): 27 -38 .
[10] 郑国梁; 李辉;. The Design and Implementation of the Syntax-Directed Editor Generator(SEG)[J]. , 1986, 1(4): 39 -48 .
版权所有 © 《计算机科学技术学报》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持:support@magtech.com.cn
总访问量: