Neural Radiance Fields for 3D Polyp Segmentation of Deformable Tissues in Endoscopy

Recent advances in neural radiance fields (NeRFs) have shown remarkable segmentation performance in three-dimensional (3D) computer vision tasks, demonstrating the ability to segment arbitrary content in static scenes. However, performing segmentation in dynamic scenes, such as soft tissue segmentation remains a challenge for most NeRFs. In this paper, a novel 3D polyp segmentation method with deformable neural radiance fields is proposed. First, we introduce PolypVol-NeRF, a new framework to extend NeRFs with 3D mask prediction capability. It is trained in two stages to reconstruct the dynamic radiance field of soft tissue by modeling color and density. Subsequently, a multiview self-prompt mechanism is used to automatically and accurately predict two-dimensional (2D) polyp masks for the rendered views generated by the PolypVol-NeRF dynamic implicit field. Finally, we introduce a view-alternating weak supervision strategy to train PolypVol-NeRF for the second time using 2D polyp mask supervision aggregated from multiple training views. This enables a dynamic 3D mask implicit field to be obtained without requiring any ground-truth volumetric annotations. The experimental results demonstrate that the proposed method achieves more comprehensive reconstruction and more accurate segmentation of the lesion region than the state-of-the-art 3D segmentation methods, resulting in the best overall performance. User study feedback shows that the proposed method has received high ratings from qualified participants in terms of segmentation accuracy, visual quality and clinical applicability.