Unsupervised Adversarial Domain Adaptation with Hierarchical Semantic Consistency for Cross-modal Nuclei Detection

Deep learning based methods have demonstrated outstanding capabilities in quantifying nuclei and cells in microscopy images. However, differences among various stain modalities would affect the performance of nuclei detection. How to fully utilize limited annotations for nuclei detection in other pathological staining images without annotations has become a significant challenge. This paper proposes an end-to-end unsupervised multi-level semantic consistent generative adversarial network (MSC-GAN) for nuclei detection across different pathological staining modalities. Specifically, we address nuclei detection on the unlabeled target domain data by first transforming the stain modality of the source domain into the target domain, and then utilizing the source domain annotations to train the nuclei detector network. A hierarchical semantic consistency loss including feature-level consistency and mask-level consistency is introduced to offer supplementary supervision to enhance the accuracy of generative adversarial learning. We further design an augmentation module to prevent the discriminator from overfitting. The experimental results on four microscopy image datasets demonstrate that MSC-GAN outperforms state-of-the-art methods in the nuclei detection tasks, achieving superior F1 scores.