Abstract: Cloud detection plays a very significant role in remote sensing image processing. This paper introduces a cloud detection method based on super pixel level classification and semantic segmentation. Firstly, remote sensing images are segmented into super pixels. Segmented super pixels compose a super pixel level remote sensing image database. Though cloud detection is essentially a binary classification task, our database is labeled into four categories to improve the generalization ability:thick cloud, cirrus cloud, building, and other culture. Secondly, the super pixel level database is used to train our cloud detection models based on convolution neural network (CNN) and deep forest. Hierarchical fusion CNN is proposed considering super pixel level images contain less semantic information than normal images. Taking full advantage of low-level features like color and texture information, it is more applicable for super pixel level classification. Besides, a distance metric is proposed to refine ambiguous super pixels. Thirdly, an end-to-end cloud detection model based on semantic segmentation is introduced. This model has no restrictions on the input size, and takes less time. Experimental results show that compared with other cloud detection methods, our proposed method achieves better performance.