Abstract: Image smoothing is a crucial image processing topic in image processing and has wide applied backgrounds. For images with rich texture, most of the existing image smoothing methods are difficult to obtain significant texture removal performance in many situations because textures containing obvious edges and large gradient changes are easy to be preserved as the main edges. In this paper, we propose a novel framework for image smoothing combined with the constraint of sparse high frequency gradient for texture image. First, we decompose the image into two components:a smooth component (constant component) and a non-smooth (high frequency) component. Second, we remove the non-smooth component containing high frequency gradient and smooth the other component combining with the constraint of sparse high frequency gradient. Experimental results demonstrate the proposed method is more competitive on efficiently texture removing than the state-of-the-art. What is more, our approach has a variety of applications including edge detection, detail magnification, image abstraction and image composition.