Abstract: In this paper, we propose a method that can extract enhanced color palettes for images, which are characterized by a heightened level of representativeness for images. The resulted palettes are not only easy to compute but also effectively convey the distribution of pixels across the color space. Based on these extracted palettes, we present a tailored color palette matching algorithm designed for our image color transfer method by solving an optimization problem to transfer colors from a reference image to the original image. Our algorithm offers the flexibility to operate in fully automatic mode or provide various levels of user interactivity, allowing for coarse-to-fine editing. Moreover, we demonstrate the adaptability of our palette-based color transfer method to diverse applications, including grayscale image colorization and temporally consistent, time-varying video color transfer. Extensive experiments and comparisons have been conducted to measure the quality of our results, employing both visual assessments and evaluation metrics. These findings demonstrate that our palette-based color transfer method efficiently and faithfully transfers color styles from reference images to both color and grayscale images and videos.