Color has great diagnostic significance in dermatoscopy. Several diagnosis methods are based on the colors detected within a lesion. Malignant lesions frequently show more than three colors, whereas in benign lesions, three or fewer colors are usually observed. Black, red, white, and blue-gray are found more frequently in melanomas than in benign nevi. In this paper, a method to automatically identify the colors of a lesion is presented. A color label identification problem is proposed and solved by maximizing the posterior probability of a pixel to belong to a label, given its color value and the neighborhood color values. The main contribution of this paper is the estimation of the different terms involved in the computation of this probability. Two evaluations are performed on a database of 200 dermoscopic images. The first one evaluates if all the colors detected in a lesion are indeed present in it. The second analyzes if each pixel within a lesion is assigned the correct color label. The results show that the proposed method performs correctly and outperforms other methods, with an average F-measure of 0.89, an accuracy of 0.90, and a Spearman correlation of 0.831.