Cultura e Scienza del Colore - Color Culture and Science 4904/15 | Cultura e Scienza del Colore - Color Culture and Science are lit by different illuminants. In the framework of digital imaging and computer vision, the illuminant estimation and correction is also referred to as white balance or computational color constancy. The second stage of the color correction pipeline transforms the image data into a standard color space. This transformation, usually called color space transformation or color matrixing, is needed because the spectral sensitivity functions of the sensor color channels rarely match those of the desired output color space. This transformation is usually performed by using a linear transformation matrix. Many illuminant estimation solutions have been proposed in the last few years, although it is known that the problem addressed is actually ill-posed as its solution lacks uniqueness and 1. INTRODUCTION There are mainly two modules responsible for the color rendering accuracy in a digital camera: the former is the illuminant estimation and correction module, the latter is the color matrix transformation. These two modules together form what may be called the color correction pipeline, or color engine. A simplified processing pipeline of a typical digital camera is reported in Fig. 1, where the color engine is highlighted. The first stage of the color correction pipeline aims to render the acquired image as it was acquired under a known canonical illuminant. This is inspired by the color constancy feature of the human visual system (HVS), i.e. the ability of perceiving relatively constant colors when objects 1Simone Bianco simone.bianco@disco.unimib.it 1Raimondo Schettini schettini@disco.unimib.it 1DISCo - Dipartimento di Informatica, Sistemistica e Comunicazione Università degli Studi di Milano-Bicocca Adaptive Illuminant Estimation and Correction for Digital Photography ABSTRACT In this paper we briefly review our recent research on classification-based color constancy, where automatically extracted features are used to drive the selection and combination of the best algorithm(s) for each image We also describe how the problem of illuminant estimation and correction is deeply intertwined with the one of color space transformation. Finally, we also highlight research trends in these fields. Figure 1 – Simplified processing pipeline of a typical digital camera 50 Cultura e Scienza del Colore - Color Culture and Science | 04/15 stability. To cope with this problem, different solutions usually exploit some assumptions about the statistical properties of the expected illuminants and/or of the object reflectances in the scene. In this paper we briefly review state of the art methods and our recent research on classification-based color constancy, where automatically extracted features are used to drive the selection and combination of the best algorithm(s) for each image. Then, we describe how the problem of illuminant estimation and correction is deeply intertwined with the one of color space transformation. And, finally, we highlight research trends in these fields. 2. REVIEW OF COMPUTATIONAL COLOR CONSTANCY ALGORITHMS Computational color constancy is the process of removing unrealistic color casts from digital images, mostly due to the acquisition conditions. From a computational perspective, computational color constancy is a two-stage process: the illuminant is estimated, and the image colors are then corrected on the basis of this estimate. The correction generates a new image of the scene as if it were taken under a known, canonical illuminant (see for example Fig. 2). A generic image acquired by a digital camera is mainly characterized by three physical factors: the illuminant spectral power distribution I(λ), the surface spectral reflectance S(λ) and the spectral sensitivities C(λ) of the sensor. Using this notation, the sensor responses at the spatial point with coordinates (x,y) can be then described as: where w is the wavelength range of the visible spectrum, ρ and C(λ) are three-component vectors. Since the three spectral sensitivities of the sensor C(λ) are usually respectively more sensitive to low, medium and high wavelengths, the three-component vector of the sensor response ρ = (ρ 1 ,ρ 2 ,ρ 3 ) is also referred to as the sensor or camera raw RGB = (R,G,B) triplet (see for example Fig. 3). Assuming that the color I of the illuminant in the scene observed by the camera only depends on the illuminant spectral power distribution I(λ) and on the spectral sensitivities C(λ) of the sensor, the computational color constancy problem is equivalent to the estimation of I by: given only the sensor responses ρ(x,y) across the image. This is an under-determined problem and therefore cannot be solved without further assumptions and/or knowledge, such as some information about the camera being used, and/ or assumptions about the statistical properties of the expected illuminants and surface reflectances. The estimation of the color of the illuminant could be performed if an achromatic patch is present in the image. This is because the spectral reflectance S(λ) of an achromatic Figure 2 – The two stages of the automatic white balance: the illuminant is estimated, and the image colors are then corrected on the basis of this estimate. The correction generates a new image of the scene as if it were taken under a known, canonical illuminant Figure 3 - Image formation process ∫= w λλλλ d)(),,()(),( Cñ yxSIyxρ ∫= w λλλ d)()( CI I 5104/15 | Cultura e Scienza del Colore - Color Culture and Science surface is approximately constant over a wide range of wavelengths, and thus the sensor response ρ is proportional to I, i.e. the RGB of the achromatic patch is proportional to that of the incident light. To reduce the dimensionality of the problem, one common method is to not estimate the whole triplet of the illuminant color, but a 2D projection of it in a chromaticity space. In fact, it is more important to estimate the chromatic components of the scene than its overall intensity. The color correction is usually based on a diagonal model of illumination change derived from the Von Kries hypothesis. This model assumes that two acquisitions of the same scene with the same imaging device but under different illuminants are related by an independent gain regulation of the three imaging channels [1][2]. A diagonal model is generally a good approximation of change in illumination, as shown by Finlayson et al.[2]. The colors in a scene, acquired under an unknown illuminant U can be transformed as they were taken under the chosen canonical illuminant by: where RGB = (R,G,B) is a color in the image acquired under the unknown illuminant, RGB' = (R',G',B') is the color in the corrected image, RGBU = (RU,GU,BU) are the sensor responses of a camera to a reference white under the unknown illuminant and RGBC = (RC,GC,BC) are the corresponding responses under the canonical illuminant. Supposing RGBC is known, to obtain the color correction matrix, we have to estimate the illuminant color RGBU. To this aim several algorithms exist in literature, each with different assumptions [3]-[11]. To improve the illuminant estimation, Schaefer et al. [12] introduced a combined physical and statistical color constancy algorithm that integrates the statistics-based Color by Correlation method with a physics-based technique, based on the dichromatic reflectance model, using a weighted combination of their likelihoods for a given illumination set and taking the maximum likelihood entry. Cardei and Funt [13] obtained good illuminant estimation by combining the results of gray world, white                     =           B G R BB GG RR B G R UC UC UC / / / ' ' ' patch and neural net methods, considering both linear and non-linear committee methods. We have investigated the idea of not relying on a single illuminant estimation method, but instead of considering a consensus decision that takes into account the compendium of the responses of several algorithms. To overcome the limitation deriving from the estimation of the illuminant color using the response of a single algorithm, methods that rely on the consensus of a set of different algorithms have been proposed [13] [14]. The underlying idea is that algorithms that give similar illuminant color estimations have to be trusted more than algorithms that give estimates that are far from the others, and thus the latter ones have to be automatically discarded. A different kind of algorithms proposed in the last few years belongs to the class of classification-based color constancy. The key idea is to exploit automatically extracted information about the content of the images and intrinsic, low level properties of the images. 3. CLASSIFICATION-BASED COLOR CONSTANCY We have in the last years proposed three different strategies of color constancy algorithm selection: a class-based (CB) [15] and a feature- based (FB) [16], and face-based (FcB) approach [17]. 3.1 CLASS-BASED COLOR CONSTANCY The class-based (CB) algorithm adopts a classification step to assign each image to either the indoor or to the outdoor class (see Fig. 4). The classifier is trained on low level features automatically extracted from the images (see [15] for a detailed description). Two different AWB algorithms have been used for the two possible classes in which the image considered can be classified: on the basis of the classification result, only the corresponding AWB algorithm selected has been applied. The algorithms for the indoor and for the outdoor class are selected from the one listed in [15], evaluating them on an independent training set. 3.2 FEATURE-BASED COLOR CONSTANCY The feature-based (FB) algorithm is based on five independent AWB algorithms and a classification step that automatically selects which AWB Figure 4 – Pipeline of the Class-based color constancy 52 Cultura e Scienza del Colore - Color Culture and Science | 04/15 Figure 5 - Pipeline of the Feature- based color constancy algorithm to use for each image (see Fig. 5). The classifier is trained on low level features automatically extracted from the images (see [16] for a detailed description). The features on which the classification-based color constancy methods rely for the image representation can be divided into two groups: general purpose and specifically designed features. The use of low level features for the automatic selection and combination of the best color constancy algorithm (or combination of multiple algorithms) permitted to outperform existing state of the art algorithms on a widely used benchmark dataset [15],[16]. The class-based and the feature-based color constancy algorithms can be thought as exploiting respectively high-level and low-level features: the CB works on the output of a scene classifier, more precisely an indoor/outdoor classifier; the FB works directly on the low-level features extracted from the image. The use of medium-level features has also been investigated [18][19]: they are used in region-based color constancy algorithms which are able to automatically select (and/or blend) among different color corrections, including a conservative do nothing strategy. A new class of feature-based algorithms replaces the use of hand-crafted features with an end- to-end learning of features and classifiers using Convolutional Neural Networks (CNNs) [37]. 3.3 FACE-BASED COLOR CONSTANCY Memory colors could be used as hints to give a more accurate estimate of the illuminant color in the scene. For example, Moreno et al. [20] obtained memory colors for three different objects (grass, snow and sky) using psychophysical experiments. They then used a supervised image segmentation method to detect memory color objects and exploit them to color correct the image using a weighted Von Kries formula. A different approach has been used in [17] where a face detector was used to find faces in the scene, and the corresponding skin colors were used to estimate the chromaticity of the illuminant. The method was based on two observations: first, skin colors tend to form a cluster in the color space, making it a cue to estimate the illuminant in the scene; second, many photographic images are portraits or contain people. An extension of this idea could be to use “memory objects” in the scene for color constancy. If we are able to automatically recognize objects and logos that have intrinsic colors, we can use them for color constancy. The performances of the proposed color constancy algorithms are very good and they are fully detailed in [14]-[17]. The performance measure adopted is the median angular error between the estimated and measured illuminant. The datasets used are standard and widely for benchmarking algorithms as they are with ground truth illuminant measurements [32]. 3.4 ADAPTIVE FACE-BASED COLOR CONSTANCY The face-based color constancy algorithm described in the next subsection has been extended in different ways [33]. Since one of the assumptions that is often violated in color constancy is the presence of a uniform illumination in the scene, we have extended the applicability of the face-based algorithm to the case of non-uniform illumination. The method is adaptive, being able to distinguish and process in different ways images of scenes taken under a uniform and those acquired under non- uniform illumination. This was the first algorithm which automatically modifies its behavior from global to local color correction according to the analysis of the image content. Furthermore, we designed a more efficient algorithm to estimate the scene illuminant from extracted skin regions using only their mean color value instead of the whole gamut. This algorithm is more suitable for resource-limited camera devices, such as consumer digital cameras and camera phones. 4. COLOR SPACE TRANSFORMATION The second stage of the color correction pipeline is the device chromatic response characterization and transforms the image data into a standard RGB color space (e.g. sRGB, ITU-R BT.709). This transformation, usually called color matrixing, is needed because the spectral sensitivity functions of the sensor color 5304/15 | Cultura e Scienza del Colore - Color Culture and Science channels rarely match those of the desired output color space. Typically this transformation is a 3-by-3 matrix with 9 variables to be optimally determined, and both algebraic [21] and optimization-based methods [22] exist to find it. The typical color correction pipeline can be thus described as follows: where RGB in are the camera raw RGB values, a is an exposure compensation common gain, the diagonal matrix diag(r awb ,g awb ,b awb ) is the channel-independent gain compensation of the illuminant, the full 3-by-3 matrix a (i,j) ,(i,j) = {1,2,3}2 = {1,2,3}G{1,2,3} is the color space conversion transform from the device- dependent RGB to the sRGB color space, γ is the gamma correction defined for the sRGB color space (where for abuse of notation it is intended to be applied component-wise), and RGB out are the output sRGB values. Usually the color matrix transform is optimized for a single illuminant and is applied as it is for all the illuminants that can occur. This could lead to high colorimetric accuracy if the occurring illuminant is the one for which the matrix has been derived (assuming that it is correctly compensated by the AWB module), and low colorimetric accuracy for different illuminants. In [23] we have shown how to compute a combined matrix for different classes of commonly occurring illuminants. If only a-priori probability distribution about the illuminant occurrences is known, the best color matrix can be found offline and applied as it is for all the shots; if the AWB is able to give a probability distribution about the illuminant in the scene (as color-by-correlation [9] does), an adaptive optimal matrix transform could be found for each shot. Starting from the observation that the illuminant estimation is not error free and being an ill-posed problem [14] a perfect algorithm does not exist, color correction matrices, in addition to color space conversion, can incorporate information about the illuminant estimation process in order to compensate for its possible errors [23],[34] (see for example Fig. 6). 4.1 ADAPTIVE COLOR SPACE TRANSFORMATION In [35] we designed and tested extended color correction pipelines for digital cameras able to obtain a higher color rendering accuracy. The pipelines proposed in [23],[34] have been further improved in two ways: i) in the illuminant estimation and correction stage, the traditional diagonal model of illuminant change has been replaced by a generalized diagonal transform found by optimization; ii) the color matrixing stage, usually performed using a linear transformation matrix optimized assuming that the illuminant in the scene has been successfully estimated and compensated for, has been extended exploiting polynomial color space conversions incorporating knowledge about illuminant estimation module behavior. [35] 5. CONCLUSIONS AND FUTURE WORKS Both the illuminant estimation process and the color correction matrix concur in the formation of the overall perceived image quality. We have briefly summarized our research activities concerning them. Interested readers may refer to the cited works to have a detailed description of the algorithms as well as an exhaustive comparison of experimental results with respect to other algorithms in the state-of- art. Independently from the merits of the single algorithms, we would like to point it out that illuminant estimation and color correction have Figure 6 - Computation of color correction matrices which, in addition to color space conversion can incorporate information about the illuminant estimation process in order to compensate for its possible errors 54 Cultura e Scienza del Colore - Color Culture and Science | 04/15 been always studied and optimized separately, thus ignoring the interactions between them. Our works, to the best of our knowledge, are the first ones that have investigated their interactions and how to optimize them for the overall color accuracy. One of the possible further extension of these works regards the illuminant correction step. Once the scene illuminant has been estimated the scene is usually corrected in the RGB device dependent color space using the diagonal Von Kries model [2]. Several studies have investigated the use of different color spaces for the illuminant correction [24],[25],[26] as well as non-diagonal models [26]. A different approach could be to use chromatic adaptation transforms (CATs) [27] to correct the scene illuminant. CATs are used in color science and color imaging to model illumination change given the source and target illuminants. Since in computational color constancy the source illuminant is unknown, the development of more accurate color constancy algorithms will allow to use more performing CATs [28]. A different strategy for improving the reliability of color constancy algorithms is to increase the quantity of information available for the color constancy by taking two pictures of each scene [29]: the first is taken as normal, while a specially chosen colored filter is placed in front of the camera when capturing the second image. The filter is chosen so that the combined image makes color constancy, or white point estimation easier to solve. The availability of more color information available could be also used to increase the colorimetric accuracy of the device [30] or even for a spectral reconstruction of the scene [31]. The recent development of 3D cameras will surely boost these approaches. BIBLIOGRAPHY [1] M. D. Fairchild, Color Appearance Models, Addison Wesley, Boston, Mass. 1997. [2] G. D. Finlayson, M. S. Drew, and B. Funt, “Diagonal transform suffice for color constancy,” in Proc. IEEE Int. Conf. Computer Vision, pp. 164–171, 1993. [3] K. Barnard, V. Cardei, and B. Funt, “A comparison of computational color constancy algorithms; part one: methodology and experiments with synthetic images,” in IEEE Trans. Image Process. 11(9), 972–984, 2002. [4] K. Barnard, V. Cardei, and B. Funt, “A comparison of computational color constancy algorithms; part two: experiments with image data,” in IEEE Trans. Image Process. 11(9), 985–996, 2002. [5] A. Bruna, F. Gasparini, F. Naccari, and R. Schettini, “Multidomain pixel analysis for illuminant estimation and compensation,” in Proc. Digital Photography II, Proc. SPIE 6069, 60690C, 2006. [6] V. Cardei, B. Funt, and K. Barndard, “White point estimation for uncalibrated images,” in Proc. IS&T/SID 7th Color Imaging Conf., pp. 97–100, 1999. [7] H. H. Chen, C. Shen, and P. Tsai, “Edge-based automatic white balance with linear illuminant constraint,” in Visual Communications and Image Processing 2007, Proc. SPIE 6508, 65081D, 2007. [8] G. D. Finlayson, “Color in perspective,” IEEE Trans. Pattern Anal. Mach. Intell. 18, 1034–1038, 1996. [9] G. D. Finlayson, P. Hubel, and S. Hordley, “Color by correlation,” in Proc. IS&T/SID 5th Color Imaging Conf.: Color Science, Systems and Applications, pp. 6–11, 1997. [10] G. D. Finlayson, S. D. Hordley, and P. M. Hubel, “Color by correlation: a simple, unifying framework for color constancy,” IEEE Trans. Pattern Anal. Mach. Intell. 23, 1209–1221, 2001. [11] D. Forsyth, “A novel algorithm for color constancy,” Int. J. Comput. Vis. 5, 5–36, 1990. [12] G. Schaefer, S. Hordley, and G. Finlayson, “A combined physical and statistical approach to colour constancy,” in Proc. Int. Conf. Comput. Vis. Pattern. Recogn., pp. 148– 153, 2005. [13] V. C. Cardei and B. Funt, “Committee-based colour constancy,” in Proc. IS&T/SID 7th Color Imaging Conf., pp. 311–313, 1999. [14] S. Bianco, F. Gasparini, and R. Schettini, “Consensus based framework for illuminant chromaticity estimation,” Journal of Electronic Imaging, vol. 17(2), pp. 023013, 2008. [15] S. Bianco, G. Ciocca, C. Cusano, and R. Schettini, “Improving color constancy using indoor-outdoor image classification,” IEEE Transactions on Image Processing, vol. 17(12), pp. 2381–2392, 2008. [16] S. Bianco, G. Ciocca, C. Cusano, and R. Schettini, “Automatic color constancy algorithm selection and combination,” Pattern Recognition, vol. 43, pp. 695–705, 2010. [17] S. Bianco, R. Schettini, “Color constancy using faces”, Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 65-72, 2012. [18] F. Gasparini and R. Schettini, “Color balancing of digital photos using simple image statistics,” Pattern Recognition, vol. 37(6), pp. 1201–1217, 2004. [19] S. Bianco, F. Gasparini, and R. Schettini, “Region- based illuminant estimation for effective color correction,” Proceedings of 15th International Conference on Image Analysis and Processing (ICIAP 2009), LNCS, vol. 5716/2009, pp. 43–52, 2009. [20] A. Moreno, B. Fernando, B. Kani, S. Saha, and S. Karaoglu, “Color correction: A novel weighted von kries model based on memory colors,” Proc. of the 2011 Computational Color Imaging Workshop (CCIW’11), vol. LNCS 6626/2011, 2011. [21] P. M. Hubel, J. Holm, G. D. Finlayson, and M. S. Drew, “Matrix calculations for digital photography,” Proceedings of the IS&T/SID Fifth Color Imaging Conference, pp. 105– 111, 1997. [22] S. Bianco, F. Gasparini, A. Russo, and R. Schettini, “A new method for RGB to XYZ transformation based on pattern search optimization,” IEEE Trans. Cons. Electr., vol. 53(3), pp. 1020–1028, 2007. [23] S. Bianco, A. Bruna, F. Naccari, R. Schettini, “Color space transformations for digital photography exploiting 5504/15 | Cultura e Scienza del Colore - Color Culture and Science information about the illuminant estimation process”, Journal of the Optical Society of America A, vol. 29(3), pp.374-384, 2012. [24] K. Barnard, F. Ciurea, and B. Funt, “Sensor sharpening for computational color constancy,” Journal of the Optical Society of America A, vol. 18, pp. 2728–2743, 2001. [25] F. Xiao, J. E. Farrell, J. M. Dicarlo, and B. A. Wandell, “Preferred color spaces for white balancing,” Proceedings of the SPIE-IS&T, Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications IV, vol. 5017, pp. 342–350, 2003. [26] B. Funt and H. Jiang, “Nondiagonal color correction,” Proceedings of the 2003 International Conference on Image Processing (ICIP 2003), pp. 481–484, 2003. [27] CIE 160:2004, “A review of chromatic adaptation transforms,” ISBN: 9783901906305, 2004. [28] S. Bianco and R. Schettini, “Two new von kries based chromatic adaptation transforms found by numerical optimization,” Color Research and Application, vol. 35(3), pp. 184–192, 2010. [29] G. D. Finlayson, S. D. Hordley, and P.Morovic, “Colour constancy using the chromagenic constraint,” Computer Vision and Pattern Recognition (CVPR’05), pp. 1079–1086, 2005. [30] S. Bianco, F. Gasparini, R. Schettini, and L. Vanneschi, “Polynomial modeling and optimization for colorimetric characterization of scanners,” J. of Electronic Imaging, vol. 17(4), pp. 043002, 2008. [31] R. Shrestha, J. Y. Hardeberg, and A. Mansouri, “One- shot multispectral color imaging with a stereo camera,” Proceedings of the SPIE-IS&T, Digital Photography VII, vol. 7876, 2011. [32] http://colorconstancy.com/ [33] S. Bianco, R. Schettini, “Adaptive color constancy using faces”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36(8), pp. 1505-1518, 2014. [34] S. Bianco, A. Bruna, F. Naccari, R. Schettini, “Color correction pipeline optimization for digital cameras.” Journal of Electronic Imaging, vol. 22(2), pp. 023014, 2013. [35] S. Bianco, R. Schettini, “Error-tolerant color rendering for digital cameras.” Journal of Mathematical Imaging and Vision, vol. 50(3), pp. 235-245, 2014. [36] G. D. Finlayson, M. S. Drew, B. V. Funt. “Color constancy: generalized diagonal transforms suffice.” Journal of the Optical Society of America A, vol. 11(11), pp. 3011-3019, 1994. [37] S. Bianco, C. Cusano, R. Schettini, “Color constancy using CNNs”, IEEE Computer Vision and Pattern Recognition Workshops (CVPRW), 2015.