Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 1, No. 3, 2022 88 A Survey of Low-light Image Enhancement Weiqiang Liu1, Peng Zhao1, *, Xiangying Song2, Bo Zhang 1 1 College of Operational Support, Rocket Force University of Engineering, Xi’an, China; 2 Troops No.96951, Beijing, China. * Corresponding author: Peng Zhao. Abstract: With the higher requirements of computer vision image enhancement of low-light image has become an important research content of computer vision. Traditional low-light image enhancement algorithms can improve image brightness and detailed visibility to varying degrees, but due to their strict mathematical derivation, such methods have bottlenecks and are difficult to break through their limits. With the development of deep learning and the birth of large-scale data sets, low-light image enhancement based on deep learning has become the mainstream trend. In this paper, first of all, the traditional low-light image enhancement algorithms are classified, summarized the improvement process of the traditional method, then the image enhancement method based on the deep learning are introduced, at the same time on the network structure and is suitable for the method of combing the network part, after the introduction to the experiment database and enhance image evaluation criteria. Based on the discussion of the above situation, combined with the actual situation, this paper points out the limitations of the current technology, and predicts its development trend. Keywords: Image enhancement; Low-light image dataset; Deep learning; Generative adversarial networks. 1. Introduction Due to the particularity of insufficient brightness and uneven distribution of night lighting sources, the brightness value of images collected at night is extremely low, and the visibility and contrast are severely reduced, accompanied by a large amount of random noise. In real life, it is sometimes impossible to avoid capturing images at night, especially in important and widely needed applications such as surveillance forensics, night vision equipment, face recognition and fault detection. Low-light image enhancement methods aim to improve the brightness and contrast of low-light images, while avoiding excessive enhancement and noise methods, etc., to improve the visual effect and meet the image information richness required by subsequent computer vision algorithms. At present, the research on low-light image enhancement algorithm has achieved good results in the continuous exploration and progress, and the enhanced image has excellent performance in both subjective visual experience and objective image quality evaluation. However, there are still some thorny problems to be solved, such as noise amplification and more serious color deviation in low-light image enhancement. 2. Traditional low-light image enhancement method 2.1. Algorithm based on Hue Mapping Tonal mapping is to map the pixel value at the same position of the image to another pixel value close to normal illumination through the mapping function, so that the difference between it and the surrounding pixels is more obvious, so as to improve the image quality. Bennett et al. obtained the large-scale features of the image through bilateral filtering, and completed the image enhancement by linear sum of the enhanced details and large-scale features. Xu et al. used logarithmic function as mapping function, and set the base of logarithmic function according to the gray value of pixels. The nonlinear tonal mapping method can well expand the dynamic range of the dark area of low-light image and keep other areas unchanged, so the processing effect is better. 2.2. Algorithm based on Histogram Equalization Based on the Histogram Equalization (HE) method, it redistributes the Histogram of low-light image, changes the dynamic range of the gray value of the image, reduces the number of pixels in the gray area, widens the Histogram distribution of the enhanced image, and achieves the effect of enhancing image brightness and contrast. However, the enhanced image pixels are mainly distributed in the middle and low regions, and the enhanced pixels of the same information cannot produce different pixels. Adaptive Histogram Equalization (AHE) adjusts the histogram of multiple regions, but amplifies the noise of the image. Contrast Limited Adaptive Histogram Equalization (CLAHE) overcomes the problem of noise amplification by Contrast limiting. Although the image enhancement algorithms based on Histogram Equalization are constantly improved by researchers, most methods are still not flexible enough to enhance local areas, and there are problems such as over- exposure and noise amplification. 2.3. Low-light Image Enhancement Method based on Retinex Retinex theory explains the basic principle that human eyes have the ability to observe the color of the target object. It assumes that the image is composed of an irradiation map and a reflection map. 𝑆(𝑥, 𝑦) = 𝐼(𝑥, 𝑦) ⊙ 𝑅(𝑥, 𝑦) (1) 𝑆 Single Scale Retinex (SSR) algorithm removes the low- frequency irradiation part of the image by central surround function, and takes the reflection image as the enhanced 89 image. Multi-scale Retinex (MSR) algorithm is improved on the basis of single-scale Retinex algorithm. It can realize color enhancement, color constancy, local dynamic range compression and global dynamic range compression while maintaining the high fidelity of image and compress the dynamic range of image. Common disadvantages of MSR include insufficient edge sharpening, abrupt shadow boundaries, partial color distortion, unclear texture, no significant improvement in the details of the highlight area, and low sensitivity to the highlight area. In addition, multi- scale Retinex algorithm (MSRCR) with color restoration based on Retinex theory and LIME algorithm can improve the brightness and contrast of image to a certain extent, but cannot suppress the amplification of image noise during the enhancement process. 2.4. Low-light image enhancement method based on atmospheric scattering model Dong et al. conducted a large number of statistical and experimental verification on the inversion images of foggy images and low-light images, and found that fog images and low-light inversion images were very similar in visual effect, and most of the pixels in the distant part had high RGB values. The imaging model of foggy images is shown in Formula 2. 𝐼(𝑥) = 𝐽(𝑥)𝑡(𝑥) + 𝐴[1 − 𝑡(𝑥)] (2) In the above formula, x is the pixel of the image, and ( )I x is the fog map collected by the camera equipment; ( )J x represents the light reflected from the scene itself, i.e., a clear fog-free image; A is the global atmospheric light value. For A fog map, the atmospheric light value is constant. ( )t x is the transmittance, and its value is between 0 and 1. The larger the transmittance is, the stronger the reflected light's ability to penetrate fog. When the transmittance is 1, it is a fog-free environment. Formula 3 is obtained by the transformation of formula 2. t is usually set as 0.1, which is set to prevent the denominator from being too small and the value of is too large, leading to the transition of the image to the white field. Therefore, the lower limit is set for ( )t x . 𝐽(𝑥) = 𝐼(𝑥)−𝐴 𝑚𝑎𝑥[𝑡(𝑥),𝑡0] + 𝐴 (3) For low light images the reversal of the figure is similar to the fog on the vision chart to this phenomenon, first will get the similar Low-light image reversal after the reversal of the fog figure image, then the transmittance of the pseudo fog figure estimates and atmospheric light will finally solve the parameters into the formula for image restoration, restoration no fog image, further enhanced by low light images. Feng et al. used pyramid dense residual block network to estimate the transmittance of the pseudo-fog image, and calculated the atmospheric light value by k-means classification method based on dark channel prior, so as to enhance the image. As a result, this method can enhance the overall brightness of low- light images, but the enhancement effect of this method on local information is insufficient because the atmospheric light value is a global quantity. 3. Low-light image enhancement method based on deep learning With the underlying image processing method to improve the quality of Low-light image enhancement results can meet the requirements of most applications, but the method itself is determined to detail has been lost night up to the ideal image enhancement as a result, the quality of enhanced image is often poor, especially for a single night image enhancement method, Therefore, it can be said that there is a bottleneck in this method, which cannot really process all night images. As deep learning has achieved good results in different fields, a large number of scholars firmly believe that the method to break the bottleneck problem of Low-light image enhancement algorithm must exist in the learning-based method. Low light enhancement methods based on deep learning can be divided into two branches. Both the Convolutional Neural Network (CNN) method and the Generative Adversarial Networks (GAN) method are proposed. Both methods are based on data - driven model training. 3.1. Low-light Image Enhancement Method based on Convolutional Neural Network CNN is a common deep learning model. A typical convolutional neural network is composed of convolution layer, pooling layer and full connection layer. The convolution layer is responsible for extracting local features of images, pooling layer is responsible for reducing the magnitude of parameters, and full connection layer is responsible for outputing desired results. Because convolutional neural network has the characteristics of local connection and weight sharing, it reduces the training parameters and reduces the complexity of network model, so it is proposed to enhance low-light images. The method based on CNN relies on pairs of images for supervised training. As the first low-light network (LLNet ) that applies deep learning to image enhancement, it builds a deep network to learn the features of low-light image and improves the local contrast so as to prevent over- amplification of already bright pixels. It can realize image contrast enhancement and noise reduction at the same time. Lv et al proposed multi-branch low-light level Enhanced network (MBLLEN), which extracted and fused features of different levels in the network to obtain enhanced images. At present, the more popular method is to combine convolutional neural network with Retinex theory to process the irradiation image or reflection image of low-light image. The traditional MSR method can be replaced by the feedforward neural network with different Gaussian convolution kernels, so Shen et al. proposed the MSR-Net network to realize image enhancement by learning the mapping relationship between Low-light images and normal images. Wei et al. proposed RetinexNet, which decomposed the low-light image into reflection component and illumination component through decomposition network, and used BM3D to suppress the reflection component. HDR-Net combines deep network with bilateral grid processing and local affine color transformation. The KinD network proposed by Zhang et al. consists of a three-word network, which performs image decomposition, reflection image recovery, illumination adjustment, training through gamma correction simulation data, and introduces more loss functions for constraints. Then, KinD++ was proposed to optimize KinD by introducing multi-scale illumination attention module. Fan et al. enhanced the reflection image and illumination image through the prior information of image semantics, and achieved good results. 3.2. Low-light Image Enhancement Method based on Generative Adversarial Network Generative adversarial network is an unsupervised deep 90 learning model. The traditional generative adversarial network consists of a generator and a discriminator, and its principle is to "trick" the discriminator with the data generated by the generator. The generator misleads the discriminator by increasing its ability to produce samples that are closer and closer to the real thing, and the discriminator learns and distinguishes the spurious samples generated by the generator. The training process can be regarded as a game between the discriminator and the generator, iterating until the discriminator cannot tell whether the received sample is from the real sample or from the generated sample. Jiang et al. 's discussion on the development of the net- work enables the development of the garden and the discussion of the garden through the "global-local" discriminator and the self-regularizing attention mechanism. Guo et al. designed a parameter estimation network zero- DCE, which gradually enhanced the brightness and contrast of the image through continuously iterated optimized estimation curves until relatively satisfactory enhancement effects were obtained. Li et al. improved zero-DCE and obtained zero-DCE ++, which significantly improved the computing efficiency while maintaining the original performance. This method can still achieve the effect of enhancing low-light images without reference images. However, due to the lack of paired training sets, the enhancement effect is not obvious for extremely low-light images, and the details of images cannot be effectively restored. Moreover, noise amplification exists in the enhancement results. 3.3. Low-light Enhancement Method based on Semi-supervision The fully supervised framework can obtain strong signal fidelity constraints to prevent deviation from the target, while unsupervised training of high-quality image data sets can ensure strong visual perception quality. Semi-supervised learning combines the advantages of supervised and unsupervised learning by using both labeled and unlabeled samples. DRBN designed by Yang et al. restores the details of low-light images by training paired images through recursive network, and improves the visual effect of low-light images by training unpaired image data through admission-learning. The RUAS model proposed by Liu et al. uses the unreferenced loss function for training and uses the method of structure search, which greatly improves the processing speed of the network. Figure 1 shows the visualized results of some of the methods. Fig. 1 Visual comparison of low-light image enhancement results. (a) Underexposed images; (b) SSR ; (c) MSR ; (d) MSRCR ; (e) Dong ; (f) LLNet ; (g) LightenNet ; (h) MBLLEN ; (i) Retinex- Net ; (j) EnlightenGAN ; (k) ExCNet ; (l) KinD ; (m) DUPE ; (n) Zero-DCE ; (o) DRBN ; (p) RRDNet ; (q) TBEFN ; (r) DSLR ; (s) KinD++ ; (t) Zero-DCE++ ; (u) Normal image. 4. Data sets and evaluation criteria 4.1. Experimental data Set The existing low-light image enhancement data set presents a trend from small to large, from single to diverse scenes, and from simple to complex data difficulty, as shown in Table 1. Early image enhancement data sets are mainly small in scale and only contain low-light images, such as DICM, LIME, MEF, NPE and VV. In recent years, with the development of deep learning, a batch of large-scale paired datasets have emerged, among which the representative LoL datasets are paired datasets obtained from real scenes by changing exposure time. However, due to the complicated experimental Settings, such as the camera needs to be fixed and objects cannot be moved, there are only 500 pairs. MIT- Adobe 5K consists of 5,000 low-light images and hand- touched images by five professional decorators (A, B, C, D, E). Pairwise data sets are mainly obtained by changing exposure time or expert modification. Experimental Settings are complicated, so most low-light image enhancement data sets belong to unpaired data sets. Figure 2 shows the examples of images in some low-light datasets. Table 1. Summary of low-light image dataset No Abbreviat ion Format Real/ Synt hetic Number Paired/ Unpaire d Sourc e 1 DICM RGB Real 64 Unpaired IEEE 2013 2 LIME RGB Real 10 Unpaired IEEE 2017 3 MEF RGB Real 17 Unpaired IEEE 2015 4 NPE RGB Real 84 Unpaired IEEE 2013 5 VV RGB Real 24 Unpaired MAT 2018 6 LoL RGB Real & Synthet ic 500 Paired BMV C 2018 7 MIT- Adobe 5K RAW Real & Synthet ic 5000 Paired CVPR 2011 8 SICE RGB Real 4413 Paired IEEE 2018 9 SID RAW Real 5094 Paired CVPR 2018 10 LLIV- Phone RGB Real 45148 Unpaired IEEE 2021 Fig. 2 Example of images in some low-light datasets. (a) LIME; (b) MEF; (c) NPE; (d) DICM; (e) (f) LoL; (g) (h) MIT-Adobe FiveK. 4.2. Image Evaluation Criteria Subjective qualitative analysis. On the subjective evaluation from the visual effect of the enhanced image evaluation, use different algorithms to enhance the same, a low-light image depends on volunteers to evaluate quality of enhanced image visual, mainly from the enhanced image if there is a natural contrast and color distortion of the distortion of phenomenon, such as whether to retain the original characteristics for evaluation. Subjective evaluation lacks generality and is easy to cause irrational judgment due to cognitive deviation. Objective quantitative analysis. As an objective way to evaluate the quality of the enhanced image with digital features as reference, objective evaluation evaluates the 91 quality of the enhanced image through digitalization. The commonly used objective evaluation indicators include Information Entropy, Natural Image Quality Evaluator, NIQE, Structural Similarity Index Measure (SSIM) and Peak Signal to Noise Ratio (PSNR). IE reflects the amount of information carried by an image. The larger IE value is, the richer the image information is, the clearer the image is and the higher the quality is. NIQE completes the evaluation of enhanced images by extracting features from natural landscapes and using a multivariate Gaussian model to describe the features. The smaller the NIQE value is, the smaller the gap between the enhanced image and the normal natural image, and the smaller the image enhancement effect is. PSNR is the ratio of the maximum semaphore to noise intensity. The larger the value is, the smaller the image distortion is. When the two images are almost the same, the PSNR value is close to 100. SSIM mainly describes the similarity between images. It uses mean to estimate brightness, standard deviation to estimate contrast, and covariance to measure structural similarity. The larger SSIM value is, the better the enhanced image effect is. Objective evaluation also has its limitations. For example, because the sensitivity of human vision to error is not absolute, the PSNR value of images with good visual effect may be lower than that of images with poor visual effect. NIQE is a global judgment of the image, which cannot be used for local quality evaluation. High level visual task. In recent years, more and more researchers have evaluated the quality of enhanced images by using downstream high-level visual tasks. By inputting the results of different low-light image enhancement methods into the same face detection model, Guo et al. studied the face detection task of low-light image enhancement methods under the condition of low illuminance. Based on the idea of "enhancement + matching", Song et al. took image matching under low-light as the evaluation standard of image enhancement. 5. The development trend of image enhancement At present, the low-light image enhancement field has gradually completed the transformation from traditional enhancement method to deep learning method. In terms of learning mechanism, the fully supervised method requires a large number of paired training samples, but currently paired data resources are not abundant, so the training method is transitioning from fully supervised to semi-supervised and unsupervised. In application scenarios, enhancement objects are now being transformed from simple scenes to complex real scenes. In terms of evaluation methods, the perception evaluation system based on visual quality is poor in generalization, and the evaluation criteria of enhanced images are transitioning from downstream high-level visual tasks. In addition, to date, most low-light enhancement method is applied to the image, and low-light video enhancement technique is based on image enhancement, in recent years, some scholars began to study the single frame image enhancement and the coherence between two frames, therefore, will become low-light of low-light video real-time enhancement of the direction of a more challenging. In the long run, low-light image enhancement is still worth looking forward to with further research. Acknowledgement This work was financially finished by my teacher, and thanks to my classmates for their help. References [1] E. P. Bennett and L. Mcmillan, “Video enhancement using per- pixel virtual exposures,” ACM Transactions on Graphics, 2005, pp.845-852. [2] Q. Xu, H. Jiang, R. Scopigno and M. Sbert, “A novel approach for enhancing very dark image sequences,” Signal Processing, 2014, pp.309-330. [3] H. Yeganeh, A. Ziaei and A. Rezaie, “A novel approach for contrast enhancement based on Histogram Equalization”, International Conference on Computer & Communication Engineering, 2008. [4] S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie and K. Zuiderveld, “Adaptive histogram equalization and its variations,” Computer Vision Graphics & Image Processing, 1987, pp.355-368. [5] S. M. Pizer, R. E. Johnston, J. P. Ericksen, B. C. Yankaskas and K. E. Muller, “Contrast-limited adaptive histogram equalization: speed and effectiveness,” Visualization in Biomedical Computing, 1990. [6] E. H. Land, “The retinex theory of color vision” Scientific American, 1997, pp: 108-128. [7] F. Tajeripour, S. Fekri-Ershad, “Developing a novel approach for stone porosity computing using modified local binary patterns and single scale retinex” Arabian Journal for Science and Engineering, 2014, pp: 875-889. [8] H. Lin and Z Shi, “Multi-scale retinex improvement for nighttime image enhancement” Optik-International Journal for Light and Electron Optics, 2014, pp: 7143–7148. [9] J. Ma, X. Fan, J. Ni, X. Zhu and C. Xiong, “Multi-scale retinex with color restoration image enhancement based on Gaussian filtering and guided filtering” International Journal of Modern Physics B, 2017. [10] X. Guo, L. Yu and H. Ling, “LIME: Low-light Image Enhancement via Illumination Map Estimation” Transactions on Image Processing, 2017, pp: 982-993. [11] X. Dong, G. Wang and Y. Pang, “Fast efficient algorithm for enhancement of low lighting video” IEEE International Conference on Multimedia & Expo (ICME), 2011, pp: 1-6. [12] X. Feng, J. Li and Z. HuaC, “Low-light image enhancement algorithm based on an atmospheric physical model,” Multimedia Tools and Applications, 2020, pp.1–25. [13] K. G. Lore, A Akintayo and S. Sarkar, “LLNet: A Deep Autoencoder Approach to Natural Low-light Image Enhancement” Pattern Recognition, 2017, pp: 650-662. [14] F. Lv, F. Lu, J. Wu and C. Lim, “MBLLEN: Low-light Image/Video Enhancement Using CNNs,” British Machine Vision Conference (BMVC), 2018. pp: 220-228. [15] L. Shen, Z. Yue, F. Feng, Q. Chen, S. Liu and J. Ma, “MSR- net: Low-light Image Enhancement Using Deep Convolutional Network,” 2017. [16] C. Wei, W. Wang, W. Yang and L. Jiaying, “Deep retinex decomposition for low-light enhancement,” British Machine Vision Conference (BMVC), 2018, pp.1–12. [17] K. Dabov, A. Foi and V. Katkovnik, “Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering” IEEE Transactions on Image Processing, 2007, pp: 2080-2095. 92 [18] M. Charbi, J. Chen, J. T. Barron, S. W. Hasinoff, “Deep bilateral learning for real-time image enhancement” ACM Transactions on Graphics, 2017, pp: 1-12. [19] Y. Zhang, J. Zhang, and X. Guo, “Kindling the darkness: A practical low-light image enhancer,” in ACMMM, 2019, pp. 1632–1640. [20] Y. Zhang, X. Guo, J. Ma, W. Liu, and J. Zhang, “Beyond Brightening Low-light Images,” International Journal of Computer Vision, 2021. [21] Y. Jiang, X. Gong, D. Liu, Y. Chen, C. Fang, X. Shen, et al. “EnlightenGAN: Deep Light Enhancement without Paired Supervision” ,2019, arXiv: 1906. 06972. [22] C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong et al. “Zero- Reference Deep Curve Estimation for Low-Light Image Enhancement” Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp: 1777-1786. [23] C. Li, C. Guo, and C. C. Loy, “Learning to enhance low-light image via zero-reference deep curve estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. [24] W. Yang, S. Wang, Y. Fang, Y. Wang and J. Liu, “From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement,” Computer Vision and Pattern Recognition (CVPR). IEEE, 2020. [25] R. Liu, L. Ma, J. Zhang, X. Fan and Z. Luo, “Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low- light Image Enhancement,” Computer Vision and Pattern Recognition (CVPR). IEEE, 2020. [26] C. Lee, C. Lee and C. S. Kim, “Contrast enhancement based on layered difference representation,” Image Processing, 2013, pp.965–968. [27] K. Ma, K. Zeng, Z. Wang, “Perceptual Quality Assessment for Multi-Exposure Image Fusion,” IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, 2015. [28] S. Wang, J. Zheng, H. M. Hu and H. Li, “Naturalness preserved enhancement algorithm for non-uniform illumination images,” IEEE transactions on image processing: a publication of the IEEE Signal Processing Society, 2013, pp.38-48. [29] V. Vonikakis, R. Kouskouridas and A. Gasteratos, “On the evaluation of illumination compensation algorithms”, Multimed Tools Appl, 2018, pp.9211–9231. [30] B. Vladimir, P. Sylvain, C. Eric and D. Fredo, “Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs” in CVPR, 2011, pp: 47-54. [31] J. Cai, S. Gu and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Trans Image Process, 2018, pp.2049–2062. [32] C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to see in the dark,” Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3291–3300. [33] C. Y. Li, L. C. Guo, L. H. Han, J. Jiang, M. M. Cheng, J. W. Gu et al. “Low-light image and video enhancement using deep learning: a survey,” 2021. [34] C. Li, J. Guo, F. Porikli, and Y. Pang, “Lightennet: A convolutional neural network for weakly illuminated image enhancement,” Pattern Recog-nition Letters, 2018, pp. 15–22. [35] L. Zhang, L. Zhang, X. Liu, Y. Shen, S. Zhang, and S. Zhao, “Zero-shot restoration of back-lit images using deep internal learning,” in ACMMM, 2019, pp. 1623–1631. [36] R. Wang, Q. Zhang, C. W. Fu, X. Shen, W. S. Zheng and J. Jia, “Underexposed Photo Enhancement Using Deep Illumination Estimation” Conference on Computer Vision and Pattern Recognition (CVPR), 2019 [37] A. Zhu, L. Zhang, Y. Shen, Y. Ma, S. Zhao, and Y. Zhou, “Zero-shot restoration of underexposed images via robust retinex decomposition,” in ICME, 2020, pp. 1–6. [38] K. Lu and L. Zhang, “TBEFN: A two-branch exposure-fusion network for low-light image enhancement,” IEEE Transactions on Multimedia, 2020. [39] S. Lim and W. Kim, “DSLR: Deep stacked laplacian restorer for low-light image enhancement,” IEEE Transactions on Multimedia, 2020. [40] W. Z. Song, M. Suganuma, X. Liu, N. Shimobayashi, D. Maruta, T. Okatani et al, “Matching in the Dark: A Dataset for Matching Image Pairs of Low-light Scenes,” arXiv e-prints, 2021.