49 Characterization and Application of Nanomaterials (2021) Volume 4 Issue 2 doi:10.24294/can.v4i2.1335 Original Research Article Defocused image restoration method based on micro-nano scale Yongjun Liu1,2*, Qiuyu Wu1, Mingxin Zhang1, Yi Wang2 1 School of Computer Science and Engineering, Changshu Institute of Technology, Changshu 215500, China. E-mail: yongjun1981@126.com 2 School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China ABSTRACT An image adaptive noise reduction enhancement algorithm based on NSCT is proposed to perform image resto- ration preprocessing on the defocused image obtained under the microscope. Defocused images acquired under mi- cro-nano scale optical microscopy, usually with inconspicuous details, edges and contours, affect the accuracy of sub- sequent observation tasks. Due to its multi-scale and multi-directionality, the NSCT transform has superior transform functions and can obtain more textures and edges of images. Combined with the characteristics of micro-nanoscale optical defocus images, the NSCT inverse transform is performed on all sub-bands to reconstruct the image. Finally, the experimental results of the standard 500 nm scale grid, conductive probe and triangular probe show that the proposed algorithm has a better image enhancement effect and significantly improves the quality of out-of-focus images. Keywords: Micro-nano Scale; Defocused Image; NSCT; Image Restoration 1. Introduction When observing samples with optical microscope, there is a prob- lem of “Abbe limit” due to the limitation of diffraction limit[1,2]. When the observation scale reaches the micro-nano level, the observed imag- es show fuzzy circular spots no matter how to adjust the focal length. Therefore, the images collected under micro-nano optical microscope have poor resolution, and unclear edges, details and contours not only affect the quality of the image, but also are not conducive to the subse- quent observation of “depth” and other information[3]. Traditional image enhancement and preprocessing methods in- clude histogram equalization algorithm[4], Retinex algorithm[5,6], ho- momorphic filtering algorithm[7], wavelet transform algorithm[8], etc., which have made great progress in recent years[4–8]. However, they have their own advantages and disadvantages, such as serious block effect and relatively complex operation in local equalization algorithm. Global equalization algorithm is also easy to cause color distortion due to over fitting. Retinex algorithm is based on color constancy. Homo- morphic filtering algorithm needs to select reasonable filter parameters in frequency domain to realize image enhancement, but it is very diffi- cult to select appropriate parameters. For the defocus image collected under the microscope, we hope to avoid adding new noise as much as possible and make the enhanced image quality better. Non-subsampled Contourlet transform (NSCT) ARTICLE INFO Received: 27 July 2021 Accepted: 17 September 2021 Available online: 24 September 2021 COPYRIGHT Copyright © 2021 Yongjun Liu, et al. EnPress Publisher LLC. This work is licensed under the Creative Commons Attribution- NonCommercial 4.0 International License (CC BY-NC 4.0). https://creativecommons.org/licenses/by- nc/4.0/ 50 has superior transformation function due to multi- scale and multi-directional, and can obtain more textures and edges of the image[9,10]. Therefore, this paper chooses NSCT transform. Firstly, the input image is transformed from spatial domain to fre- quency domain to obtain high-frequency sub-band and low-frequency sub-band; secondly, the low-fre- quency coefficients are enhanced by anti-sharpening mask algorithm, and the high-frequency coefficients are enhanced by adaptive filtering algorithm[11,12]; thirdly, NSCT is used to inverse transform the image to construct a new image; finally, in order to further enhance the effect of image details, the anti-sharpen- ing mask algorithm is used for image processing. 2. NonsubsampledContourlet trans- formation (NSCT) NSCT is conducive to better maintain the image edge information and contour and enhance the trans- lation invariance of the image. It is divided into two processes: multi-scale decomposition and directional filtering. The two processes are independent of each other. Figure 1 is the overall structure diagram. Figure 1. Overall structure diagram. In view of the translation invariance of La- place filter, the multi-scale nature of NSCT can be guaranteed by using non-subsampled pyramid for image decomposition. NSCT consists of low-pass decomposition filter module H0(z) and high pass decomposition filter module H1(z) = 1 − H0(z), and some low-pass synthetic filtering modules, which are composed of high pass filtering module G0(z) = G1(z) = 1. Figure 2. A schematic diagram of the decomposition of three layers of the pyramid. Figure 2 is the decomposition diagram of three layers of the non-subsampled pyramid, in which the ideal band-pass support region of the low-pass filter decomposed in layer j is [(π/2J), (π/2J)]2, while the ideal band-pass region of the high pass filter is the complement of the low-pass sub-bands [(−π/2J), (π/2J)]2/[(−π/2J), (π/2J)]2. Non-subsampling direction filter banks: the fil- ter is based on two channels and decomposes the co- efficients into different sub-bands through a tree like structure. The non-subsampling direction filter banks remove the down-sampling module and up-sampling module in the directional filter bank, so that the sam- pling results at different scales are consistent, and effectively improves the distortion in the sampling process. 3. Micro-nano optical image en- hancement algorithm based on NSCT Due to the multi-scale and multi-directional na- ture, NSCT has superior transformation function and can effectively improve the image quality. Therefore, we propose a micro-nano optical image enhancement algorithm based on NSCT to preprocess the defocus image obtained under the microscope. Firstly, NSCT decomposes the input image to generate a series of low-frequency and high-frequency components; sec- ondly, the low-frequency coefficients are enhanced by anti-sharpening mask algorithm; adaptive filtering algorithm is used for high-frequency coefficients; finally, all sub-bands are reconstructed by NSCT in- 51 verse transform. 3.1 NSCT low-frequency coefficient transfor- mation The low-frequency coefficients obtained by NSCT decomposition contain some basic informa- tion in the image. In this paper, the anti-sharpening mask algorithm is used to enhance it. (1) where, in the formula represent the input image and output image respectively, and K is the adjust- ment coefficient, taking 2. is the blurred image or approximate image, which can be obtained from equation (2). (2) Where, W is the filter window. 3.2 NSCT high-frequency coefficient conver- sion High-frequency coefficients represent detailed information, such as image edges and textures, but also contain a lot of noise. High-frequency coeffi- cient transformation aims to achieve enhancement based on noise reduction. High-frequency coeffi- cients with large values contain image edge details and should be retained. Noise is distributed in NSCT high-frequency coefficients with small values. Threshold noise reduction is to set an appropriate threshold, and noise reduction is performed by com- paring the absolute value of NSCT coefficient with the threshold. When the absolute value is greater than the threshold, NSCT coefficient is considered to contain important image information and is re- tained. When the absolute value of NSCT coefficient decreases or becomes 0 accordingly, it is used to filter noise. By using the coefficient processed by threshold, the NSCT inverse transformation is used to reconstruct the image, so as to eliminate the noise. Threshold de-noising eliminates the noise and retains the image coefficient as much as possible. Therefore, the key of de-noising is to determine the threshold. Less than the threshold is regarded as the noise to be suppressed, and more than the threshold is regarded as the edge information to be enhanced. (3) T1,k is the noise threshold in the K direction of the 1 layer, and P is the coefficient value of the sub- band. In this paper, an adaptive filtering algorithm based on BayesShrink is used. The threshold can be calculated by equation (4). (4) Where, c is a constant between 0 and 1, σ2 is the noise variance, and σ2 x is the signal variance. The standard deviation of noise can be calculated by equation (5). (5) Where, Cj,k is the high-frequency coefficient of the first layer of the image after NSCT decomposi- tion, i.e., take the median of the absolute value of NSCT coefficients, and calculate the noise standard deviation with 0.6745 as the adjustment coefficient. (6) Where, m and n represent the length and width of the image respectively, and the coefficient located in X(i, j) in the K direction image of layer 1 is Xi, j(1, K). The obtained threshold is used to suppress the noise. (7) Where, Xi,j is the coefficient of the high-frequen- cy sub-band of the original image and is the coeffi- cient of the processed image. The main steps of the algorithm are as follows: Step 1: NSCT decomposes the defocus image obtained under the microscope to generate a series of low-frequency and high-frequency components; Step 2: Calculate the low-frequency coefficient obtained by equation (1) to obtain a higher contrast of the image; Step 3: Adaptive filtering algorithm is used for threshold de-noising of high-frequency coefficients; 52 Step 4: The image is inversely transformed into the spatial domain by NSCT, and the new image is reconstructed; Step 5: The reconstructed image is further pro- cessed by anti-sharpening mask technology to obtain the optimized clear image. 4. Experimental results and analysis The experiments are carried out by using the traditional histogram equalization algorithm (HE), Retinex image enhancement algorithm (MSR) and the algorithms proposed in this paper. The experi- ments are carried out by using the standard nano grid image obtained under the actual optical microscope, the conductive probe and triangular probe sample images collected under the atomic force microscope. Figure 3, Figure 4, Figure 5 show the results of preprocessing three kinds of images by three algo- rithms. Among them, (a) is the original image, (b) is the image enhanced by histogram equalization al- gorithm (HE), (c) is the image enhanced by Retinex algorithm (MER), and (d) is the image enhanced by the algorithm proposed in this paper. 4.1 Subjective evaluation (1) Standard 500 nm raster image (2) Conductive probe image (3) Triangular probe image Figure 3. Grid image preprocessing results of three algorithms. As can be seen from Figure 3, Figure 4, Figure 5, compared with the original image, the clarity and contrast of the three images processed by HE algo- rithm are greatly improved, and the overall enhance- ment effect is better. However, the image processed by this algorithm has more noise, and the processing of the edge details of the image is not ideal. Some- times there will be local distortion and block phe- nomenon. For example, the standard 500 nm scale grid image is obviously noisy after being processed by HE algorithm; after the conductive probe and triangular probe are processed by HE algorithm, the probe tip appears local distortion. Figure 4. Preprocessing results of three algorithms on conduc- tive probe. Figure 5. Preprocessing results of three algorithms on triangular probe. Retinex algorithm decomposes the image into illumination image and reflectance image, and then eliminates or weakens the influence of illumination, so as to achieve the effect of image enhancement. Therefore, compared with the original image, the clarity and contrast of the three kinds of images pro- cessed by Retinex algorithm are greatly improved, and the overall enhancement effect is better. How- ever, due to the logarithmic processing, the display range of the bright area of the image is compressed, resulting in the weakening of the details of the im- age. For example, the details of the edges and cor- ners of the standard 500 nm scale grid, the needles of the conductive probe and the triangular probe are significantly weakened. Compared with the original image, the sharp- ness and contrast of the three images processed by the proposed algorithm are improved, and the over- 53 all effect is better; moreover, the algorithm adopts an adaptive filtering algorithm for high-frequency coefficients to determine the noise determination threshold suitable for different sub-band coefficients. Therefore, compared with the original image, the noise of the three images processed by the algorithm proposed in this paper is very low. 4.2 Objective evaluation Sharpness, contrast and peak signal-to-noise ratio are three objective indexes commonly used to measure the effect of image enhancement. These three indexes have been used for objective calcula- tion and evaluation in this paper. (1) Clarity It mainly reflects the vein part of the image. The larger is the measured value, the clearer is the repre- sentative image, as shown in equation (8). (8) ∆mx(i, j), ∆nx(i, j) are the difference points of points (i, j) in vertical and horizontal directions. (2) Contrast It represents the overall contrast intensity of the image. Generally, the greater is the contrast, the better is the image enhancement effect. The contrast calculation formula is: (9) Where, µx represents the average value of gray scale, M and N represent the number of pixels in im- age rows and columns. VAR is the variance of image, and the calculation formula is: (10) (3) Peak signal-to-noise ratio According to the ratio of maximum signal pow- er and noise power, the greater is the PSNR, the bet- ter is the image enhancement effect. The calculation formula of PSNR is: (11) Where, L is the gray level range, usually 255, and MSE is the mean square error, which can be cal- culated by equation (12). (12) Where, fij and Oij are the gray values of the input image and the output image at points (i, j), and M and N represent the number of pixels in the rows and columns of the image. Table 1, Table 2 and Table 3 respectively show the objective evaluation indexes of the images of standard nano grid, triangular probe and conductive Table 1. Comparison of objective evaluation indexes of three algorithms for raster image processing Evaluating indicators Original image HE algorithm Retinex algorithm Textual algorithm Clarity 3.63 21.02 13.65 5.71 Contrast 10.92 64.65 41.98 14.56 Peak signal-to-noise ratio — 10.84 15.79 20.49 Table 2. Comparison of objective evaluation indexes of three algorithms for processing conductive probe images Evaluating indicators Original image HE algorithm Retinex algorithm Textual algorithm Clarity 3.68 19.03 12.11 5.16 Contrast 20.35 64.50 29.64 26.76 Peak signal-to-noise ratio — 10.68 15.12 17.67 Table 3. Comparison of objective evaluation indexes of three algorithms for processing triangular probe images Evaluating indicators Original image HE algorithm Ritinex algorithm Textual algorithm Clarity 3.71 25.32 17.52 5.91 Contrast 13.39 64.63 28.80 15.51 Peak signal-to-noise ratio — 9.88 14.94 18.67 54 probe in atomic force microscope processed by three algorithms. (1) Standard 500 nm scale grid (2) Conductive probe (3) Triangular probe From Table 1 to Table 3, it can be seen that the clarity of different defocused images processed by HE algorithm and Retinex algorithm is relatively high. Although the clarity of defocused images pro- cessed by the algorithm proposed in this paper is not very high, it also improves the clarity of different defocused images to a certain extent and can effec- tively improve the image quality. The contrast of different defocused images pro- cessed by HE algorithm and Retinex algorithm is relatively high. Although the contrast of defocused images processed by the algorithm proposed in this paper is not very high, it also improves the contrast of different defocused images to a certain extent and can effectively improve the image quality. It can be seen from the peak signal-to-noise ratio data that compared with HE algorithm and Retinex algorithm, the proposed algorithm has the largest peak signal-to-noise ratio, the most information re- tained, the least distortion and better visual effect. Therefore, through the comparative experi- ments of standard nano-grid images obtained under the actual optical microscope, triangular probe and conductive probe sample images collected under the atomic force microscope, it can be seen that the im- age enhancement effect of the proposed algorithm is relatively better than that of histogram equalization algorithm (HE) and Retinex algorithm (MER), and can effectively improve the image quality. 5. Conclusion The out of focus images collected under mi- cro-nano optical microscope are not very clear in detail, edge and contour. Combined with NSCT im- age transformation method, which has multi-scale and multi-directional, and superior transformation function, more texture and edge of the image can be obtained. At the same time, an adaptive noise reduc- tion and enhancement algorithm of micro-nano opti- cal image based on NSCT is proposed to restore the defocus image obtained under the microscope. The experimental results of standard nano-grid images, triangular probe images and conductive probe imag- es show that the enhanced image effect of the al- gorithm proposed in this paper is better than histo- gram equalization algorithm (HE) and multi-scale Retinex algorithm, which not only improves the quality of defocused images, but is also conducive to the depth information recovery of subsequent defocused images. Conflict of interest No conflict of interest was reported by the au- thor. Acknowledgements Project: The next generation Internet tech- nology innovation project of Saier network of the Ministry of Education, “Emergency and Critical Intelligent Medical Cloud Platform based on IPv6” (NGII20170709). References 1. Hu R. Scanning near-field optical microscopy: Instrumentation and applications [Master’s thesis]. Nanjing: Nanjing University; 2018. p. 91. 2. Lv M, Yu Z. Study on automatic focusing algorithm of optical microscope. China Measurement & Test 2018; 44(6): 11–16. 3. He Z. Research on depth recovery based on defo- cused images [Master’s thesis]. Harbin: Northeast Agricultural University; 2014. p. 62. 4. Zhu Z, Yin H, Chai Y, et al. A novel multi-modality image fusion method based on image decomposi- tion and sparse representation. Information Sciences 2018; 432: 516–529. 5. Chen BH, Wu YL. An entropy-preserving histogram modification algorithm for image contrast enhance- ment. Proceedings of the 2017 IEEE International Conference on Applied System Innovation (ICASI); 2017; Sapporo. New York: IEEE; 2017. p. 1285– 1288. 6. Zhang S. Research on enhancement algorithms of low illumination images and videos based on Retinex 55 theory [Master’s thesis]. Nanjing: Nanjing University of Posts and Telecommunications; 2018. p. 76. 7. Gao Y, Liu G, Ma C. Dense hazy image enhancement based on generalized imaging model. 2018 IEEE 3rd International Conference on Image, Vision and Com- puting (ICIVC); Jun 27–Jun 29, 2018; Chongqing. New York: IEEE; 2018. p. 410–414. 8. Zhang D. Research on adaptive compressive sam- pling imaging based on wavelet transformation [Master’s thesis]. Chengdu: Southwest Jiaotong University; 2018. p. 69. 9. Liu Z, Xu T, Song Y, et al. Image fusion technolo- gy based on NSCT and robust principal component analysis model with similar information. Journal of Jilin University (Engineering and Technology Edi- tion) 2018; (5): 1614–1620. 10. Liu L. Research on image enhancement algorithm based on NSCT [Master’s Thesis]. Urumqi: Xinjiang University; 2016. 11. Pal NS, Lal S, Shinghal K. A robust framework for visibility enhancement of foggy images. Engineering Science and Technology 2019; 22(1): 22–32. 12. Chen B, Wu Y, Shi L. A fast image contrast enhance- ment algorithm using entropy-preserving mapping prior. IEEE Transactions on Circuits and Systems for Video Technology 2017; 20(2): 212–232.