Microsoft Word - 4-PETI#6082 32-41.docx Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 Mixed Noise Removal by Processing of Patches Rithu James 1,* , Harsha Appukuttan 1 , Liza Annie Joseph 2 1 Department of Electronics and Communication Engineering; 2 Department of Applied Electronics and Instrumentation Engineering, Rajagiri School of Engineering and Technology, India Received 20 July 2020; received in revised form 09 October 2020; accepted 07 December 2020 DOI: https://doi.org/10.46604/peti.2021.6082 Abstract Sonar images are degraded by mixed noise which has an adverse impact on detection and classification of underwater objects. Existing denoising methods of sonar images remove either additive noise or multiplicative noise. In this study, the mixed noise in sonar images, the additive Gaussian noise and the multiplicative speckle effect are handled by the data adaptive methods. A patch based denoising is applied in two phases to remove the additive Gaussian and multiplicative speckle noises. In the first phase, the adaptive processing of local patches is used to remove the additive Gaussian noise by exploiting the sonar image local sparsity. The PCA and SVD methods are used for denoising the noisy image patches and blocks of patches. In the second phase, the weighted maximum likelihood denoising of the nonlocal patches reduces the speckle effect by exploiting the non-local similarity in a probability distribution. Experiments on side scan sonar images are conducted and the results show the importance of removing both the additive and multiplicative components from the sonar images. Keywords: speckle, patches, blind denoising, blocks 1. Introduction Imaging sonars [1] is a point source that emanates acoustic plane wave of frequency in the range of 100 KHz to 2 MHz. Sonar images have rare edges as it is produced by the coherent nature of the scattering phenomenon and is influencedby a multiplicative speckle [2] component. The degradation model in [3] describes the multiple noise affected sonar images and the fully developed speckle amplitude has a multiplicative Rayleigh distribution [4]. The multiple noises in sonar images are the additive Gaussian noise and the speckle noise. Due to the distinct characteristics of mixed noise, it is not easy to handle both the degradation process simultaneously. The methods developed for one noise cannot effectively suppress the other noise effect. The design of classical filters is for the additive noise removal only. In some of the speckle suppression methods for a sonar image, the multiplicative noise model is converted into an additive one by considering the logarithm of the image, which is called the homomorphic approach and it assumes the noise is Gaussian. In the proposed method, a framework for the mixed noise removal in sonar images was done. The methodology assumes that the sonar images are degraded by multiple noises which hurt detection and classification performance. Wavelet transform uses a fixed basis for the representation of sonar images. In this work, both locally and globally adaptive basis were used for the sonar image sparse representation. In the second phase patch based on the mixed denoising algorithm, the additive white Gaussian noise removal is by the sparse approximation of local image patches in the first phase and despeckling is done on non-local patches in the second phase. * Corresponding author. E-mail address: rithujames@gmail.com Tel.: +919387211328 Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 33 2. Various Filtering Techniques Wiener filter is a classical filter designed for additive noise removal. Some of the speckle suppression methods take a homomorphic approach which converts the multiplicative noise into additive noise. But such a transformation reduces the contrast in the image and introduces a bias [5]. Adaptive filters are usually employed to reduce speckle noise in homogeneous areas of the sonar images while preserving texture and high frequency information in heterogeneous areas. Available adaptive classical speckle filters are the Lee, the Kuan, the Frost filters, the iterative Speckle Reducing Anisotropic Diffusion (SRAD) filter, and the multi-scale wavelet decomposition based on filters. A deep learning approach to directly estimate the speckle noise in the logarithmic domain according to the convolutional neural network (CNN) is established in [6-7]. In [8] the restoration of images was corrupted by impulse noise via the distribution-transformed network (DTN), which utilizes the convolutional neural network to learn pixel-distribution features from noisy images. However, to the author’s knowledge, very few publications on the removal of the multiple noises in sonar are available. The total variation image intensity is used for Gaussian and Poisson noise removal by Than et al. [9]. A mixed noise removal method utilizing Direction Weighted Median Filter and Block Matching 3D Filtering method is used by Takuro et al [10]. 3. Methodology First given a side scan sonar image, the unknown standard deviation �� of the image is estimated using the noise level estimation algorithm. The estimation of the noise level in the image and the non-blind denoising are the two processes of the blind denoising in sonar images. For a non-blind denoising algorithm, the noise level ��is assumed to be a known parameter. Nevertheless, for the blind denoising algorithm, the noise level �� need to be estimated together with the denoising process. Here a blind denoising method is used. Fig. 1 depicts the methodology adopted. Fig. 1 Methodology First the methodology adopted is to estimate the noise variance of the given sonar image, then patch based processing is done for both the additive Gaussian noise removal and the multiplicative speckle removal. The adaptive processing of local patches was used to eliminate the additive Gaussian noise. This uses Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) methods by processing the noisy image in a single patch and blocks of patches. The multiplicative speckle noise is then removed by processing of non-local patches. This uses the weighted maximum likelihood denoising of the non-local patches for reducing the speckle effect. Simulations on sonar images were conducted and the results show the Sonar Image Estimate the noise variance of the image Denoising by processing of local patches Non local patch processing for despeckling Denoised Image Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 34 effectiveness of mixed noise removal in sonar images. The different methods were compared by using different without-reference image performance evaluation criteria. 3.1. Noise level estimation For noise variance estimation, the sparse nature of sonar image is taken into consideration. Due to the natural sonar image redundancy, only low-dimensional subspace is spanned by it. The image is made up of several patches. If the data of patches��� � ∈ � × spans a subspace whose dimension is smaller than � × �, then such patches are called low-rank patches. In the proposed algorithm, low rank patches are selected. This avoids high frequency components in the noisy image. By doing this, a texture strength metric based on the local image gradient matrix and its statistical properties is used. The Principal Component Analysis [11] is used to estimate the noise level of the additive Gaussian noise from the selected patches. An input noisy image is decomposed into several patches in a raster scan form in the patch-based approaches [12]. The patch with the smallest standard deviation will have the least change of intensity among all the decomposed patches. The patches’ data model is: 1, 2, 3, , , = i i i My z n i= + … (1) where M represents the number of patches, �� denotes the �� patch which is noiseless with size � × �, and each patch is defined by its center pixel. �� is the observed patch represented as a vector. The observed patch is corrupted by noise vector �� assumed to be of zero-mean and variance�� �. The minimum variance direction is obtained as in [13]. This is the eigenvector corresponding to the minimum eigen value of the covariance matrix defined by: 1 1 M T y i i i y y M = ∑ = ∑ (2) where for the noisy patch y, ∑� represents its covariance matrix. 2 ( ) ( ) min y min z n ∑ = ∑ +λ λ σ (3) For the patch �� , ∑� is the covariance matrix obtained after removing the additive noise, and for the matrix∑, ���� (∑) is the minimum eigen value. In the method, the minimum eigen value of the covariance matrix ���� (∑) is taken as zero. The noise variance is estimated from the subspace which is spanned by the eigenvectors with zero eigen values of the covariance matrix ∑�. As a result it is assumed that the Gaussian noise has the same power in all direction and eigen values are the same. 2 ( ) n min y = ∑σ λ (4) The non blind denoising is performed with the variance obtained for the image. 3.2. Denoising by processing of local patches The patch based on the estimation method described above is used for getting the additive noise variance �� � for blind denoising. The noisy image is decomposed into patches using Principal Components [14] coefficients or SVD coefficients. Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 35 These coefficients help to represent the noisy image using fewer values and the clean coefficients are estimated using the LMMSE estimator. The procedure is to estimate the noise variance of the Principal Component coefficients, �� � for each patch using the maximum likelihood estimator. 2 22 1 0, [ ] i nn iM max y −= ∑ σσ (5) The coefficients are denoised using LMMSE estimator. 2 2 2 ( )l i i i n z y= × +σ σ (6) where �� is clean coefficient or the denoised image coefficient obtained after the removal of the additive Gaussian noise. First, divide the image into non overlapping patches [15] of size � × �. Next patches are transformed to one dimensional training vector. A matrix of training vector S of size � � × � is made by stacking these training vectors as columns. M is the number of patches. The noise free image is generated by denoising separately all the patches of the image. The eigenvectors of (�� � )(��) are taken as the PC basis functions. The principal components of the corresponding training vectors are represented by each column in the matrix. By projecting the training vectors in � onto the PC basis functions, the PC coefficients �( , !) for = 1, ⋯ , � � and ! = 1, ⋯ , � is got. The noisy image is represented by a fewer values using PC coefficients. For SVD based on the denoising obtain the singular values of (�� � )(��). SVD coefficients are calculated by projecting the training vectors in S onto corresponding basis function. The compression is obtained as SVD, converging the image information into fewer coefficients. The requirement of complex calculations is less and the memory needed is also reduced. Relevant data resides with the higher singular values. The left singular vectors are taken as the basis function in the process. To get back the denoised image, the entire decomposition process is reversed. The patches are arranged properly after the denoised training vectors were transformed back to patches for image reconstruction.In the approach of blocks of patch processing, denoising is performed in blocks [16]. A block matching technique is used to find the training samples of the local patch. This is by selecting a group of similar patches in the neighborhood by exploiting the images’ self-similarity. The entire S matrix is divided into blocks of the fixed size. The selection of the block size is critical. Block sizes that are too small results in the complexity of computations. On the other hand, block sizes that are too large results in dissimilar patches getting grouped together and high calculation errors. The block size selection should be such that keeps a tradeoff between the number of similar patches and the computation speed. Find the PC and the singular value coefficients after the block formation. Then denoise the image at the block level and then reconstruct the final image. 3.3. Despeckling by processing of non local patches In the second phase of denoising, the sonar image is assumed to be degraded by the Rayleigh distributed speckle noise model. s s s z u f= (7) Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 36 where �% is the observed noisy value at site s and &% its underlying noise-free value. The denoising process is a weighted maximum likelihood (WMLE) estimation problem [17] with the estimate &' % defined by: max logˆ ( , ) ( ) s s t t s u arg pu w s t z u= ∑ (8) where t is a pixel index and ((), *) is a data-driven weight [18] depending on the similarity between pixels with index s and t. Due to the distribution characteristics of the speckle effected sonar images, the non-local patch based on the filter is very efficient. The pixel values far apart can be averaged together according to the weight values or neighboring patch similarity. The new measure of similarity between pixels will better reflect the image content. For the non-additive noise model, the weight ((), *). 1/ ( , ) ( ) h s t w s t p u u v ∆ ∆ =≜ (9) where &∆, and &∆- are the sub-image extracts from the image u in the respective windows ∆% and ∆� , and ℎ > 0 is a scalar parameter. The pixel amplitudes 1% are modeled as independent and identically distributed according to the following Rayleigh distribution: 2 2 12ˆ ˆ ˆ( ) ( ) ( ) L L s s s sL s s LAL p A R A exp L R R − Γ = − (10) where �2 is the given reflectivity image and L the equivalent number of looks and the noise-free amplitude image 13 is the square root of the reflectivity image �2 . The following estimation is from the first order optimality condition, 2 ( , )ˆ ( , ) t t s t w s t A R w s t ∑ = ∑ (11) It must be held to maximize the WMLE according to weights derivation for multiplicative noise: 2 1 , , , , , , 2 2 , , , ˆ ˆ( , ) , L s k t k s k t k s k t k s k t k A A p A A R R a A A −   −      (12) with the final weights as: , , , , 2 1 ( , ) s k t k k t k s k A AL w s t exp log A Ah −    = −∑ +          (13) which can be used for the processing. The size of the number of significant weights to control the amount of filtering is related to h parameter in the NL means algorithm. The h parameter counter balances the invalidity of the patch independence assumption. For the processing side scan sonar image L is the mean to standard deviation square and h is the α quantile of the centered similarity criterion with α = 0.88 set in the experiments. Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 37 4. Performance Evaluation The images were taken from www.edgetech.com. The proposed method is assessed using the visual quality of the image for the detection of targets and by the quantitative measures. The different assessment parameters used are given below 4.1. Equivalent number of looks (ENL) In the sonar image intensity data, the speckle variance is proportional to the mean intensity squared. To quantify the quality of despeckled sonar images, the metric equivalent number of looks (ENL) is commonly used. With a smaller spread of values due to speckle, the larger will be ENL. The calculation of ENL is: 2ˆ ˆ ( )x x ENL = µ σ (14) where 78' and �8' are mean and standard deviation respectively of a homogeneous area, 9' in the despeckled image. The performance of despeckling technique is better if the ENL value is high. 4.2. Speckle suppression index (SSI) Let x and 9' be original and the speckle reduced sonar images respectively. The speckle suppression index (SSI) is denoted as the ratio of the SSI coefficient of variance (ratio of standard deviation to mean). ˆ( ) ( ) ˆ( ) ( ) var x mean x SSL mean x var x = (15) smaller values of SSI, if more speckle suppression. 4.3. Correlation coefficient (CC) The correlation coefficient (CC) for reflectivity image x and the denoised image 9' is calculated as follows: ˆ ˆ, ˆ ˆ[( )( )] x x x x x x E x x− − = µ µ µ σ σ (16) where 78 and 78' are mean values, �8 and �8' are standard deviations of reflectivity and denoised sonar images respectively. It can be found that the higher CC the better. 4.4. Speckle suppression and mean preservation index (SMPI) Therefore, SMPI is used for simultaneous estimation of speckle suppression and mean preservation capabilities of despeckling technique. It can be found that the lower SMPI the better. ˆ( ) ˆ( ) var x SMPI Q var x = (17) where ˆ( ) ( )Q K mean x mean x= + − (18) Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 38 ˆ( ( ) ( )) ( ) K max mean x min x mean x = − (19) 4.5. Coefficient of variation The coefficient of variation (CV) is the ratio of the standard deviation to the mean. The higher the coefficient of variation, the greater the level of dispersion around the mean [19]. It is generally expressed as a percentage. The lower the value of the coefficient of variation, the more precise the estimation is CV= standard deviation/mean (20) 5. Results and Discussion In this research, a patch based blind denoising procedure is followed to estimate the noise level in the sonar images which results in an improved denoising performance. The patch based denoising is applied to additive Gaussian and multiplicative speckle noises simultaneously. Additive noise removal helps to smoothen the image while speckle removal helps to get a better image with details preserved. The method enforces smoothness while preserving edges. Additive Gaussian noise is removed using the PCA and SVD basedlocal patch processing method and then the nonlocal patch processing is done on the same image to remove the speckle noise. After a number of simulations, for the additive noise level estimation, a patch of size 8x8 was used. The local patch based processing in the method uses a patch size of 8x8. In the block based processing, 8 adjacent patches are combined to form a block. For the nonlocal patch processing, a search window of size 24x24 and a similarity window of size 8x8 are used. The denoising capability of such mixed noise removal will help in avoiding false detection of targets, but there can be undetected ones. For the nonlocal patch processing which is used for the multiplicative noise removal, the window width is varied and the despeckling results are also compared. The visual quality performance comparison is shown in Fig. 2. It shows that the proposed method denoises without distorting useful image information as well as the important image edges. The additive denoising is approached using the sparse approximation of local image patches which is a characteristic of sonar images. For the multiplicative noise removal non-homomorphic approach was followed, so that the unnecessary introduction of the bias in the image was avoided. (a) Real Image (b) Wiener Filter (c) Wavelet (d) SRAD (e) SVD 1 st Phase (f) SVD 2 nd Phase (g) PCA 1 st Phase 1 (h) PCA 2 nd Phase Fig. 2 Demonstrating the visual quality difference in different methods applied on SSS image Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 39 Fig. 2 shows that the proposed method denoise without distorting useful image information as well as the important image edges and Table 1 depicts the numerical comparison and the graphical comparison in Fig. 3. Table 1 Quantitative comparison of the denoising technique with some of the existing methods for a real side scan sonar image of ENL 5.8422 Metric Wiener SRAD Wavelet SVD 1 st Phase PCA 1 st Phase SVD 2 st Phase PCA 2 st Phase ENL 6.9486 6.1805 5.5116 7.3797 7.3781 7.3800 7.3845 SSI 0.9171 0.9724 1.0297 0.8899 0.8900 0.8899 0.8896 CC 0.9646 0.9898 0.9933 0.9467 0.9467 0.9466 0.9465 SMPI 0.2023 0.9552 3.0623 0.1093 0.1102 0.1143 0.1139 COV 0.3794 0.4022 0.4260 0.3681 0.3682 0.3681 0.3680 Fig. 3 Comparison by metric values (on vertical axis) to grade the performance of the proposed Method The discarding of the lower coefficients results in the elimination of some noise associated with sensors along with the advantage of dimensionality reduction. As SVD eliminates noise components associated with both the sensors and the pings, it is more effective in denoising than PCA. The method is having the advantage of adaptively determining the local basis functions from the local image patches unlike the fixed ones for the entire image as in wavelet based technique. These basis functions are then being applied adaptively to local image patches. The metric value comparisons for the varying window width are depicted in the Table 2 and the graphical comparison in Fig. 4. The higher despeckling capability at the lower window width is at the cost of increased computational complexity. The improvement in the ENL is a clear indication of the speckle reduction in the second stage. The SMPI values are of small metric values to be clearly displayed on the selected scale of the graphical representation. Fig. 4 The metric values for varying window width Table 2 Quantitative comparison of denoising techniques Metric values for varying window size for a real side scan sonar image of ENL 5.8422 Window 2 3 4 5 6 7 8 ENL 9.1561 9.1584 8.7726 8.257 7.9164 7.7095 7.5502 SSI 0.79893 0.79883 0.8162 0.8413 0.85921 0.87066 0.8798 CC 0.8989 0.89473 0.90658 0.92205 0.93147 0.93715 0.94143 SMPI 0.033575 0.00942 0.016907 0.046955 0.012042 0.025944 0.039381 COV 0.33048 0.33044 0.33763 0.34801 0.35542 0.36015 0.36393 0 2 4 6 8 ENL SSI CC SMPI COV Wiener SRAD Wavelet SVD 1st Phase PCA 1st Phase SVD 2st Phase PCA 2st Phase 0 2 4 6 8 10 ENL SSI CC SMPI COV 2x2 3x3 4x4 5x5 6x6 7x7 8x8 Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 40 6. Conclusions The traditional filtering method on sonar images can only remove one kind of noise and the details of the image are blurred in various degrees. Mixed noise needs to be removed from the sonar images for further applications such as object detection. In this research, the sonar image local sparsity and its nonlocal similarity in a probability distribution are simultaneously exploited in removing the mixed additive Gaussian and multiplicative speckle noise respectively. Experimental results have demonstrated its better performance compared with other algorithms that remove only one type of noise. The performance can further be enhanced by patch size adaptation. Creating blocks of similar patches rather than adjacent patches can give good improvement on a statistical basis in the block adaptive method. The despeckling effects can be improved if the adaptation of the PCA and SVD are for the selected similar patches or having them applied in a non-local means filtered data. The proposed method in the research deals with only locally adaptive basis, but globally adaptive basis can also be analyzed as future research. The future research also includes the extensions on the potential of such denoised sonar images to be used for better sonar image segmentation and registration. Conflicts of Interest The authors declare no conflict of interest. References [1] P. Blondel, The Handbook of Sidescan Sonar, 1st ed. UK: Springer Praxis Publishing, 2009. [2] J. W. Goodman, “Some Fundamental Properties of Speckle,” Journal of Optical Society of America, vol. 66, no. 11, pp. 1145-1150, 1976. [3] A. K. Jain, Fundamentals of Digital Processing, USA: Prentice-Hall, Inc, 1989. [4] F. Maussang, J. Chanussot, A. Htet, and M. Amate, “Mean-Standard Deviation Representation of Sonar Images for Echo Detection: Application to SAS Images,” IEEE Journal of Oceanic Engineering, vol. 32, no. 4, October 2007. [5] A. Achim, A. Bezerianos, and P. Tsakalides, “Novel Bayesian Multiscale Method for Speckle Removal in Medical Ultrasound Images,” IEEE Transacton Medical Imaging, vol. 20, no. 8, pp. 772-783, August 2001. [6] R. W. Liu, Y. Lu, F. Chen, and L. Xie, “Learning a Deep Convolutional Network for Speckle Noise Reduction in Underwater Sonar Images,” Proceedings of the 2019 11th International Conference on Machine Learning and Computing, February 2019, pp. 445-450. [7] U. Anithaa, S. Malarkkanb, G. D. Anbarasi Jebaselvia, and R. Narmadhaa, “Sonar Image Segmentation and Quality Assessment Using Prominent Image Processing Techniques,” Applied Acoustics, vol. 148, pp. 300-307, May 2019. [8] G. Li, F. Zhang, and Q. Liu, “Distribution-Transformed Network for Impulse Noise Removal,” Journal of Shanghai Jiaotong University (Science), June 2020. [9] N. H. T. Dang, S. D. Dvoenko, and V. S. Dinh, “A Mixed Noise Removal Method Based on Total Variation,” Informatica, vol. 40, no. 2, pp. 159-167, July 2016. [10] T. Yamaguchi, A. Suzuki, and M. Ikehara, “Detail Preserving Mixed Noise Removal by DWM Filter and BM3D,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. EI00-A, no. 11, pp. 2451-2457, November 2017. [11] X. Liu, M. Tanaka, and M. Okutomi, “Single-Image Noise Level Estimation for Blind Denoising,” IEEE Transactions on Image Processing,” vol. 22, no. 12, pp. 5226-5237, December 2013. [12] S. Pyatykh, J. Hesser, and L. Zheng “Image Noise Level Estimation by Principal Component Analysis,” IEEE Transactions on Image Processing, vol. 22, no. 2, pp. 687-699, February 2013. [13] C. M. Bishop, Pattern Recognition and Machine Learning, New York: Springer, 2006. [14] V. P. S. Naidu and J. R. Raol, “Pixel-level Image Fusion Using Wavelets and Principal Component Analysis,” Defense Science Journal, vol. 58, no. 3, pp. 338-352, March 2008. [15] R. James and M.H. Supriya, “Sonar Image Denoising using Adaptive Processing of Local Patches,” 2015 International Symposium on Ocean Electronics (SYMPOL), IEEE Press, November 2015, pp. 1-7. [16] Y. He, T. Gan, W. Chen, and H. Wang, “Adaptive Denoising by Singular Value Decomposition,” IEEE Signal Processing Letters, vol. 18, no. 4, pp. 215-218, April 2011. Proceedings of Engineering and Technology Innovation, vol. 17, 2021, pp. 32-41 41 [17] C. A. Deledalle, L. Denis, and F. Tupin, “Iterative Weighted Maximum Likelihood Denoising with Probabilistic Patch-Based Weights,” IEEE Transactions on Image Processing, vol. 18, no. 12, pp. 2661-2672, December 2009. [18] A. Buades, B. Coll, and J. Morel “A Non-Local Algorithm for Image Denoising,” 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), IEEE Press, July 2005, pp. 60-65. [19] C. Kervrann, P. Perez, and J. Boulanger, “Bayesian Non-local Means Image Redundancy and Adaptive Estimation for Image Representation and Applications,” SIAM Conference on Imaging Science, July 2008. Copyright© by the authors. Licensee TAETI, Taiwan. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY-NC) license (https://creativecommons.org/licenses/by-nc/4.0/).