Template for an Acta IMEKO event paper ACTA IMEKO ISSN: 2221-870X December 2016, Volume 5, Number 4, 19-23 ACTA IMEKO | www.imeko.org December 2016 | Volume 5 | Number 4 | 19 The application of texture features to quality control of metal surfaces Konstantin Trambitckii1,2, Katharina Anding2, Galina Polte2, Daniel Garten3, Victor Musalimov1, Petr Kuritcyn2 1National Research University of Information Technologies, Mechanics and Optics, Av. Kronverksky 49, 197101 Saint-Petersburg, Russia 2Ilmenau University of Technology, Gustav-Kirchhoff-Platz 2, 98693 Ilmenau, Germany 3Society for Production Engineering and Development (GFE e.V.), Naeherstiller Strasse 10, 98639 Schmalkalden, Germany Section: RESEARCH PAPER Keywords: image processing; texture features; surface quality Citation: Konstantin Trambitckii, Katharina Anding, Galina Polte, Daniel Garten, Victor Musalimov, Petr Kuritcyn, The application of texture features to quality control of metal surfaces, Acta IMEKO, vol. 5, no. 4, article 4, December 2016, identifier: IMEKO-ACTA-05 (2016)-04-04 Section Editor: Lorenzo Ciani, University of Florence, Italy Received September 22, 2016; In final form November 7, 2016; Published December 2016 Copyright: © 2016 IMEKO. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited Funding: The research project, which forms the basis of this paper, was supported by the Thuringian Ministry of Economy, Employment and Technology (TMWAT) with means from the European Social Fund (ESF). The author Konstantin Trambitckii was supported by the Michail Lomonosov grant Corresponding author: Konstantin Trambitckii, e-mail: konstantin.trambitckii@tu-ilmenau.de 1. INTRODUCTION A surfaces quality control is an important step during the production process of metal parts. It is necessary in order to check whether surface quality meets the requirements. The quality control gives the possibility to detect parts with certain defects and can help to increase the outcome and the reliability of good products. A surface roughness assessment is one of the main ways to control the quality of a surface. There are two main groups of roughness assessment methods: contact and non-contact measurements. Non-contact measurements in comparison with contact measurements have the advantage that a measured surface remains undamaged after the end of the process. Nowadays optical industrial cameras are widely used in the non-contact quality assessment of metal surfaces. The measurement speed of this group of surface assessment methods is higher in comparison with contact methods. If the surface of a measured part has a curved shape and the camera lens has a narrow depth of field (because of the short focal length), some regions of the image, obtained by the camera, can be out-of-focus. These regions have less information about the surface, because fewer details can be obtained from them. Trambitckii et al. used different texture features to segment and remove such regions [1]. When the out-of-focus regions were removed, further surface quality analysis can be performed. Authors of different papers proposed various methods of optical surface quality control. The image texture analysis was already used in the field of quality assessment of metal surfaces. Laws [2] and Haralick [3] descriptors are used for detection of various defects of metal surfaces. Laws developed an approach, based on the calculation of texture energy ABSTRACT Quality assessment is an important step in production processes of metal parts. This step is required in order to check whether surface quality meets the requirements. Progress in the field of computing technologies and computer vision gives the possibility of visual surface quality control with industrial cameras and image processing methods. Authors of different papers proposed various texture feature algorithms which are suitable for different fields of image processing. In this research 27 texture features were calculated for surface images taken in different lighting conditions. Correlation coefficients between these 2D texture features and 11 roughness 3D parameters were calculated. A strong correlation between 2D features and 3D parameters occurred for images captured under ring light conditions. ACTA IMEKO | www.imeko.org December 2016 | Volume 5 | Number 4 | 20 parameters. Convolution masks are used to filter the source image. Then the texture energy descriptors are computed by applying non-linear window operations to the filtered images. Finally, all the descriptors are combined to achieve rotation invariance of the resulting feature. To evaluate Haralick’s descriptors, firstly the grey level co-occurrence matrix (GLCM) is calculated for an image. Values of the GLCM are calculated considering a spatial location of neighbourhood pixels. GLCMs can be created for different directions and for different distances taking into account locations of neighbourhood pixels. After that step for each GLCM Haralick’s descriptors (energy, contrast, correlation, etc.) are calculated. Alegre et al. used these both groups of descriptors as input vectors for k-NN classifiers to divide stainless steel surfaces into two classes based on their roughness quality [4]. Alves et al. used Haralick’s descriptors to define roughness of the surface and used multilayer perceptron (artificial neural network) to classify these surfaces into three classes [5]. Another way to calculate descriptors for classification is the wavelet transform. Alegre et al. [6] proposed to apply the Haar transform to decompose original surface images. As the next step, Haralick’s features were calculated for these decompositions. This method showed a reliable classification of surfaces based on their finish quality. Quality assessment can be based on the Fourier transform as well. Tsai et al. used the two-dimensional Fourier transform to classify cast surfaces with different roughness into several groups [7]. Naïve Bayes and neural network classifiers were implemented to solve this task. The negative side of an application of the Fourier transform is computational complexity of the method. Another group of texture features is represented by focus features. These features are successfully used in autofocus systems of cameras, microscopes, etc. Authors of several papers developed and suggested different focus features. The focus features are essential for the autofocus algorithms, giving numerical values to a whole image or to different regions. The focus features give the maximum response to the sharpest image from a set of images. The higher the contrast of the specific region, the higher the value of the focus feature will be. It is possible to use these features for in-focus surface images as the descriptors of surface roughness. The different surface quality will influence the surface visual appearance on the image. These surfaces will have different contrast and intensity variations, which can be described with the help of the focus texture features. Nayar [8] developed a feature, which is based on the Laplacian operator. This feature was called as the sum modified Laplacian (SML). The SML can be used as a descriptor for sharpness of specific regions. Pech et al. [9] suggested to use variance of the Laplacian operator convoluted with an input image. Thelen et al. [10] created a modification of Nayar’s SML feature, additionally taking into account the intensity information from diagonal pixels. Due to the increased amount of points used for calculation, this improved feature showed more reliable results on noisy images. In this paper the focus is on the quality control of metal surfaces using various texture features. For this goal a correlation between different texture features and roughness parameters was calculated. It gives the possibility to find the most correlated pairs, suitable for non-contact visual surface quality assessment. The algorithms were tested on surfaces produced by a milling and drilling operation. Chapter 2 gives detailed information about the data acquisition system for both 2D and 3D surface images. Chapter 3 and 4 contain descriptions of some texture features and roughness parameters, used in this research. There is an explanation of the correlation calculation algorithm in Chapter 5. In Chapter 6 the main results of this work are presented. Chapter 7 contains brief conclusions, which summarize the performed research. 2. DATA ACQUISITION In this research, metal parts with cone shaped surfaces were used. Following the main concept of this experiment, processed test surfaces of metal parts were marked, as shown in Figure 1. That gives a possibility to find the same regions in both 2D images and 3D surface data. In our research the size of the region of interest is around 1×1 mm2. 3D roughness information of metal surfaces was obtained with the Alicona 3D Infinite Focus G4 measurement system. A lens with the magnification of 20× was used. The lateral resolution (along X- and Y-axis) of the measurement system with 20× lens is 2.93 µm, the vertical resolution (along the Z-axis) is around 100 nm. 2D images were obtained with an optical camera which was built-in in the same system. In the optical measurements of metal surfaces light plays an important role because of the complex reflectance characteristics of such surfaces. In this research, different light sources were used. The light through the lens system of the Alicona scanner was used, as well as a ring light in different modes. The main advantage of the ring light is rotation invariance of shadows on surface images relatively to the lighting source. A sample image of the surface with the ring light source, used in this research, is shown in Figure 2. The observed workpiece area is a counter sink. The cutting speed of the tool, used to produce these drill hole, varied from 175 to 185 m/min. Figure 1. Marked regions in both 2D image (top) and 3D surface (bottom). ACTA IMEKO | www.imeko.org December 2016 | Volume 5 | Number 4 | 21 3. 2D FEATURES EXTRACTION 27 different texture feature maps were calculated for each image of the metal surfaces. The most correlated features are described in the present chapter. Algorithms were written in MATLAB. 3.1. Thresholded gradient The thresholded absolute gradient is calculated using the following equation [11]: ,),(),1( 1 1 ∑∑ = = −+= M i N j GRAT jiIjiIF (1) while ν>−+ ),(),1( jiIjiI , where M and N are the numbers of horizontal and vertical pixels of the image, I(i,j) is the grey level intensity of pixel (i,j), and v is the gradient threshold. 3.2. Absolute central moment Shirvaikar proposed to calculate the absolute central moment of an image. That texture feature showed robust work on the test images. It has peak values when the region of interest has high contrast. The absolute central moment FACM is defined by [12]: ,)( 1 0 ∑ − = ⋅−= L k ACM khkF µ (2) where L is the number of grey levels in the image, μ is an average value of grey levels of the image, and h(k) is the value of the histogram h for the k-th grey level. 3.3. Spatial frequency Frequencies for rows and columns are defined by [13] ∑∑ = = −+ ⋅ = M i N j jiIjiI NM RFreq 1 1 2),(),1( 1 (3) and .),()1,( 1 1 1 2∑∑ = = −+ ⋅ = M i N j jiIjiI NM CFreq (4) Thus, the spatial frequency is defined as .)()( 22 CFreqRFreqFSFRQ += (5) 3.4. Laplacian based features As described in the introduction, Nayar suggested to use the SML feature, based on the Laplacian operator. The modified Laplacian feature is calculated by using the following equation [8]: ,),(),(),(2 ),(),(),(2 1 1 stjiIstjiIjiI jstiIjstiIjiIF M i N j ML +−−− ++−−−= ∑∑ = = (6) where st is a distance between pixels, used to compute the feature. Thelen et al. improved this feature by taking into account the intensity information from diagonal pixels [10]. It is defined by .),(),(),(2 2 1 ),(),(),(2 2 1 ),(),(),(2 ),(),(),(2 1 1 stjstiIstjstiIjiI stjstiIstjstiIjiI stjiIstjiIjiI jstiIjstiIjiIF M i N j XML −+−+−− +++−−−− ++−−− +−−−−= ∑∑ = = (7) The improved feature showed the more robust results for noisy images. 3.5. Brenner’s gradient Brenner et al. proposed to calculate the sum of squares of the second derivatives, which is larger than a threshold value v [14]: ( ) ,),(),2( 1 1 2∑∑ = = −+= M i N j BREN jiIjiIF (8) while ( ) ν>−+ 2),(),2( jiIjiI . 4. 3D ROUGHNESS PARAMETERS The surface quality can be estimated using roughness parameters established in international standards [15]. In this research the following ISO roughness parameters were used: Sa (arithmetical mean deviation of the assessed surface), Sq (root mean square deviation of the surface), Ssk (skewness of the surface), Sku (kurtosis of the surface), Sv (maximum valley height of the surface), Sp (maximum peak height of the surface), Sz (maximum height of the surface, i.e. the difference between the highest peak and the deepest valley), S10z (10 point maximum height), Sdq (root mean square surface slope) and Sdr (developed interfacial area ratio). Along with the ISO parameters listed above another roughness parameter from another source was used – Ssc (mean summit curvature) [16]. All these parameters were calculated for marked areas of the 3D surface, which correspond to the same areas of the 2D images. 5. CORRELATION EVALUATION For this research 6 sample surfaces were taken and marked. These marks can be visible both on the images obtained with a 2D industrial camera and on the surfaces obtained with the Alicona system, as shown in Figure 1. That gives the Figure 2. Sample 2D image of metal surface. ACTA IMEKO | www.imeko.org December 2016 | Volume 5 | Number 4 | 22 opportunity to calculate different texture features and roughness parameters for the corresponding regions of the surface. As the first step, regions of interest were set for every obtained 2D image and 3D surface. These regions of interest correspond to the marked areas of surfaces. Then marked areas were divided into subregions. The subregions for the 2D images and the 3D surfaces have different sizes because of their different original dimensions. The marked areas of the 2D images were divided into the subregions of 35×35 pixels, the marked areas of the 3D surfaces were divided into the subregions of 140×140 points. These region sizes result in the equal sizes for the corresponding arrays. It is an important condition for further estimation of correlation. In the next step, the texture features were calculated for each subregion of the 2D images. For every image, every single texture feature results in an array of size corresponding to the number of subregions. Roughness parameters for the 3D surfaces were calculated the same way. After the texture features and roughness parameter arrays were calculated, correlation coefficients between every pair of the parameters were estimated. Having two data arrays of similar sizes gives the possibility to find the correlation coefficient, which shows statistical relationships between these data values. If the coefficient is equal to 0, that means that two values are linearly independent. The closer the coefficient to -1 or 1, the stronger the correlation between these two values. In other words, the closer the coefficient to these values, the stronger the linear dependency they have. For the correlation coefficient an interpretation of the Brosius criteria [17] was used. These criteria are listed in Table 1. This interpretation of the correlation coefficients gives an easier explanation whether values have weak or strong correlations. 6. RESULTS As the first step, a set of texture features was tested against two sets of roughness parameters, calculated for raw surfaces and calculated for surfaces with a removed curvature shape. The correlation for the second set was very weak (correlation coefficients are below 0.39), and in the next research only the raw surfaces data were used. Images under ring light conditions gave interesting results. The texture features calculated for these images showed strong (up to 0.85) correlation between the roughness parameters. The correlation for a ring light environment is stronger than for light through lens conditions, see the box plot in Figure 3. The correlation coefficients between 27 texture features and 11 roughness parameters were calculated. It results into an array of 297 pair-wise correlation coefficients. For all the coefficients absolute values were taken, as some of them have negative values. Then all pairs were sorted from the highest to the lowest values of the correlation coefficient. This information was used to draw a plot, which shows the distribution of the correlation coefficients for all 297 pairs, see Figure 4. The X-axis represents the absolute correlation coefficient value. The Y-axis is an index of pair, sorted by the correlation coefficient value in descending order. The performed studies showed that for our conditions the most correlated roughness parameters are Sa, Sq, Sv, Sp, S10z, Ssc, Sdq and Sdr. The most correlated texture features are the absolute central moment, the thresholded gradient and the spatial frequency. An example of the most correlated pair is shown in Figure 5. An average value of the correlation coefficient between the average central moment and the average roughness (Sa) is equal to 0.85. Linear regression (the Table 1. Correlation coefficients interpretation. Absolute value of coefficients Interpretation 0 no correlation 0 < r < 0.2 very weak 0.2 < r < 0.4 weak 0.4 < r < 0.6 medium 0.6 < r < 0.8 strong 0.8 < r < 1 very strong 1 perfect Figure 3. Box plot for ring light and light through lens conditions. Figure 4. Distribution of correlation coefficients. ACTA IMEKO | www.imeko.org December 2016 | Volume 5 | Number 4 | 23 red line in Figure 5) was performed to get the relationship between these two values. 7. CONCLUSIONS This research showed the successful results of using a computer vision approach for roughness assessment of a metal surface with help of different texture features extracted from 2D images. Such method of non-contact quality control gives a possibility to detect parts with certain defects in a fast and reliable way and can increase the outcome of good products. Overall, 27 texture features were calculated for surface images taken in different lighting environments. Correlation coefficients between these texture features and 11 roughness parameters were calculated. Strong correlations between features and parameters showed up for images under ring light conditions. Future work will analyse surface images taken with other industrial cameras and different lens systems. The results showed that surface defects, which are reflected in a deviation of roughness parameters, can be detected with fast operating low-cost 2D cameras using 2D texture analysis instead of using time-consuming and expensive 3D measurement devices. ACKNOWLEDGEMENT The research project, which forms the basis of this paper, was supported by the Thuringian Ministry of Economy, Employment and Technology (TMWAT) with means from the European Social Fund (ESF). The author Konstantin Trambitckii was supported by the Michail Lomonosov grant. The responsibility for the content of this paper lies with the authors. REFERENCES [1] K. Trambitckii, K. Anding, G. Polte, D. Garten, “Elimination of out-of-focus regions for surface analysis in 2-d colour images”, Proc. of Ilmenau Scientific Colloquium, 2014, vol. 58. [2] K. Laws, “Texture energy measures”, Image Understanding Workshop, DARPA, 1979. [3] R. M. Haralick, K. Shanmugam, I. Dinstein, “Textural Features for Image Classification”, IEEE Trans. on Systems, Man and Cybernetics, vol. SMC-3, No. 6, 1973, pp. 610-621. [4] E. Alegre, J. Barreiro, M. Castejón, S. Suarez, “Computer Vision and Classification Techniques on the Surface Finish Control in Machining Processes”, Lecture Notes in Computer Science, vol. 5112, pp. 1101-1110. [5] M.L. Alves, E. Clua, F.R. Leta, “Evaluation of Surface Roughness Standards applying Haralick Parameters and Artificial Neural Networks”, IEEE Trans. on Systems, Signals and Image Processing, April 2012, pp. 452-455. [6] P. Morala-Argueello, J. Barreiro, E. Alegre, S. Suarez, V. Gonzalez-Castro, “Qualitive surface roughness evaluation using Haralick features and wavelet transform”, Annals of DAAAM & Proceedings, January 2009, pp. 1241-1242. [7] D.M. Tsai, C.F. Tseng, “Surface roughness classification for castings”, Pattern Recognition, vol. 32, No. 3, March 1999, pp. 389–405. [8] S.K. Nayar, Tech. report: Shape from focus. 1989. [9] J. Pech, G. Cristobal, J. Chamorro, J. Fernandez, Diatom autofocusing in brightfield microscopy: a comparative study. 2000. [10] A.Thelen, S. Frey, S. Hirsch, P. Hering, Interpolation Improvements in Shape-From-Focus for Holographic Reconstructions With Regard to Focus Operators, Neighborhood-Size, and Height Value. 2009 [11] A. Santos, C. Ortiz De Solórzano, J.J. Vaquero, J.M. Peña, N. Malpica, F.Del Pozo, “Evaluation of autofocus functions in molecular cytogenetic analysis”, Journal of Microscopy, December 1997, vol. 188, No. 3, pp. 264–272. [12] M.V. Shirvaikar, “An Optimal Measure for Camera Focus and Exposure”, Proc. of the Thirty-Sixth Southeastern Symposium on System Theory, 2004, pp. 472-475. [13] A.M. Eskicioglu, P.S. Fisher, “Image Quality Measures and Their Performance”, IEEE transactions on communications, 1995, vol. 43, No. 12. [14] J.F. Brenner, B.S. Dew, J.B. Horton, J.B. King, P.W. Neirath, W.D. Sellers, “An automated microscope for cytologic research,” J. Histochem. Cytochem, Vol. 24, pp. 100-111, 1971. [15] ISO 25178: Geometric Product Specifications (GPS) – Surface texture: areal. [16] K.J. Stout, P.J. Sullivan, W.P. Dong, E. Mainsah, N. Luo, T. Mathia, H. Zahouani, “The development of methods for the characterization of roughness on three dimensions”, Publication No. EUR 15178 EN of the Commission of the European Communities, Luxembourg, 1994. [17] F. Brosius, “SPSS 8“, International Thomson Publishing, 1998, p. 503. Figure 5. A graph showing the relationship between the average surface roughness (Sa) and the absolute central moment (FACM). The application of texture features to quality control of metal surfaces