AP06_5.vp 1 Introduction The vision and image processing systems are widely em- ployed in current manufacturing processes. These kinds of systems provide measures to inspect and control the manu- facturing process of the producing location. In addition, the vision systems make automatic inspection and process ex- amination possible with the help of “intelligent” software components. The project, on which this paper is based, is initiated in such a circumstance in order to associate and sim- plify the qualtiy control of free-form surface manufacturing, specially shining water taps in sanitary industries. The first stage in the fabrication of water taps is the casting of the rough part. The material that is mainly used is brass. After the casting process some other machining processes, like drilling, milling and threading are carried out. Then the water tap is ground and polished sequentially in order to obtain high surface quality. Finally, the end product is finished by electroplating. It is crucial to find out the potential defects existing on the workpiece surface after grinding and polish- ing before the final electroplating. If a defect is found on a part after electroplating, manufacturer has to painfully discard this part or remove the electroplated coating. In this case removing the coating is a very expensive process because its harmful to the evironment. Therefore cost will be saved if the defects can be detected at an early stage. In addition, it is always useful to find out which type of a defect has been iden- tified. Once the type of a defect is known, the decision can be made whether the part is discarded or can be retouched with proper compensatory engineering processes and if an adjustment of a previous machining process is necessary. So far the tasks of defect inspection and categorization are performed by human operators in a traditional “see and evaluation” way. It is a labour-intensive and therefore also cost-intensive job. This process is necessarily automated to improve the efficiency of defects inspection, releasing worker from unpleasant working environments, and finally reducing the overall manufacture cost, specially in the countries where wages are high. To automate the defects inspection and classi- fication, a vision system is installed and integrated into the manufacturing chain. In our project, the inspection process takes place after the end of the grinding and polishing pro- cesses. If no defects are found on the surface, the workpiece is accepted for the next processing step. Otherwise, it is classi- fied automatically in order to determine if a removal of the defect is possible or the workpiece should be rejected directly. The vision system consists of a carrier, a camera system, a lighting system, other accessories and the software. The sys- tem hardware is responsible to provide a constant lighting enviroment and obtain the digital images under this constant circumstance. The software provides the solution to examine the images from the camera system, locating and classifing the defects on workpiece surfaces. The challenge in our project now is to characterise and determine the type of defects from predefined categories and additional foulings (which are called pseudo-defects in our project) on the surface. To this end, this paper presents measures and considerations regarding both theoretical and practical aspects, to efficiently classify all defects as well as a separation of real-defects from pseudo defects. 2 Automatic classification system 2.1 Defects definition The defects have been generally divided into 15 classes according to their physical attributes and the consequent handling operations. Fig. 1. gives samples for all defect cate- gories. All defects mainly can be split up into two major categories, real-defects and pseudo-defects that are actually foulings, like dust or oil, on the surface or shades caused by uneven lighting on the free-form surfaces. The first ten defects in Fig. 1. are real-defects and the last five are pseudo- -defects. The pseudo-defects are causing no quality problem; while the real-defects should be critically picked out because they not only spoil the aesthetic aspect but sometimes also result in malfunction of final products. The vision system con- siders both kinds as failures at first and distinguish them in the classification phase. Therefore, two indice have to be taken into account, the overall right classficaition ratio and wrong classification ratio between the real-defects and pseu- do-defect. In addition, the wrong classification ratio from real- to pseudo-defects is more crucial than that from pseu- do- to real-defects. In the previous case, a product with real- 8 © Czech Technical University Publishing House http://ctn.cvut.cz/ap/ Acta Polytechnica Vol. 46 No. 5/2006 Quality Control in Automated Manufacturing Processes – Combined Features for Image Processing B. Kuhlenkötter, X. Zhang, C.Krewet In production processes the use of image processing systems is widespread. Hardware solutions and cameras respectively are available for nearly every application. One important challenge of image processing systems is the development and selection of appropriate algorithms and software solutions in order to realise ambitious quality control for production processes. This article characterises the development of innovative software by combining features for an automatic defect classification on product surfaces. The artificial intelligent method Support Vector Machine (SVM) is used to execute the classification task according to the combined features. This software is one crucial element for the automation of a manually operated production process. Keywords: quality control, image processing, defect classification, support vector machine. -defects will be accepted as qualified and put to the next processing step, electroplating. This product cannot be sold to the customer after electroplating. All manufacturing costs, including the expensive electroplating, are wasted. Even if it is remediable, the electroplated coating has to be removed at first. This coating remove process is also expensive. In the second case, the product just run through one additional round of grinding or polishing, although this would not be necessary. 2.2 Classification framework Four basic steps are necessary for the classification to work: defects location, defects segmentation, feature ex- traction and classification. These four steps are executed sequentially in the system. Fig. 2. shows the framework of the whole system. At the beginning, the system detects and locates the flaw in the greyscale image obtained from the camera system. Before encoding the bitmap into some meaningful feature values, the area of the defect in the image should be pre-pro- cessed and segmented. In this phase we get defect image units that are ready for the subsequent feature extraction. This paper will mainly focus on the later two parts: feature extrac- tion and classification. The feature extraction is the most important part in the system. It defines the rules to describe and express the defects © Czech Technical University Publishing House http://ctn.cvut.cz/ap/ 9 Acta Polytechnica Vol. 46 No. 5/2006 mechanical damage casting peel pore and embedding solids fine lined mark porosity crack swirl vapour lock swirl fragment turning defect fluff dust burned residues grease residues polishing shade Fig. 1: Samples of 15 defect categories defects location defects segmentation feature extraction feature 1 feature 2 feature 3 feature n classifier Fig. 2: The framework of the automatic identification and classification system inside an image in a form that the classifier can understand and utilize to distinguish one class from others. The features are not limited to the shape feature, but also can be the tex- ture features and some statistical features of the segmented images. After that, the features are applied as the training data to the classifier. Many artificial intelligent techniques have the capability of multi-class classification, e.g. Multi- -Layer Perceptron (MLP) [1], Support Vector Machine (SVM) [2], Learning Vector Quantization (LVQ) etc. SVM is used in this paper. 3 Feature extraction and classification 3.1 Review of previous work The theoretic methodology and practical considerations of this automatic classification system are presented in our previous article [3]. The article discussed various feature ex- traction methods and their mathematical bases, including shape features, features through different filter banks, statisti- cal features based on co-occurrence matrix [4]. In general, feature extraction digitizes the defect images in a way that enlarges the distinctions among categories and discards the similarities at the same time. The filter bank technology is a widely used method for pattern recognition. A group of fil- ters, which is called a filter bank, is employed and each filter in the bank contains intensity variations over a narrow range of frequency or orientation, specifying the regularity, coarse- ness and directionality of the original image [5]. The energy of each filtered image is used as a feature. The statistical fea- tures are, in comparison, extracted from the co-occurrence matrix of the original image. Fig. 3 and Fig. 4 illustrate fil- tered images by Gabor filter bank [6] and co-occurrence matrices respectively. In addition, the classification structure was also intro- duced to combine these various features as a complete system, see Fig. 5. First, the classification task is conducted by using single kind of features, i.e. only one switch is closed in Fig. 5. The experimental results are shown in Table 1. It can be con- cluded from Table 1 that the shape feature is not suitable for this application. There are two reasons for that. The first one is that there are no clear differences in the shape between some defects, for example, Fluff and Pore, Fine Lined Mark and Polishing Shade. The second one is that the geometric in- formation of some kinds of defects cannot be exactly defined. For example, it is not easy to describe and to differentiate the shape of a Swirl and a Polishing Shade. The pattern informa- tion is more effective than the simple geometric information in this sense. The Gabor features excel all other features, obtaining 74.4 % overall classification ratio. The results of sta- tistical parameters and Laws filter are also very impressive and competitive. In the second step, we combined features of different tech- nologies together, i.e. more than one switch in Fig. 5 is closed. Through combining Gabor and statistical features, a 82.3 % classification ratio can be obtained. The real-to-pseudo wrong classification ratio of this model is about 2.5 %. The wrong classification ratio of pseudo-to-real is about 6.6 %. Table 2 shows the combined matrix of defects classification using Gabor and statistical features. One item in the combined ma- trix denotes the number of defects that are classified from one class (indicated by row elements) to another class (indicated by column elements). For example, we can conclude from the combined matrix in Table 2 that 494 from 573 Pores (Label 3) 10 © Czech Technical University Publishing House http://ctn.cvut.cz/ap/ Acta Polytechnica Vol. 46 No. 5/2006 Fig. 3: The Gabor filter bank. The left image is the original image of a defect “vapour lock” and the images on the right side are the fil- tered images at different central frequencies. The orientation is 0°. Fig. 4: The co-occurrence matrixes of a pore, a vapour lock, a pol- ishing shade and a casting peel (from left to right) Feature Feature Num. Training classification ratio (%) Testing classification ratio (%) Shape 8 46.6 % 61.8 % Laws 25 88.7 % 70.1 % DCT 3x3 9 98.9 % 62.3 % DCT 5x5 25 97.9 % 64.9 % Gabor 16 98.9 % 74.4 % Statistical 15 76.5 % 72.1 % Table 1: Training and testing classification ratio of varied features are correctly classified, 9 of them are recognized as Vapour Lock (Label 8) and 64 are identified as Polishing Shade (Label 105). In comparison, 39 defects, which are recognized as the Pore (Label 3), are actually Fluffs (Label 101). The combined matrix of an idealistic classification system has only non-zero values on the diagonal, meaning that none defects are wrongly classified. The last row of Table 2 means the reliability of the prediction. For instance, if the reliability of prediction of Pore (Label 3) is 73.0 %, it is 73.0 % probably correct when the system predicts that it is a pore. The last column is the right classification ratio of each class. Refer to the original paper [3] for more details about the feature extraction technologies, classifier design, numerical results and analyses. © Czech Technical University Publishing House http://ctn.cvut.cz/ap/ 11 Acta Polytechnica Vol. 46 No. 5/2006 5 6 6 tr a in in g p a ir s 1 8 2 0 te st in g p a ir s Shape Laws filters 3x3 DCT 5x5 DCT Gabor filters Stat. features feature selection 8 25 9 25 16 15 classification SVM1 SVM2 SVM3 SVMm m=k(k-1)/2 Voting C1 C2 C3 C4 C14 C15 feature extraction Fig. 5: The structure of the classification system Labels 1 2 3 4 5 6 7 8 9 10 101 102 103 104 105 N Class. Ratio (%) 1 19 0 4 0 0 0 0 1 0 0 0 0 0 0 2 26 73.1 2 0 21 7 0 0 0 0 0 0 0 0 0 0 0 2 30 70.0 3 0 1 545 1 1 0 0 4 0 0 11 4 3 0 3 573 95.1 4 0 0 2 40 0 0 0 2 0 1 0 0 0 0 3 48 83.3 5 0 0 19 0 37 0 0 1 0 0 0 0 0 0 0 57 64.9 6 0 0 8 1 1 17 0 0 0 0 3 0 0 0 2 32 53.1 7 0 0 20 0 0 0 29 0 0 1 3 0 1 0 3 57 50.9 8 0 0 26 0 0 0 0 58 0 0 0 0 1 0 2 87 66.7 9 0 0 3 0 0 0 0 0 8 0 0 0 0 0 0 11 72.7 10 0 0 3 0 0 0 0 0 0 178 0 0 0 0 3 184 96.7 101 0 0 26 1 0 0 0 0 0 0 62 0 1 0 11 101 61.4 102 0 0 18 0 0 0 0 0 0 0 0 24 0 0 7 49 49.0 103 0 0 29 0 0 0 0 4 0 1 1 0 38 0 6 79 48.1 104 0 0 6 0 0 0 0 0 0 0 0 0 1 29 5 41 70.7 105 0 0 31 4 0 0 0 0 0 1 5 0 7 4 393 445 88.3 Reliability (%) 100 95.5 73.0 85.1 94.9 100 100 82.9 100 97.8 72.9 85.7 73.1 87.9 88.9 – 82.3 Table 2: Classification result obtained by combining Gabor features and statistical features 3.2 Further work From Table 2, we can see that the four most common de- fects are pore (Label 3), polishing phade (Label 105), turning defect (Label 10) and fluff (Label 101). The pore and polish- ing phade constitute majority of all defects. Some defects, e.g. crack and casting peel, have only few samples in the defect database. Based on this database, the classifier was inclined to classify the major categories correctly and neglected the wrong classification of minor categories because the right classification of major categories contributes much more to the overall classification ratio than the minor categories. The samples of the database should be enriched in order to make the classifier to identify also defects in minority. Another fact is that some kinds of defects are not part of the new database due to improvements of both the manu- facturing process and the vision system. For example, the mechanical damage occurs rarely in the new manufacturing process in our project and fluffs are nearly eliminated if the workpieces are blowed by high pressure air before they are fed to the vision system. Therefore, the number of classification target categories is reduced. A new database of defects was built by collecting the samples on the current manufacturing process. In this new database, the manual classification of pseudo-defects was reorganized with more consideration of their appearance. Besides the features shown in Table 1, we use additionally the average and standard deviation of greyscale value of the defect image as greyscale features. The greyscale features should be localized considering different size of the various defects. The greyscale information are obtained in four areas in the defect image respectively, see Fig. 6. In this case, the number of greyscale features is eight, two of each area. Table 3 shows the classification results by combining Gabor features, statistical features and additional greyscale features. The overall classification ratio is 81.1 % even that all defect types are balanced in batch compared to the result based on the old database. The real-to-pseudo wrong classifi- cation ratio is about 2.6 % while the pseudo-to-real is about 5.1 %. Human trained operators are achieving a similar rate. 4 Software implementation Fig. 7 shows the architecture of the implemented software system. This software system provides a platform to manage defects data and evaluate the feature extraction technologies and classifier design. The defect database contains informa- tion for all collected defects, including manual classification results, defect images and other necessary entries. Each de- fect corresponds to a record in the database. The feature extraction reads out the information from the database and calculates the features of the defect. The features and the 12 © Czech Technical University Publishing House http://ctn.cvut.cz/ap/ Acta Polytechnica Vol. 46 No. 5/2006 Fig. 6: Greyscale information in four areas Labels 2 3 4 6 7 8 10 101 102 103 104 105 N Class. Ratio (%) 2 95 4 0 0 4 0 0 0 1 0 0 4 108 88.0 3 0 453 8 0 0 20 0 42 10 1 2 4 540 83.9 4 0 20 217 4 4 8 0 0 2 1 0 0 256 84.8 6 0 0 2 87 2 3 2 0 0 3 1 0 100 87.0 7 4 7 6 0 137 0 0 1 4 1 0 0 160 85.6 8 0 1 7 0 11 197 0 0 8 0 8 0 232 84.9 10 0 0 0 0 0 0 568 0 0 0 0 8 576 98.6 101 2 23 0 0 0 0 0 129 10 0 0 0 164 78.7 102 7 57 9 15 23 9 3 14 460 63 26 70 756 60.8 103 0 2 1 2 0 0 10 0 16 268 0 1 300 89.3 104 4 0 0 0 3 8 0 0 7 5 254 19 300 84.7 105 1 3 4 0 7 0 3 6 61 5 13 273 376 72.6 Reliability (%) 84.1 79.5 85.4 80.6 71.7 80.4 96.9 67.2 79.4 77.2 83.6 72.0 – 81.1 Table 3: Classification result by combining Gabor features, statistical features and greyscale information. 1572 defects for training, 3868 for testing in which 3138 are classified in the right way. manual classification results are used for training and testing the SVM models. The models can be used then by the classi- fier to perform identification task. The feature extraction and classifier design comprise the core component, which is called Opticlass, of the software. The software system has the capability of self-learning. If the manufacturing process or the vision system is altered, the only part we should change is the classification model. We need no modification of the software code, it is only necessary to build new models using the new collected defects database. First, the training and testing files are generated by the soft- ware system based on the new database. After that, the SVM models are produced by the training and testing files. Finally, the new created models replace the old models in use. Opticlass is a relatively independent software component. It can be one part of the off-line program and can also be © Czech Technical University Publishing House http://ctn.cvut.cz/ap/ 13 Acta Polytechnica Vol. 46 No. 5/2006 Defect Database Management Graphic User Interface (GUI) Classifier Opticlass Feature Extraction Models T ra in in g Figure 7: Software architecture Fig. 8: User interface of the software system directly integrated into the software of the vision system to online classify the occurred defects. Moreover, Opticlass is extendable. With enrichment of feature extraction methods and various implementations of classifiers, Opticlass is not strictly limited to our project to classify defects on water taps in sanitary industries but also applicable to other digital im- age based classification tasks. Fig. 8 shows the user interface of the software system. 5 Summary This article introduces a vision system that inspects and identifies the defects on water taps in sanitary industrie. With this vision system the quality control process is fully auto- mated without intervention of human operators. The vision system is divided into two parts, the hardware part and the software part. The hardware part cleans up product surfaces and then takes photos under a constant illuminating con- dition. The software part does the image processing and realizes the automatic identification. This paper presents the two major software components, features extraction and classifier design. A large range of methods is introduced to present defect images. It is possible to obtain about 81.1 % overall classification ratio for 13 predefined defects by combining the Gabor filter features, statistical features and grayscale features. The software structure and implementation is also de- scribed. The collected defects are stored in a uniform data- base. Therefore, the data of defects can be easily handled by the software. The core components, feature extraction and classifier design, are implemented independently. They ser- vice not only our simulation software platform offline but also can be integrated into the vision system and work online. The modularity of the software ensures that the vision system has the capability of self-learning and extendibility to other applications. Acknowledgment This research and development project is funded by the “Bundesministerium für Bildung und Forschung” (BMBF) within the framework of “Research for the production of tomorrow” and supervised by the project supporter of the BMBF for Production and Manufacturing technologies (PFT), research centre in Karlsruhe. References [1] Haykin, S.: Neural Networks – A Comprehensive Foundation. Macmillan College Publ. Co., 1995. [2] Vapnik, V.: The nature of statistical learning theory. New York : Springer, 2000. [3] Zhang, X., Krewet, C.; Kuhlenkötter, B.: Automatic Classification of Defects on the Product Surface in Grinding and Polishing. International Journal of Machine Tools and Manufacture. Vol. 46 (2006), No. 1, p. 59–69. [4] Haralick, R. M.: Statistical and Structural Approaches to Texture. In: “Proceeding of IEEE Bd. 67”, 1979, p. 786–804. [5] Laws, K.: Textured Image Segmentation. PhD Disserta- tion, University of Southern California, 1980. [6] Daugman, J. G.: Complete Discrete 2D Gabor Transfor- mation by Neural Networks for Image Analysis and Compression. IEEE Transactions on Acoustics, Speech and Signal Processing. Vol. 36 (1988), No. 7, p. 1169–1179. Dr.-Ing. Bernd Kuhlenkötter phone: +490 231 755 5615 e-mail: bernd.kuhlenkoetter@udo.edu M.-Eng. Xiang Zhang Dipl.-Ing. Carsten Krewet Industrial Robotics and Handling Systems Robotics Research Institute Dortmund University Otto Hahn-Strasse 8 44221 Dortmund, Germany 14 © Czech Technical University Publishing House http://ctn.cvut.cz/ap/ Acta Polytechnica Vol. 46 No. 5/2006