Frontiers in Business, Economics and Management ISSN: 2766-824X | Vol. 1, No. 3, 2021 1 On the Determination of the Chip Nozzle Recognition System by Using Machine Vision Jing Qiu1, a, Yun Xu2, b and Siyi Liu3, c 1School of Mechanical Engineering, Sichuan University of Science & Engineering, Sichuan, 644000, China 2Sichuan Provincial Key Lab of Process Equipment and Control, Sichuan, 644000, China 3School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney, Sydney 2006, Australia aqiujingsichuan@163.com, bxuyunzigong@163.com, csliu9548@uni.sydney.edu.au Abstract: To solve the problem of chip damage caused by the using the wrong type of vacuum nozzle during the packaging of semiconductor chips. A recognition system of vacuum nozzle based on machine vision was proposed. In this research, 29 kinds of lifting nozzles are selected as test samples. The backlight intensity of two lifting nozzle images (one strong and one weak separately) is collected at the first beginning. Then, the Blob analysis method is using to analyze the weak backlighting image. The area of the lifting nozzle and the minimum outer rectangular feature can be obtained subsequently. To identify the shape of the liftin nozzle (round or square), the area ratio is calculated. At the same time, the minimum outer rectangular of the lifting nozzle is selected as the reference rectangle. Then, construct the measurement rectangle. The 2-dimensional size of the lifting nozzle is measured as well. Meanwhile, for the strong backlight image, the average value of the grayscale which located within the minimum outer rectangle is calculated. Therefore, the color (black, white, or beige) of the nozzle can be identified. Finally, the sample data is saved to the database as the sample database. During the recognition process, the shape, color, and size of the lifting nozzle being analyzing are using as the parameter to realize the condition inquire. The experimental results show that the recognition accuracy of this method is 98.85%, and the recognition time of one nozzle is around 1 second, which meets the requirements of practical application. Keywords: Vision, Image processing, Vacuum nozzle, Halcon. 1. Introduction In the semiconductor chip packaging process, picking and transferring the chip precisely is of vital importance. As it is a prerequisite to ensure that the packaging process can proceed normally [1]. The key to pick up and transfer chips correctly is using the corresponding type of vacuum nozzles. Due to the vacuum nozzles are in micro- size and the appearance of some nozzles are quite similar, it is difficult for human eyes to distinguish them. Once they are taken incorrectly, it is very likely to cause problems such as unable to lift the chips or the chip damage which is caused by the chip falling off from the nozzles during transfer. These problems seriously affect the accuracy and efficiency of the packaging process. However, machine vision is rarely used in the recognition and classification of chip vacuum nozzles. To ensure the smooth progress of chip packaging, a chip nozzle recognition system based on machine vision is proposed in this paper. Recognizing and classifying the nozzle to ensure that the packaging engineer will not take the wrong nozzle, before using the nozzle to pick them up. Halcon is an image processing library which is widely used in the industry. It has highly complete operators, efficient operation, and high compatibility with various industrial cameras. This greatly shortens the development period of machine vision applications. Therefore, the Halcon image processing library is selected to process the chip nozzle classification and recognition system, and Qt is used to develop desktop applications. Qt Creator 4.7.2 is a cross-platform integrated development environment for C++applications, which is widely used to develop GUI programs. 2. Set up of Nozzle Recognition System 2.1. The Composition of the Machine Vision System The chip nozzle classification and recognition system consisting of hardware and software, which is shown in Figure 1. In the software part, Qt Creator 4.7.2 is selected to be the software programming platform. Qt is used in the software interface part, connecting and querying the sample database. While Halcon is used to write algorithms such as image acquisition, serial communication, and image processing. The hardware mainly includes a CMOS camera, a telecentric lens, a backlight, a light source controller, and a PC. The detection accuracy of the system is 0.2mm. The camera used here is the acA3800-10gm-Baslerace which is a kind of black and white industrial camera produced by Basler in Germany. By sending serial command to the light source controller to change the brightness of the backlight, two nozzle images with one strong and one weak backlight intensity are continuously collected. Figure 1. Schematic diagram of the machine vision system 2 2.2. System Calibration The calibration of the camera is the cornerstone to ensure the accuracy of the measurement system. The conversion relationship between the image coordinates and the world coordinates needs to be obtained. This requires all the points on the image are corresponded with the actual points in the real world. A monocular camera is used to measure the size of the nozzle in this system. Therefore, it only requires a two- dimensional coordinate conversion. As shown in Figure 2, the world coordinate system is set as 𝑂 and the image coordinate system is then set as 𝑂 . The angle between these two coordinate systems is θ. Figure 2. Coordinate conversion diagram Their formula of translation and rotation the coordinate conversion is derivated as follow: x=x'cosθ - y'sinθ + x0 y=x'sinθ - y'cosθ + y0 (1) Two points in the world coordinate are set as 𝑥 , 𝑦 , 𝑥 , 𝑦 . Then these two points can be transferred to the image coordinate as 𝑥 , 𝑦 , 𝑥 , 𝑦 , which can be obtained from the formula 1. x1 - x2=(x1 ' - x2 ' )cosθ - (y 1 ' - y2 ' )sinθ y 1 - y2=(x1 ' - x2 ' )sinθ - (y 1 ' - y 2 ' )cosθ (2) Therefore, the angle θ between these two coordinates can be calculated by knowing the distance between two points in the actual world coordinate and the pixel distance in the image coordinate. When the θ is zero degrees, the actual size of an object 𝑙 can be calculated by the number of pixels n and the pixel equivalent d. The calculation formula is as follows: l=n ⅹ d (3) Where d is a unit pixels (pixel equivalent), representing the corresponding proportional relationship between the actual distance and the pixel distance, The pixel equivalent d can be obtained from the formula 4. d= l n =0.0151342 (4) The calibration method selected by this system is to place a coin with a diameter of 25mm in the detection area as a reference object to do further measurement. Then Halcon's calliper tool is used to measure its diameter. The number of pixels is measured to be 1651.88 pixels here, which is shown in Figure 3: Figure 3. Calibration results 3. Blob Analysis Blob analysis is to analyze the connected regions of the same pixels in the image. It not only calculates the total number, position, direction, area, and perimeter of the blob but also provide the topological structure among related blobs. Compared with traditional pixel-based algorithms, blob analysis uses faster line processing, since each line of the image is coded with run length to represent the adjacent target range. Important contents of Blob analysis include image segmentation, connected regions analysis, and feature collection. As the target and background are contrasted clearly on the nozzle image with weak backlight, an automatic threshold algorithm can be used here to divide these pixels to target pixels and background pixels. Where target pixels are assigned a value of 1, and background pixels are assigned a value of 0. Then connecting region analysis is performed to aggregate all adjacent target images into discrete blob connectors, while target pixels that are not adjacent will not be regarded as part of blobs. At this time, the grayscale of the image has been transformed into image features information. Finally, the interference blob is filtered out by setting the threshold for the feature and the nozzle area is obtained. The processing result is shown in Figure 4. The core program is showing below: binary_threshold (Image1, Region, 'max_separability', 'dark', UsedThreshold) connection (Region, ConnectedRegions) select_shape (Region, Region, ['area','row'], 'and', [30000,100], [9000000, 2000]) fill_up (Region, RegionFillUp) area_center (RegionFillUp, Area, Row1, Column1) smallest_rectangle2 (Region, Row, Column, Phi, Length1, Length2) Figure 4. Blob analysis results The area_center and smallest_rectangle2 operators are used to calculate the area of the nozzle and its smallest outer rectangle. Then the smallest outer rectangle is set as the ROI, and subsequent feature extraction will be performed in this ROI. 3 4. Feature Extraction 4.1. Shape Feature Extraction The profile of the lifting nozzle is divided into the internal and external profile. The internal profile is the profile of the through-hole of the nozzle. Another important thing is that both the shape of the internal and external profiles are neither circular or square. Therefore, this system focuses on the recognition of rectangles and circles. After some research and analysis, it is found that the rectangular area and the circular area have the following mathematical relationship with their smallest outer rectangle: α= Srectangle SMBR ≈1.00 Scircular SMBR ≈ π 4 ≈0.78 (5) In this formula, α is the ratio of the recognition area of the smallest outer rectangle, and 𝑆 is the area of the smallest outer rectangle. It can be seen that rectangles and circles can be identified by setting a suitable threshold for α. The area ratio threshold is initially set to 0.9. When α≥0.9, the shape is a rectangle, otherwise it is a circle. For the recognition of the outer profile shape of the nozzle, the area of the nozzle and its smallest outer rectangle area which are obtained in the blob analysis in section 3 are used here to calculate the corresponding α value directly. While the inner profile of the lifting nozzle needs to perform another blob analysis on the image. This analysis is base on the minimum outer rectangle of the outer profile. The aim is to obtain the area of the through-hole and the minimum outer rectangle on the lifting nozzle. Therefore, the alpha value of the through-hole shape can be calculated. The recognition result is shown in Figure 5. Figure 5. Shape recognition results 4.2. Extraction of Feature Dimension The dimension feature extracted by this system is the length and width of the lifting nozzle and the through-hole. The Halcon one-dimensional size measurement is adopted here to get the dimension. The measurement steps are as follows: 1) The gen_measure_rectangle2 operator is selected to get a rectangular measurement object on the weakly backlit image. 2) The measure_pos operator is selected to obtain the result of a one-dimensional measurement. “gen_measure_ectangle2“is the core operator in one- dimensional size measurement. There are 9 parameters in this operator. The centre point, length, width, and angle of the smallest outer rectangle information is obtained as references by using the blob analysis in section 3. Then the first 7 parameters are selected to form a rectangular ROI of this operator. At the same time, equidistant lines with the length of the ROI width and the spacing of a pixel will be established in the rectangular ROI. Then the midpoints of each equidistant line will be connected to form a directional segment. This segment is called a cross-section construction line. The direction of the cross-section construction line is the direction of the rectangular ROI which is shown in Figure 6. The eighth parameter of the gen_measure_rectangle2 operator is "Interpolation", which is also known as the interpolation algorithm. It is used to process the interpolation grayscale of non-horizontal or vertical equidistant lines. Therefore, the equidistant lines can pass through a complete pixel. There are three kinds of interpolation algorithms, namely “Nearest_neighbor interpolation algorithms “, „“ Bicubic interpolation algorithms “, and “ Bilinear interpolation algorithms“. Figure 6. Region of interest The nearest neighbour interpolation method assigns the grayscale of an adjacent pixel which is closest to the interpolated pixel to the exact interpolated pixel. The algorithm is the fastest algorithms but it has the lowest accuracy. The bicubic interpolation method assigns the sum of the weight convolution of 16 adjacent pixels of the pixel to the interpolation pixel. This method has high accuracy, but the algorithm execution efficiency is low. The bilinear interpolation method is using linear interpolation in two directions, basing on the grayscale of the 4 adjacent pixels of the interpolation pixel. In this way, the bilinear interpolation method balances accuracy and algorithm execution efficiency. Therefore, the bilinear interpolation algorithm is selected to perform to further steps by setting "Interpolation= bilinear". The last parameter of gen_measure_rectangle2 is an output handle. Through the measure_pos operator, all the properties values of the rectangular measurement object can be obtained. The actual size of the nozzle can be measured by multiplying the measured pixel size by the pixel equivalent. The measurement result is shown in Figure 7. The core program of size measurement is: *measure width. gen_measure_rectangle2 (Row, Column, Phi, Length1+20, 4 20, Width, Height,'bilinear', MeasureHandle) measure_pos (Image, MeasureHandle, 5, 40, 'all', 'all', RowEdge, ColumnEdge, Amplitude, width) realWidth:=with*d *measure height. gen_measure_rectangle2 (Row, Column, Phi+rad(90), Length1+20, 20, Width, Height,'bilinear', MeasureHandle) measure_pos (Image, MeasureHandle, 5, 40, 'all', 'all', RowEdge, ColumnEdge, Amplitude, height) real height:=height*d Figure 7. Measurement results 4.3. Color Feature Extraction The color of the nozzle is black, white, and beige. The CMOS camera of this system collects single-channel images, so color recognition algorithms based on RGB, HSV, and other color spaces cannot be used. However, through studying and analyzing the characteristics of the nozzle, it is found that under different backlights of intensities, the image brightness of the nozzle with different colors shows the difference as well. Therefore, the method of calculating the average grayscale to identify these three colors works. As one of the important concepts in image processing shows, the average grayscale reflects the brightness of the image. The larger the grayscale, the greater the brightness of the image, and vice versa. The mathematical expression is as follows: mean A = ∑ ∑ aij H j=1 W i=1 MN (6) Where A is an image with a size of MⅹN, and 𝑎 is the pixel value at pixel point (i, j). To find a suitable threshold of the average grayscale to distinguish these three colors, a simple experiment is designed in this study. In this experiment, 3 different color nozzles with black, white, and beige are selected to carry on the experiment. The light source controller in this system supports 700-level brightness adjustment. Therefore, the image collected under 7 integer-level backlight brightness from 100 to 700 is shown in Figure 8(a). The“ intensity‘ operator is used to calculate the average grayscale of the lifting nozzle in the smallest outer rectangular region obtained in the Blob analysis as discussed in Section 3,.The calculation result is shown in Figure 8(b), Figure 8 (a). Nozzle under 100~700 brightness Figure 8(b). The average grayscale of the nozzle under 100~700 brightness It can be seen that as the brightness of the backlight increases, the average grayscale of each color nozzle also increases. In the 700-level backlight, the average grayscale of the three color nozzles has the largest difference. At this time, the black nozzle is the darkest while the white nozzle is brighter than it. And the beige nozzle is the brightest, almost blending with the background. Therefore, the 700-level backlight is selected for color recognition in this system, and the grayscale initially thresholds are selected as formula 8. Figure 9 shows the color recognition results. val<150, black 150≤val<210, white 210≤val≤255, Beige (7) Figure 9. Color recognition results 5. Experimental Test and Result Analysis The characteristic information of 29 kinds of nozzles is shown in Table 1. 5 Table 1. Sample datasheet id color shape outerW outerH innerW innerH 1 black Outer:squre Inner:circle 11.992 6.006 1.000 1.000 2 black Outer:squre Inner:circle 5.095 2.87 1.038 1.038 3 Beige Outer:squre Inner: squre 7.581 7.58 2.502 2.494 4 black Outer:squre Inner:circle 8.975 6.973 0.875 0.875 5 black Outer:squre Inner: squre 10.181 10.121 2.418 2.41 6 White Outer:squre Inner: squre 5.044 5.026 1.504 1.465 7 Beige Outer:cirle Inner: cirle 4.614 4.614 1.47 1.47 8 White Outer:squre Inner: squre 10.148 10.13 2.466 2.406 9 black Outer:squre Inner:circle 9.995 7.959 0.847 0.847 10 black Outer:squre Inner:circle 8.024 7.996 0.859 0.859 11 black Outer:squre Inner:circle 8.007 6.073 0.832 0.832 12 black Outer:squre Inner:circle 5.957 5.936 0.892 0.892 13 Beige Outer:cirle Inner: cirle 4.579 4.579 1.528 1.528 14 Beige Outer:cirle Inner: cirle 10.248 10.248 2.544 2.544 15 black Outer:squre Inner:circle 7.437 3.993 1.000 1.000 16 black Outer:squre Inner: squre 5.067 4.971 1.489 1.459 17 White Outer:squre Inner: squre 5.052 4.984 1.493 1.455 18 black Outer:squre Inner:circle 12.005 11.981 1.447 1.447 19 black Outer:squre Inner:circle 6.987 6.955 0.832 0.832 20 black Outer:squre Inner:circle 11.854 6.824 0.836 0.836 21 black Outer:squre Inner:circle 6.919 6.904 0.83 0.83 22 black Outer:squre Inner:circle 7.665 6.929 0.888 0.888 23 black Outer:cirle Inner: cirle 4.474 4.474 0.902 0.902 24 White Outer:cirle Inner: cirle 3.803 3.803 0.895 0.895 25 White Outer:cirle Inner: cirle 3.803 3.803 1.17 1.17 26 White Outer:cirle Inner: cirle 3.94 3.94 1.301 1.301 27 black Outer:squre Inner:circle 6.991 5.261 1.001 1.001 28 White Outer:cirle Inner: cirle 3.942 3.942 1.526 1.526 29 black Outer:squre Inner:circle 14.408 7.534 1.002 1.002 The QtSql module is used to connect the sample database. Then the shape, size, and color features of the nozzle to be tested are extracted and saved to the sample database. In the nozzle recognition stage, the feature query can be realized by 6 using a conditional query. If the query is successful, the id string of the nozzle is returned, otherwise the string is empty. Among them, the shape and color features are precisely inquired. For the dimension query, the roughly inquired is used. And the query tolerance zone of the outer size features is queried as 0.2mm, while the query tolerance zone of the inter size features is queried only half of the outer one which is 0.1mm. The experiment is set to 3 groups, each with 29 different nozzles, and the experimental results are shown in Table 2. Table 2. Experimental results Total samples number Correct number Unable to recognition Wrong recognition Accuracy 87 86 1 0 98.85% As can be seen from table2, except for the unrecognized nozzle, the other nozzles can be classified correctly with no misjudgment. The overall recognition accuracy of the nozzle reaches 98.85%. The misidentified nozzle is No. 23, and the color of the nozzle was wrongly recognized as white. Its analysis revealed that the average grayscale of the No. 23nozzle was 164.392. After a series of repeating tests, it is found that the grayscale of the nozzle fluctuates in the range of 148.685~165.937, which slightly exceeds the upper thresholds limit of the average grayscale of the black nozzle which was set as 150 at the beginning. Therefore, reset the threshold of the average grayscale of the black and white nozzles to 170. Then the test was repeated and all the samples can be recognized correctly. The system software is shown in Figure 10. Figure 10. System software 6. Conclusion This paper is aiming at solving the problem that the lifting nozzles of chips cannot be recognized correctly in the semiconductor packaging process. A semiconductor chip nozzles recognition system based on machine vision is designed. The method of extracting nozzle shape feature using here based on area ratio, color extracting, and average grayscale. SQl sentence is used to perform a conditional query in the sample database to realize the classification and recognition of nozzles. Experimental results show that the recognition accuracy rate of the system is as high as 98.85%, and the accuracy rate is significantly better than human eye recognition, which can provide an effective way for the classification and management of chip nozzles. References [1] Zhang Buyang. Research on key technologies of visual positioning during semiconductor chip packaging [D]. Wuhan, Hubei: Huazhong University of Science and Technology, 2016. [2] Zhang Jingping. Design of QFN chip surface inspection system based on machine vision [D]. Nanjing: Southeast University, 2017. [3] Zhang Jichun, Zhang Senlin, He Yan. Design of HALCON- based optical fiber transceiver PCB board patch target detection and correction system [J]. Light Industry Machinery, 2019, 37(4): 67-72. [4] Xiao Jinhui, Xiang Xinjian. Halcon-based steel rule end-edge measurement system design[J]. Journal of Zhejiang University of Science and Technology, 2018, 30(6): 488-495. 7 [5] Yue Hao, Wu Shuanhu, Xu Jindong, et al. Design of medical bottle cap quality inspection system based on machine vision[J]. Instrument Technology and Sensors, 2019, 10: 83-107. [6] Fu Gui, Liu Liwen, Guo Xiangchuan. Research on bearing seat classification and recognition system based on machine vision[J]. Mechanical and Electrical Engineering, 2019, 36(10): 1115-1118. [7] Chen Junren, Gao Fengqiang, Xiong Kui. Design and implementation of a shoe mold glue spray system based on machine vision[J]. Internet of Things Technology, 2019, 5:40- 42. [8] Zhu Xianfeng, Pan Hongjun. Defect detection and one- dimensional measurement of toner cartridge based on Halcon[J]. Journal of Jilin University (Information Science Edition), 2014, 32(3) 308-315. [9] Zhang Chi, Li Yuanbo, Wang Meimei, Chen Long. Electroacoustic device solder joint positioning system based on Halcon[J]. Electric Welding Machine, 2017, 47(3): 49-53. [10] Zheng Liqiang, Zhao Fei, Guo Jiangtao. Jewelry recognition is based on machine vision[J]. Computer Technology and Automation, 2019, 38(3): 103-106.