Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 10, No. 1, 2024 32 Research on Grading Method for Jin Guan Apples based on Machine Vision Yi Shi * Department of Computer Science and Technology, Shandong University of Technology, Linyi, Shandong, CO 255000, China * Corresponding author Email: 98540492@163.com Abstract: As the demand for apples increases, so do the expectations for apple quality. To address the time-consuming and labor-intensive nature of apple grading, this paper focuses on the Jin-Guan apple variety and applies machine vision techniques to extract three key features: maximum transverse diameter, shape index, and surface color. A backpropagation neural network (BP neural network) is employed to achieve accurate grading of the apples. First, by analyzing the relationships between the RGB color space components in the original images, the apple images are extracted using the method of comparing the R channel value to the B channel value. Gaussian filtering is then applied to remove image noise. Second, the complete apple contour is extracted using an improved Sobel operator method. The maximum transverse diameter and shape index of the apple are calculated based on the pixel coordinates along the contour, and the apple's surface color is determined using a nearest-neighbor classifier. Finally, a BP neural network model is established and trained to perform the grading. A total of 160 apples were used for training, and 140 for testing. The results show that the grading accuracy reached 92.14%, and the grading efficiency was approximately 1.7 times that of a skilled human grader. This grading method provides a technical reference for apple grading processes. Keywords: Machine Vision; Apple Grading; BP Neural Network; Image Processing. 1. Introduction With the improvement of living standards, the demand for fruits has significantly increased. Apples, as one of the most commonly consumed fruits, are highly favored by consumers. In recent years, China's apple production has been rising steadily, with a total output exceeding 40 million tons in 2017. However, only a small portion of apples is exported due to the inconsistent quality and relatively outdated grading technology. In contrast, many foreign countries have already implemented online monitoring for certain fruits [1-2]. Currently, apple grading in China primarily relies on manual methods, which are time-consuming, labor-intensive, and subject to considerable subjective bias. As machine vision technology gradually permeates the agricultural sector from its industrial origins, it has begun to be applied in the inspection of agricultural products [3]. Numerous researchers have utilized machine vision technology in the grading of apples. Li Long et al. [4] employed machine vision techniques to assess the appearance of apples under static conditions. Chen Yanjun et al. [5] established a linear model to map the diameter pixel counts from three different surface images of apples to their actual diameters, which reflected the size of the apples to some extent. Leemans et al. [6] proposed a detection method for Jin-Guan apples based on color information. Sofu et al. [7] introduced a real-time processing-based automatic sorting and quality inspection system for apples. To enhance grading accuracy, many researchers have integrated machine vision with various classification algorithms. Bao Xiaoan et al. [8] employed an improved artificial neural network algorithm for apple grade determination, achieving an accuracy of 88.9%. Kavdr et al. [9] combined a backpropagation (BP) neural network with statistical classifiers such as decision trees, K-nearest neighbors, and Bayesian methods to classify the surface quality of apples, yielding favorable results. Mizushima et al. [10] proposed an apple image segmentation algorithm based on a linear support vector machine (SVM) and Otsu's thresholding method. This approach automatically adjusts the classification hyperplane using linear SVM, effectively mitigating the effects of varying lighting conditions and fruit color changes. Testing on 300 apples with different color features demonstrated that the segmentation error was less than 2%. Jana et al. [11] introduced a novel grading method for apples and other agricultural products, extracting texture features from the gray-level co-occurrence matrix and statistical color features from segmented images, followed by training and prediction based on SVM. Sergio A et al. [12] implemented automatic apple grading by combining multi- image features with a weighted K-means clustering algorithm. Huang Chen et al. [13] proposed a grading method that integrates multiple apple features, such as fruit diameter and color, utilizing decision trees and SVM for grading based on different features and performing decision fusion on the results of both grading methods. Currently, the technology for grading based on machine vision has not been widely promoted domestically, primarily due to insufficient grading accuracy. To enhance grading accuracy, this paper focuses on the Jin-Guan apple variety. By employing machine vision techniques, the maximum transverse diameter, shape index, and surface color are extracted, and a backpropagation (BP) neural network is utilized for training and testing to achieve accurate grading of the apples. 2. The Grading System The Jin-Guan apple grading system is illustrated in Figure 1, which primarily consists of a conveyor structure (1), an industrial camera (3), a computer (4), a dark chamber (5), and a light source (6). 33 The conveyor structure is composed of a conveyor belt, a DC motor, and a driver, designed for transporting apples. The industrial camera used is model SHL-019, operating at a frame rate of 30 FPS with a resolution of 1280 × 1024. The computer utilized is a personal computer with an Intel® Core™ i5-6300HQ CPU running at 2.3 GHz and equipped with 12.0 GB of RAM. The dark chamber is employed for capturing images of the apples, with the overhead view of the dark chamber shown as (7) in Figure 1. It is evident from the figure that three industrial cameras are positioned around the apple at 120° intervals to accurately capture the apple's shape parameters. The light source consists of a ring-shaped LED array, ensuring uniform illumination around the apples. First, the apples to be inspected are placed at equal intervals on the conveyor belt within the conveyor structure. Driven by the motor, the apples are transported into the dark chamber. When the apples reach the center of the dark chamber, a photoelectric switch is triggered to control the industrial cameras to capture images of the apples from different directions. The captured images are then transmitted to the computer for processing and analysis. Finally, based on the grading results, the system sends corresponding control signals to the sorting device to achieve the grading of the apples. This paper will focus on the image preprocessing and grading module, which includes background segmentation, Gaussian filtering, contour extraction, and grading using the BP neural network. Figure 1. Schematic Diagram of the Jin-Guan Apple Grading System1.Conveyor Structure,2. Apple,3. Industrial Camera, 4. Computer, 5. Dark Chamber,6. Light Source,7. Overhead View of the Dark Chamber 3. Image Preprocessing To accurately extract the external features of Jin-Guan apples, a series of preprocessing steps is necessary, including background segmentation, Gaussian filtering, and contour extraction. 3.1. Background Segmentation To facilitate the extraction of apple images, the inner walls of the dark chamber used for grading are painted black. Despite this, reflections from the industrial camera, conveyor belt, and inner walls of the dark chamber can still be observed in the captured images. If the background is not addressed, it will adversely affect the extraction of subsequent feature parameters, ultimately impacting the grading results. Therefore, it is essential to segment the background of the captured apple images. Typically, the steps involved in background segmentation include converting the image into a different color space, selecting a suitable single channel to transform the color image into a grayscale image, applying threshold segmentation to obtain a binary image, and finally reconstructing the color image. This method can be relatively complex and sometimes yields suboptimal results. In this study, we analyze the characteristics of the three channel components (R, G, and B) in the RGB color space for the images collected by the grading system. As shown in Figure 2, the histograms of the R, G, and B channel components at the line 1 position of the apple image reveal that the R channel component is maximized in the apple region, while the B channel component is minimized. In contrast, the R, G, and B channel components of the black background area are quite similar, and in the conveyor belt area, the B channel component is maximized while the R channel component is minimized. Based on these characteristics, we selectively choose areas where the R channel component exceeds the B channel component. Figure 3 shows the image after background segmentation. Figure 2. Histogram of the Three Channel Components Under Line Contours Figure 3. Apple Image After Background Segmentation 34 3.2. Gaussian Filtering Before extracting the contours of the apples, the images must first be converted into grayscale. Through comparative analysis, it was found that the image obtained from the R channel component exhibits higher contrast between the apple region and the background. However, due to noise caused by the light source and system structure, some noise points still persist, as shown in Figure 4. In this study, the Gaussian filtering method is employed to eliminate these noise points. Figure 4. Noise Points in the Image Gaussian filtering is a linear filtering technique commonly used to handle sharp noise. The filter template is calculated using the Gaussian formula, which is based on the two- dimensional Gaussian distribution function as shown in Equation (1). Where, (x, y) represents the template coordinates, σ is the correlation coefficient for x and y, and G (x, y) denotes the weight at the point (x, y). 2 2 22 2 1 ( , ) 2 x y σG x y e πσ   (1) In this study, a 7×7 template is used to establish a coordinate system with the center of the template as the origin. By substituting the obtained coordinates into Equation (1), the weight for each point can be calculated. Within the 7×7 neighborhood, the pixel values at each position are multiplied by their corresponding weights and summed to obtain the Gaussian blurred value for the target pixel. After traversing all points in the image, the Gaussian filtered image is produced. Figure 5 illustrates the image after Gaussian filtering. Compared to Figure 4, there is a significant reduction in noise points within the background. Figure 5. Apple Image After Gaussian Filtering 3.3. Contour Extraction Contour extraction is crucial as it directly influences the extraction of features. The most commonly used contour extraction operators include the Canny and Sobel operators. While the Canny operator is effective in eliminating noise, it is prone to detecting false edges. Therefore, this study selects the Sobel operator for extracting the contours of the apples. The Sobel operator is characterized by its simplicity and ease of computation. Traditional algorithms typically use only horizontal and vertical templates, which may not effectively extract contours of diagonal edges. In this study, an improved Sobel operator is employed for contour extraction, with the following steps: (1) In addition to the existing horizontal and vertical operators, operators at 45° and 135° are introduced, resulting in four types of operators, as shown in Figure 6. (a) Cross operator (b) Longitudinal operator (c)45° operator (d)135° operator Figure 6. Templates of the Four Operators (2) The templates shown in Figure 6 are convolved with the pixel points on the image, The absolute values of the results are then summed to obtain the pixel value at the center of the image. (3) The contours of the apples are binarized using a fixed threshold method. In the histogram of the gradient image, the apple contours are concentrated in regions with higher gradient values. By selecting an appropriate threshold, the apple contours can be effectively extracted, as described by the formula in Equation (2). ' ' 255 ( , ) ( , ) 0 G x y T f x y others       (2) Where, ' ( , )f x y represents the pixel value of the processed point (x, y), ' ( , )G x y denotes the gradient value at point (x, y), and T is the selected threshold. Figure 7 illustrates a comparison of the apple contours extracted before and after the improvement of the Sobel operator. The figure clearly shows that the contours extracted using the improved Sobel operator are more complete. (a) Apple contour extracted by Sobel operator (b) Apple contour extracted by improved Sobel operator Figure 7. Apple Contours Extracted Before and After the Improvement of the Sobel Operator. 4. Feature Extraction The grading system presented in this study classifies Golden Crown apples based on three features: maximum diameter, shape index, and external color. The maximum diameter and shape index are extracted from the contours of the apples, while the external color is derived from the original images. 35 4.1. Maximum Diameter Extraction To measure the size of an apple, the diameter of the maximum cross-section is generally used. The term "cross- section" refers to the section of the apple that is perpendicular to the line connecting the stem and the calyx. For the contours of the apples extracted in this study, the maximum diameter is defined as the difference between the maximum and minimum horizontal coordinate values of the points on the contour, as shown in Equation (3). max miniB x x  (3)  max 1 2 1max , , , ,n nx x x x x    (4)  min 1 2 1min , , , ,n nx x x x x    (5) Where, iB represents the maximum diameter extracted from the apple image captured by the i-th industrial camera, and x denotes the horizontal coordinate of a point on the contour. Due to the irregular structure of the apple, the maximum diameter measured from different angles may exhibit some discrepancies. To ensure that the extracted maximum diameter is relatively accurate, this study employs three industrial cameras positioned at 120° angles to capture the apple images. Ultimately, the maximum diameter is calculated as the average of the maximum diameters obtained from the three cameras, as expressed in Equation (6). Similarly, the maximum vertical diameter HHH can be determined. 1 2 3 3 B B B B    (6) The extracted maximum diameter in the previous section is measured in pixels, whereas the typical unit for this measurement is millimeters, requiring a unit conversion. First, the maximum diameter of each apple is manually measured three times, and the average value is taken. This value is then compared with the maximum diameter extracted by the computer, as shown in Figure 8. The scatter plot reveals an approximately linear relationship between the two. Through linear fitting, the relationship between the actual maximum diameter and the computer-extracted size can be expressed by Equation (7). 0.0986 0.84952acB B  (7) Figure 8. Pixel Values of Maximum Horizontal Diameter of Apples vs. Actual Values 4.2. Shape Index The shape index refers to the ratio of the apple's maximum vertical diameter to its maximum horizontal diameter, expressed as H δ B  . The shape index has become an important criterion for assessing apple quality. A shape index between 0.9 and 1.0 indicates a conical shape, while a shape index between 0.8 and 0.9 suggests a round shape. According to Chinese national standards, apples eligible for grading must have a shape index greater than 0.7, indicating that when viewed from the front, an apple's height being slightly smaller or larger than its width contributes to a more aesthetically pleasing shape. 4.3. Exterior Color Extraction (a) Light green (b) Light yellow (c)Dark yellow Figure 9. Training Samples The identification of an apple's exterior color can be classified into two main types. The first type applies to apples with uneven surface coloring after harvest, such as 36 Fuji apples, where the red coloration rate of the apple surface is used to represent color. The second type applies to apples with a uniform surface color after harvest, such as Golden Crown apples, where the color of the apple surface is directly recognized. In this study, the focus is on the Golden Crown apple, whose exterior color is mainly categorized into light green, light yellow, and deep yellow. These three exterior colors indicate different levels of ripeness. When the apple appears light green, its taste is sweet and sour. When the exterior is light yellow, the taste is sweet. A deep yellow exterior suggests a fragrant and sweet flavor. Each of these taste profiles has a certain appeal among consumers. A classifier is used to extract the exterior color of Golden Crown apples. The classification process consists of two stages: training and classification. The training stage involves calculating the feature vector values for each sample and classifying them according to the chosen classification method. Figure 9 illustrates some of the samples used in the training process. In this study, the exterior color of Golden Crown apples is categorized into three classes, representing color ranges rather than specific colors. The Nearest Neighbor Classification Method is employed, which is suitable when the samples belong to different categories and the samples in each category are not concentrated in a single cluster in the feature space. The algorithm is implemented through the following steps: (1) Calculate the feature vector values of the sample; (2) Compute the distance between the feature vector values and all samples in each known category. The smallest distance is considered the distance from the feature vector value to that category; (3) Compare the distances from the feature vector value to all categories, and assign the sample to the category with the shortest distance. 5. BP Neural Network Grading In this study, the BP (Backpropagation) neural network is utilized to grade Golden Crown apples. The BP neural network adjusts its thresholds and weights through backpropagation of errors, gradually making the output approach the desired result. Figure 10. Schematic Diagram of the BP Neural Network Model Structure The design of parameters in a BP neural network classifier is crucial as it directly impacts the classification error. Below are the specific design parameters for this system: (1) Determining the Number of Hidden Layers: Increasing the number of hidden layers will increase training time and reduce training efficiency. Therefore, this study uses a single hidden layer. (2) Determining the Number of Neurons in the Input and Output Layers: In this study, the extracted features for Golden Crown apples are the maximum diameter, shape index, and appearance color. Hence, the number of neurons in the input layer is set to 3. Based on the Chinese apple grading standard, Golden Crown apples are classified into four categories: premium, first grade, second grade, and others. Therefore, the number of neurons in the output layer is set to 4. (3) Determining the Number of Neurons in the Hidden Layer: The hidden layer neurons find patterns within the input sample data. If there are too many or too few neurons, it hinders pattern recognition and reduces grading accuracy. Therefore, selecting the appropriate number of hidden layer neurons is essential. Several theories exist for determining the number of hidden layer neurons. After testing and comparison, this study applies the method based on Kolmogorov’s theorem, as shown in Equation (8). 2 1n m  (8) Where, n represents the number of neurons in the hidden layer, while m denotes the number of neurons in the input layer. (4) Determining the training function is crucial. Commonly used functions for the hidden layer include sigmoid and tansig, while the output layer typically uses tansig and purelin functions. In this study, different combinations of functions were selected for the hidden and output layers and tested experimentally, with the mean squared error (MSE) calculated for each combination. As shown in Table 1, the MSE was minimized when the tansig function was used for the hidden layer and the purelin function for the output layer. Therefore, this study adopts the tansig function for the hidden layer and the purelin function for the output layer. Table 1. Three Scheme comparing Hidden layer output layer error of mean square sigmoid tansig 0.2013 sigmoid tansig 0.1472 tansig purelin 0.0932 tansig purelin 0.1566 (5) The determination of the learning rate is critical, as it influences the magnitude of weight updates and the convergence speed of the model. A learning rate that is too high may lead to rapid convergence but risks overshooting local minima, while a learning rate that is too low results in slow convergence. Typically, the learning rate is set within the range of 0.01 to 0.8. Figure 11 illustrates the training error obtained at intervals of 0.05 for different learning rates. As shown in the figure, the training error is minimized when the learning rate is set to 0.2. Therefore, in this study, the learning rate is established at 0.2. 37 Figure 11. Network training error at different learning rates. 6. Results and Analysis To validate the universality of the grading system, this study involved 300 samples of Golden Crown apples collected from three provinces in China: Shandong, Hebei, and Liaoning. The grading standards for Golden Crown apples were established based on the national standards outlined in China's "Fresh Apples" and "Special Standards for Export Fresh Apples," as shown in Table 2. Initially, all apple samples were manually graded to determine their respective grades. Each sample was first assessed based on a single characteristic parameter. Ultimately, the lowest grade among the three characteristic parameters was assigned as the grade for that sample. Figure 12 illustrates the grade distribution of the 300 apple samples. Table 2. Grading Standards for Golden Crown Apples Characteristic Parameter Extra Class Class I Class II Maximum Diameter >80mm >70mm >65mm Shape Index >1.0 >0.85 >0.7 Appearance Color Dark yellow Light yellow light green Figure 12. Grade Distribution of All Samples To validate the effectiveness of the proposed method, the samples were divided into training and testing groups. A total of 160 samples in the training group were used for method development and parameter selection, while 140 samples in the testing group were used for method verification and result statistics. Experimental results indicate that the average error of the maximum diameter extracted by the system compared to manually measured values is 0.32 mm, the average error of the shape index is 0.14, and the accuracy rate of appearance color extraction is 96.4%. Table 3 presents the statistical results of the system's grading. Table 3. Statistical Results of the System's Grading Grade Number of Manual Grading Number of System Grading Accuracy Rate Special 35 33 94.28% First 49 45 91.84% Second 40 36 90.00% Others 16 15 93.75% Total 140 129 92.14% The overall grading accuracy of the proposed grading system for Jin Guan apples is 92.14%. While the system successfully achieves accurate grading for the majority of Jin Guan apples, there remains a certain probability of misclassification. After a careful analysis of the entire grading process, the potential causes of misclassification are as follows: (1) Reflection on the Inner Wall of the Dark Box: There is a certain degree of reflection on the inner wall of the dark box. During the process of using Gaussian filtering to eliminate this effect, the edges of the apple were also blurred, resulting in less accurate contour extraction. (2) Parameter Setting in BP Neural Network: In the process of setting parameters for the BP neural network, the focus on grading efficiency inevitably led to a relatively larger grading error. (3) Insufficient Training Samples: The number of samples used in the training process was not adequate. Future work will involve using a larger dataset for training. To assess the grading efficiency of the system, three skilled graders were selected to independently grade the 140 apples in the test group. Statistical analysis indicated that the system's grading efficiency is approximately 1.7 times that of the average efficiency of the skilled graders. 7. Conclusion (1) This study, using Jin Guan apples as an example, employed machine vision techniques to extract three features: maximum horizontal diameter, fruit shape index, and external color, and subsequently applied a BP neural network model for apple grading. (2) During the image preprocessing stage, additional operators at 45º and 135º were incorporated alongside the existing horizontal and vertical operators, resulting in clearer and more complete contour extraction of the apples. (3) Testing on 140 apple samples revealed a grading accuracy of 92.14%, with the grading efficiency being approximately 1.7 times that of skilled graders. This indicates the feasibility of the proposed apple grading method and provides a technical reference for apple classification. References [1] Yang Q. An approach to apple surface feature detection by machine vision [J]. Computers & Electronics in Agriculture, 1994, 11(2-3): 249-264. 38 [2] Nakano K. Application of neural networks to the color grading of apples [J]. Computers and Electronics in Agriculture, 1997, 18(2-3): 105-116. [3] Chen H, Fei J Y. Online apple grading based on decision fusion of image features [J]. Transactions of the Chinese Society of Agricultural Engineering, 2017, 33(01): 285-291. [4] Li Long, Peng Yankun, Li Yongyu. Design and experiment on grading system for online non-destructive detection of internal and external quality of apple [J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2018, 34(9): 267-275. [5] Chen Yanjun, Zhang Junxiong, Li Wei, et al. Grading method of apple by maximum cross-sectional diameter based on computer vision[J]. Transactions of the CSAE, 2012, 28(2): 284-288. [6] Leemans V, Magein H, Destain M F. Defects segmentation on ‘Golden Delicious’ apples by using colour machine vision [J]. Computers and Electronics in Agriculture, 1998, 20(2): 117- 130. [7] Sofu M M, Er O, Kayacan M C, et al. Design of an automatic apple sorting system using machine vision [J]. Computers and Electronics in Agriculture, 2016, 127: 395-405. [8] Bao Xiaoan, Zhang ruilin, Zhong lehai. Apple grade identification method based on artificial neural network and image processing [J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2018, 34(9): 267-275. [9] Kavdr R, Guyer D E. Comparison of Artificial Neural Networks and Statistical Classifiers in Apple Sorting using Textural Features [J]. Biosystems Engineering, 2004, 89(3): 331-344. [10] Mizushima A, Lu R. An image segmentation method for apple sorting and grading using support vector machine and Otsu’s method [J]. Computers & Electronics in Agriculture, 2013, 94: 29-37. [11] Jana S, Basak S, Parekh R. Automatic fruit recognition from natural images using color and texture features [C]. Devices for Integrated Circuit. IEEE, 2017: 620-624. [12] Yu Y, Velastin S A, Yin F. Automatic grading of apples based on multi-features and weighted K-means clustering algorithm [J]. Information Processing in Agriculture, 2019, 1-10. [13] Huang Chen, Fei Jiyou. Online apple grading based on decision fusion of image features[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2017, 33(1): 285-291.