293 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) ISSN (Print) 2313-4410, ISSN (Online) 2313-4402 © Global Society of Scientific Research and Researchers http://asrjetsjournal.org/ Automatic Grasping Region Extraction Using Shape Profile Based and Geometrical Features Approach Saint Saint Pyonea*, Wut Yi Winb, Aung Myat Sanc a,b,cDepartment of Mechatronic Engineering, Mandalay Technological University, Mandalay, Myanmar aEmail: saintsaint85@gmail.com bEmail: wutyiwin@gmail.com cEmail: agmyatsanmc@gmail.com Abstract Many applications of robotics include the grasping and manipulation of objects. Working in assembly robotic environments, the robot has to accurately not only locate the part but also to recognize it in readiness for grasping. In order to determine a grasping position, it is necessary to recognize the types of object, and detect portions which are suitable for grasp. According to get the important data clearly and correctly from the images, the detection and extraction methods are essential. This paper is mainly focused on the method of extracting the PCA and Shaped Profile with geometrical feature. Our proposed method is the combination of shapes based approach with the ratio and hole features. The proposed system has been tested successfully to a dataset of 336 images for seven types of common hand tools and achieved good accuracy and less computation complexity for 2D images by using a single camera. The overall recognition accuracy of PCA method with geometrical feature approach is 69.0476% on the same set of test images whereas overall accuracy of shape profile based method with geometrical feature approach is 97.9167%. Base on the experiment, this system is robust for the industrial robots for grasping tasks. This paper intends to implement machine vision system for industrial robotic grasping tasks. Keywords: accuracy; grasping region extraction; PCA features; Shaped profile feature; machine vision. 1. Introduction Robotics execute a wide variation of tasks in an industrial setting. Grasping and manipulation of every kind of object is arguably the most distinctive practical skill of human beings, and erect posture has likely evolved in order to free the upper limbs and make of the hands two unmatchable tools. ------------------------------------------------------------------------ * Corresponding author. http://asrjetsjournal.org/ American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 294 Despite the great efforts that have been and are being put on it, grasping in robotics is largely an unsolved problem due to its inherent complexity and the limited adaptive skills of present day robots in visual. In order to successfully grasp an object, some of its features must be known. Essential features are its location and dimensions. Other information could also be useful. Vision guided robotics has a rich research history that dates back to the late seventies and early eighties. While many elements of vision guided robotics have been thoroughly researched, few vision guided robotic systems have found their way into industry. Most vision guided robotic systems were too slow and too sensitive to the environment to be useful in an industrial setting. With the rapid rise in computing power and the drop in price of high quality robotic and vision systems, the application of vision guided robotic systems to an industrial setting is becoming a reality. However, there are still barriers and limitations to the production of generic, robust, and practical vision guided robotic solutions. Robots are often built for specific tasks. In this case, the manipulator is set to distinguish the objects and place them to different places automatically. This fundamental vision guided robot has applications in many domains ranging from industry use to daily life. To achieve this goal, the robot must be equipped with sensors in order to perceive the 2D environment, and allows it to operate smoothly within that environment. Visual serving is a good way to provide sufficient information for the manipulator. A fair amount of work has been done in applications of autonomous robotics. In this study, we use feature based matching vision algorithms to determine the objects and estimate their current distance and grasping region. The principal contribution of this paper is the definition of a grasping region of object involved in vision-based grasping tasks. These region constitutes a bridge between cognitive science and robotics research and includes all the steps required for performing a successful grasping tasks from visual data. In this paper, combination of shaped profile feature, hole feature, height and width of object pixel ratio feature approach is researched for grasping region extraction system. There are two main stages. The first stage is measuring the distance from camera to object which can be computed by interpolation method. The second stage is grasping region extraction in which objects are recognized using Backpropagation Neural Network. The rest of this paper is organized as follows. Section 2 reviews the related work. Section 3 describes the key frame generation and rectangle shape detection. It also discusses the text area detection and extraction and then the enhancement of number plate region. The experimental results are shown in Section 4. Finally, concluding remarks are presented in Section 5. 2. Related Works Recent work on grasp detection requires full knowledge of 2D or 3D models of objects. Based on this information, methods such as one based on the mechanics of grasping and the finger-object contact interactions [2, 3] focus on designing control and planning algorithms to achieve successful and stable grasps. Significant past work uses 3D simulations to find good grasps [4,5,6,7]. These approaches are powerful but rely on a full 3D model and other physical information about an object to find an appropriate grasp. Full object models are often not known a priori. General purpose robots may need to grasp objects without first building complex 3D American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 295 models of the object. In real-world grasping, the full 3D shape of the object is hard to perceive. Some early work considers only objects of simple shapes to address this problem. For example, Miller and his colleagues [8] used heuristic rules to generate and evaluate grasps for three-fingered hands by assuming that the objects are made of basic shapes such as spheres, boxes, cones and cylinders, each with pre-computed grasp primitives. Other methods focus on grasping 2D planar objects using edges and contours to determine form and force closure. Reference [9] also considered grasping planar objects by classifying them into a few basic shapes, and then used pre-scripted rules based on fuzzy logic to predict the grasp. Saxena Miller and his colleagues [10] showed that a ‘grasping point’ could be estimated from the image using supervised learning algorithms, and that this method generalized to a large number of novel objects. This paper is addressed the same problem as Saxena Miller and his colleagues but use a different real 2D images and processing that is capable of higher accuracy at much faster speeds. Recently, it has been proposed a PCA based method to extract feature from images for grasping region extraction [14]. The main contribution of this paper is the use of a simple shape profile feature approach by combining hole and height and width of object pixel ratio. 3. Proposed Method In the proposed method, two methods, the Principal Component Analysis (PCA) and shaped profile method are used for extracting features and Backpropagation (BP) Neural Network is used to classify the object types. The main concept of the proposed method is to identify a location at which to grasp the object by using its image. The block diagram of this proposed object recognition system is illustrated in Figure 1. Preprocessing Feature Extraction Preprocessing Feature Extraction Classification Train Image Test Image Matching Finding Distance from Camera to Object Finding Grasping Point Training Testing Database Figure1: Block diagram of the proposed system The process starts from imported images into the system by camera, detected image is extracted feature and trained the object by backpropagation neutral network. Objects are recognized by comparison with the data that American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 296 are trained in neural network. The proposed method is to recognize the objects and provide the gripper to grasp them efficiently in various appearance changes for grasping tasks of industrial robots. This system can provide image information that the distance from camera to object to move the gripper. 3.1. Image Acquisition Digital camera is used as image acquisition devices. The images of handling tools are stored as jpg images in data set, such as screw, Hex-key, wrench, etc. 3.2. Data Preprocessing In data preprocessing stage, image data are pre-processed to make it noise free or clearer for feature extraction process. Image processing is the technique that would enhance image quality for preparing images for measurement of the present features. In the preprocessing step, it is necessary to perform these stages such as resizing, changing color, opening and filling holes. Overall, the operations devised here can help to eliminate noise and irrelevant artifacts from images so as to obtain more accurate recognition of shapes; they can also help to identify defects on objects by locating specific features of interest. After the preprocessing step, PCA features and shaped features extraction steps are expressed in next sub section. In both section, the input image is firstly converted into binary image and filtered this image to remove noise, long and narrow region by using area opening. And then, hole feature is extracted and orientation of object are calculated to rotate the object into standard position (90˚ or 180˚). After that, height and width of pixel of object are also calculated. If pixels of height are less than width, the images are rotated to vertical position (90˚). These images are modified and enhanced with morphology to connect and fill boundaries and region. After the preprocessing steps, this image is cropped to get the required region. After cropping the object, the ratio of height and width are calculated to extract the ratio feature. For both training and testing process in order to be uniform sized inputs, all of images are resized into 500×200 pixels for PCA feature extraction method and 110×30 pixels for shaped feature extraction method. By using uniform size image, the feature vector for each image is correctly extracted in feature extraction step. After finishing these steps, the feature extraction of PCA and shaped are presented in the next sub section. 3.3. Feature extraction The purpose of feature extraction is to extract the significant features of images [11]. Firstly, the number of features that want to take for each image are defined because the number of features that have to train must equal for all images. In the features extraction step, 202 features, 200 PCA features, 2 features of the hole and pixel ratio are extracted for PCA based method and 202 features, 200 shaped features, 2 features of the hole and pixel ratio of object are extracted for shaped based method. This features are input to the Neural Network for recognition purpose. The extracted features are saved in data1 file and it is applied as the input of the Neural Networks. 3.3.1 Principal Component Analysis (PCA) Feature Extraction American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 297 According to the assumption of the research method, the resized image must be 500×200 pixels for PCA feature extraction method. Thus, after we have achieved the resizing image, we have to extract features by using Principal Component Analysis. The preprocessing steps are the same as shaped based method. PCA computes the basis of a space which is represented by its training vectors [12]. PCA can be computed by the following steps: Figure 2: Extracted feature points with PCA • Get 2D image data. • Subtract the mean for individual dimensions • Calculate the covariance matrix: Covariance is a measure of how much the two dimensions vary from the mean with respect to each other. • Calculate the eigenvalues and eigenvectors • Selection of components and transformation (getting new data) The combination features of PCA with hole and ratio of pixel are input to the Neural Network for recognition purpose. The extracted features are saved in data1 file and which is applied to the input of the Neural Networks. Figure 2 shows extracted feature points with PCA. 3.3.2 Shaped Feature Extraction After we have achieved the resizing image 110×30 pixels from preprocessing step discussed in previous section, we have to extract features. In the features extraction step, shaped feature extraction method is used for extracting 200 features. In this method, firstly we find the locations of pixel value “1” between row is greater than 10 and less than 110. From this pixel location, left data are put to the maximum value of column and right data are put to the minimum value of column. Therefore, the total data feature points are 200, 100 from left data and 100 from right. And then, we combined these 200 shaped features with hole and ratio of pixel. This features are input to the Neural Network for recognition purpose. The extracted features are saved in data1 file and which is applied to the input of the Neural Networks. Extracted feature points with shaped are shown in Figure 3. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 298 Figure 3: Extracted feature points with shape 3.4. Architecture of the Neural Network Model In our experiments, we obtained about 84 training data sets. In this system, BP is designed with three layers; input layer, hidden layer and output layer. The input layer accepts the prepared images of the objects from the image processing steps. The input layer has 200 nodes to accept the pixel values from image processing steps. The number of units of output layer is 7 because the number of object types is 7. The learning result changes depending on the number of units of hiding layers, but it is difficult to decide the best number of units. In our experiments, numbers of units of the second, hidden layers are 100. The Sigmoid function is used for the response function of units. The input and output data set of the network is denoted as {(p1, t1), (p2, t2)…. (pQ,tQ)}.The objective function of the network is defined as q T q Q q S i M qq Q q eeioitV m ∑∑∑ === =         −= 11 2 1 2 1))()(( 2 1 (1) where M qo is the output of the ith node corresponding to pq, tq(i) is the target output of the ith node, and Sm is the number of the output layer node. The network reaches convergence after taking about 879 iteration steps. 3.5. Object Types Classification This portion of the system is used to identify the name of the device. The process starts from imported images into the system by camera, detected image is extracted by PCA and trained the object by backpropagation neutral network. The saved neural network is loaded first, then the input feature vector is extracted from the user input image file. After the inputs features of collected device images are extracted, it is put into the neural network. Objects are classified by comparison with the data that are trained in neural network. There are 84 images to use in train data set, all of them can be classified after the Neural Network has trained. The result is displayed with output box. The images in the Dataset is classified into seven distinct categories, with categories like “brush”, American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 299 “hex key”, “open-end wrench”, “combination wrench”, “screw driver”, and so on. Each device image input to recognition system and classified correctly as shown in Figure 4. This system can provide image information that the distance from camera to object to move the gripper and grasping region to grasp efficiently. So, this process is presented in the next sub section. Figure 4: Testing of Wrench, Screw driver and Hex-key 3.6. Distance from Camera to Object Estimation and Grasping Region Extraction This paper is estimated the distance to the object by using a single camera based on an interpolation that is robust to changes in appearance of objects that have different shapes by using predefined rules. In order to find the parameters of the interpolation function, a set of pixels’ ratios with predefined distance from camera is used, and then the distance of object from the camera is calculated. Proposed distance from camera to object estimation algorithm can be described as Table 2. There are introduced: • BW, S, R1, binary image, cropped image, and pixel ratio of objects, respectively; • R1(1), R1(2), R1(3), pixel ratio of objects from 1ft, 2ft, 3ft respectively; • Height, height of object in image; • Width, weight of object in image; • D, distance from camera to object For most objects, there is typically a small region that a human (using a two-fingered pinch grasp) would choose to grasp it; with some abuse of terminology, it will be informally referring to this region as the “grasping point”. There can be more than one good grasp, and among these, some grasps may be more desirable. For example, a screw driver may be grasped both by the handle or the shaft, however the handle is preferred due to its size, material and such. In this proposed system, examples of grasping points include the mid-point region for a brush, the centroid region for a screw driver, etc. Grasping regions are given in red rectangle as shown in Figure 5. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 300 Table 2: Distance from Camera to Object Table 3: Grasping Region Calculation American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 301 After finishing the above four main processes, we can receive suitable region of object to grasp. The accuracy of the grasping region extraction can be seen in Section 4. Figure 5: Result of grasping region 4. Experimental Results & Discussions In this section, we are going to represent the accuracy measurement of the research method with many different images. The objective of research method is to measure the distance and extract the grasping region in more precisely. To measure and proof of the accuracy of the proposed method, experimental settings and some experimental results can be shown as follow. 4.1. Implementation of the proposed system The proposed method is tested with many images, with different distance and different types of hand tools. Among our experiments, the results of three different distance which have different orientations are discussed. In the proposed method, Matlab 2016Ra is used to simulate the proposed processing procedures. Under the implementation process, we have first to acquire image to preprocess in second stage. After the second stage has finished, the feature extraction process is applied. Finally, we have to classify of the object. After classifying the object, interpolation method is used for measuring the distance from camera to object and the grasping region is extracted by using predefined rules as discussed in the above section that is robust to changes in appearance of objects that have different shapes. The implementation process is demonstrated with GUIs as described in the following Figure 6. 4.2. Experimental results In this section, the experimental results are discussed based on feature extraction results. The accuracy and processing time of the proposed method (PCA Based Method with hole and ratio feature approach) is evaluated and compared with previous method (Shaped Profile Based Method with hole and ratio feature approach). The experimental results of the proposed method are carried out many images with different types of hand tool by using Intel (R) Core(TM) i7-4510U CPU@ 2.00GHz. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 302 The purpose of the proposed system is how much precisely measured the distance from camera to object and extracted the grasping region from images. The accuracy can be described as the following formula. T TNTPAccuracy + = (2) where, TP = number of true positive images TN = number of true negative images T = total number of test images Figure 6: Illustration of system implementation and its processing simulations According to the recognition accuracy result in Table 1, shape profile feature based method has higher accuracy rate than the PCA method. In the proposed method, most of hand tools can be recognized well with highest accuracy, but two of hand tools ‘Combination Wrench’ and ‘Open-end Wrench’ have lowest recognition accuracy of 95.8333%. The features of these two hand tool are mixed up with the features of ‘Box-end Wrench’. In PCA method, ‘Hex-key’ has the lowest recognition accuracy rate of 31.2500% when it is very similar with the feature of ‘Open-box Wrench’. To calculate the accuracy, we need the value of numbers of true positive and negative images and total numbers of all images. For example, in combination wrench, the value of correct American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 303 image, TP and TN are 19 and 6 and the value of all images is 48. So, the accuracy value of combination wrench with PCA feature implemented by the proposed system is 52.0833%. The overall recognition accuracy of PCA method with geometrical feature approach is 69.0476% on the same set of test images whereas overall accuracy of shape profile based method with geometrical feature approach is 97.9167%. Table1: Accuracy of PCA method with geometrical feature Approach and proposed method. No Types of Hand Tool No of Total Test Images PCA Method with Geometrical Feature Approach PCA Method with Geometrical Feature Approach No of Correct Images Percentage of Correct Images No of Correct Images Percentage of Correct Images 1 Combination Wrench 48 25 52.0833% 46 95.8333% 2 Brush 48 27 56.2500% 48 100% 3 Hex-key 48 15 31.2500% 48 100% 4 Screw Driver 48 47 97.9167% 47 97.9167% 5 Open-end Wrench 48 30 62.5000% 46 95.8333% 6 Box-end Wrench 48 41 85.4167% 47 97.9167% 7 Bracket 48 47 97.9167% 47 97.9167% Total Average Recognition Rate 69.0476% 97.9167% 5. Conclusion This paper presented vision based extraction of grasping region of objects that predicts the grasping region of an object directly as a function of its image. The numerical estimation of proposed method is less computation complexity for 2D images by using a single camera but it can give satisfied outputs according to the results of experiment. To extract the feature of the image, two methods are used. According to the simulation results, the shape profile feature is more reliable to extract the feature than PCA feature. Therefore, the complete grasping region extraction system process uses the shape profile feature to recognize and extract grasping region. The proposed algorithm automatically extracts grasping points that are trained images of a different number of objects. The proposed system has been tested successfully to a dataset of 336 images for seven types. The experiments are carried out by using MATLAB programming language. The system can give the average accuracy rate of 69.0476% with PCA whereas it was 97.9167% with proposed method. As the proposed method achieved the most precise accuracy, the proposed method outperformed in feature extraction for this system. This system can also provide the distance from camera to object that to move the robotic gripper. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 304 6. Recommendation This study is only a simulation model of machine vision system for grasping region extraction system. The limitation of the proposed system is able to detect for known objects between 1ft and 3ft and is dependent on a class of objects. The processing unit is used with a LENOVO laptop (PC). In real, grasping robot has its own control system. This study needs to change the hardware interfacing as the interface of control system of robotic. For further extension, the program will be extended to execute many cases of automation as a machine vision system in industrial and then continue the grasp planning research with unstructured environment. For example, the dimension of object can be measured by using a machine vision system. Moreover, the hardware system will be constructed, and the complete system can be promoted to real robotic grasping control system. Acknowledgements A special thank is offered to Dr. Wut Yi Win, Professor and Head of Department of Mechatronic, Mandalay Technological University, for her encouragement, constructive guidance and kindly advice throughout the preparation of this paper. And then, the author is thankful to all teachers in Department of Mechatronic Engineering, Mandalay Technological University for their effective guidance, helpful suggestion and supervision for this paper and all of her friends who have directly or indirectly assisted her throughout the year. References [1] I. López-Juárez, R. Rios-Cabrera, M Peña-Cabrera Fast Object Recognition for Grasping Tasks using Industrial Robots Computación y Sistemas Vol. 16 No. 4, pp. 421 432 ISSN 1405-5546, 2012. [2] S. Chen, B. Mulgrew, and P. M. Grant, “A clustering technique for digital communications channel equalization Using radial basis function networks,” IEEE Trans. on Neural Networks, vol. 4, pp. 570- 578, July 1993. [3] J. U. Duncombe, “Infrared navigation—Part I: An assessment of feasibility,” IEEE Trans. Electron Devices, vol. ED-11, pp. 34-39, Jan. 1959. [4] A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,” in IEEE International Conference on Robotics & Automation (ICRA). Citeseer, 2000, pp. 348–353. [5] A. T. Miller, S. Knoop, H. I. Christensen, and P. K. Allen, “Automatic grasp planning using shape primitives,” in IEEE International Conference on Robotics & Automation (ICRA), vol. 2. IEEE, 2003, pp.1824–1829. [6] A. T. Miller and P. K. Allen, “Graspit! a versatile simulator for robotic grasping,” Robotics & Automation Magazine, IEEE, vol. 11, no. 4, pp.110–122, 2004. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2017) Volume 38, No 1, pp 293-305 305 [7] R. Pelossof, A. Miller, P. Allen, and T. Jebara, “An svm learning approach to robotic grasping,” in IEEE International Conference on Robotics & Automation (ICRA), vol. 4. IEEE, 2004, pp. 3512–3518. C. [8] A. T. Miller, S. Knoop, P. K. Allen, and H. I. Christensen, “Automatic grasp planning using shape primitives,” in ICRA, 2003. [9] D. Bowers and R. Lumia, “Manipulation of unmodeled objects using intelligent grasping schemes,” IEEE Trans Fuzzy Sys, vol. 11, no. 3, 2003. [10] A. Saxena, J. Driemeyer, and A. Y. Ng, “Robotic grasping of novel objects using vision,” The International Journal of Robotics Research, vol. 27, no. 2, pp. 157–173, 2008. [11] Gonzalez, Woods, E. R. C. R. E. S. L.: Digital Image Processing Using MATLAB, Pearson Prentice Hall Pearson Education, Iric, New Jersey, USA, 2004. [12] Mark Richardson, Principal Component Analysis, May 2009. [13] N. Yamagishi, Sh. Oe and K. Terada, “A Method of Distance Measurement by Using Monocular Camera”, The 36th SICE Annual Conference, 1997. [14] Saint Saint Pyone, and Wut Yi Win, " Object Recognition for Grasping Tasks using Industrial Robots", 7th International Conference on Science and Engineering, Dec 2016. [15] Saint Saint Pyone, Wut Yi Win and Aung Myat San, " Vision Based Extraction of Grasping Region of Objects ", 246th The IIER International Conference on Recent Innovations in Engineering and Technology (ICRIET), Singapore, 2nd Aug 2017, pp-16-21. Robotics execute a wide variation of tasks in an industrial setting. Grasping and manipulation of every kind of object is arguably the most distinctive practical skill of human beings, and erect posture has likely evolved in order to free the upper li... Despite the great efforts that have been and are being put on it, grasping in robotics is largely an unsolved problem due to its inherent complexity and the limited adaptive skills of present day robots in visual. In order to successfully grasp an obj...