• 189 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) ISSN (Print) 2313-4410, ISSN (Online) 2313-4402 © Global Society of Scientific Research and Researchers http://asrjetsjournal.org/ Hatching Egg Image Segmentation Based on Dense Blocks and the Hierarchical Sampling of Pixels Jing Zhang a* , Caijuan Gao b a,b School of Electronics and Information Engineering, Tiangong University, Tianjin 300387, China a Email: 1262105581@qq.com b Email: 1935303288@qq.com Abstract Fertility detection of hatching eggs is crucial in the manufacturing of vaccines. For hatching egg images, the segmentation results of blood vessels, cracks and air chambers are important for detecting the fertility of hatching eggs. In this paper, we propose an image segmentation method based on dense blocks and the hierarchical sampling of pixels. Dense blocks are used instead of the traditional layer-by-layer structure to improve efficiency and model robustness. The hierarchical sampling of pixels uses small batch sampling to add diversity during batch updates, which can accelerate learning. The sampled features are sparsely arranged and classified using an Multi-layer Perceptron(MLP), which can introduce complex nonlinear predictors and improve accuracy. The experimental results show that the MIoU reaches 90.5%. The proposed method can significantly improve the segmentation performance. Keywords: Deep learning; Semantic segmentation; Dense Block; Hierarchical sampling strategy; Hatching eggs. 1. Introduction Avian influenza is an infectious disease of birds that is caused by the influenza A virus. In general, the avian influenza virus only infects poultry and other birds. However, in recent years, a new type of avian influenza virus has emerged, which has not only caused serious damage to the poultry breeding industry, but also caused human casualties, and higher requirements for the prevention and control of avian influenza have been proposed. The prevention of avian influenza is mainly through vaccination [1]. At present, the preparation of the avian influenza vaccine is mainly carried out via viral cultivation in egg embryos. In the process of hatching egg cultivation, unseparated dead embryos will cause the cultivation to fail. ------------------------------------------------------------------------ * Corresponding author. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 190 Therefore, fertility detection and classification of hatching eggs is key in the manufacturing of vaccines. The artificial eggs irradiation method, which is inefficient, still used to detect the fertility detection of hatching eggs. Thus, using machine vision to judge this activity through images has gradually become a hot research topic. In the process of egg cultivation for vaccine development, the key periods for the detection and classification of egg embryo activity are: at day 5,9,14, and 16. The embryos at each period have their own characteristics. Before the 9th day, the vascular development of the embryos is not too strong, and the edges of blood vessels and other tissues (liquid in the egg) in the image of hatching eggs are blurred, which makes it difficult to accurately segment images of hatching eggs. Therefore, it is very important to improve the segmentation results of the hatching egg image before day 9. The activity detection of egg embryos is mainly studied by obtaining the physiological information of embryos, which is roughly divided into bioelectricity detection methods, acoustic vibration analysis methods, heart rate monitoring methods, ultrasonic imaging methods, hyperspectral imaging technology detection methods, multi-information fusion technology detection methods, machine vision methods, etc. The bioelectrical signal of an egg embryo is relatively weak, and it is highly susceptible to environmental noise when using the bioelectrical detection method [2,3]; the acoustic vibration method requires strict detection conditions under constant temperature conditions. At the same time, this method only has advantages in crack detection [4,5]; the heart rate detection method is destructive to the egg embryos, and the egg embryos that are destroyed can easily breed bacteria [6-8]; ultrasonic imaging can also cause damage to egg embryos. Similarly, the method is also inefficient [9,10]; the hyperspectral imaging technology detection method requires expensive imaging equipment and is not suitable for detecting egg embryo activity after the injection of a virus [11-14]; the multi-information fusion technology detection method needs to collect multiple kinds of information at the same time, resulting in low efficiency, and the collection of one or more kinds of information may cause damage to the egg embryos [15]; the machine vision method does not damage the egg embryos, but feature extraction for images of hatching eggs is more difficult [16-19]. Deep learning avoids the complicated image processing process and greatly improves the feature extraction ability. It can avoid the influence of high-brightness speckle noise and other noise from image acquisition, and can accurately extract the characteristics of the egg embryos. Hence, this paper proposes an image segmentation method based on dense blocks and the hierarchical sampling of pixels. the method refers to the dense blocks. The dense connection structure allows each layer to obtain gradients directly from the loss function and the original input signal. Combining the feature maps learned by different layers will increase the changes in input in subsequent layers and improve efficiency. The shallow and deep features can be freely combined, which makes the model results more robust. The method refers to the hierarchical sampling of pixels. The hierarchical sampling of pixels makes full use of the information from adjacent pixels, so that the segmentation results are better along the edges. The method proposed in this paper enhances the segmentation results of hatching egg images. After the fertility detection of egg embryos to be performed, it is much better than the traditional image algorithm. 2. Methods A dense block is a block in which all layers remain densely connected, and there is no dense connection between different blocks that are connected through the transition layer. The dense block in CNN makes the network structure narrow, and the number of parameters decreases. The number of output feature maps of each convolution layer in the dense connection block is very small. The dense connection makes the transmission of American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 191 features and gradients more efficient, and the network is easier to train. In addition, experiments have found that dense connections have a regularization effect that helps to alleviate the overfitting problem in tasks on small- scale datasets. The hierarchical sampling of pixels acquires multiscale features in the network. The hierarchical sampling of pixels uses a more efficient strategy to achieve sparse image representation: the use of efficient small batch generation during training. Assuming an input image X and a sparse set of sampled pixel locations ΩP  , whereΩ is the set of all pixel locations, the process of hierarchical sampling of pixels is as follows: (1) Perform forward propagation to calculate the dense convolution response output   : , ΩiC p i p  for all layers, where  iC p represents the eigenvector of the ith layer convolution output centered on pixel p; (2) for each sample pixel ΩP  , calculate the same feature ph extracted from multiple layers of a pixel. Where        1 2, , ,p Mh X C p C p C p    ; (3) reparse  :ph p P into a matrix that can be used for downstream processing and send the MLP to implement pixel classification. In the hierarchical sampling strategy of pixels, only the multilayer features of P are calculated instead of the multilayer features of the entire set of size Ω , which reduces the amount of computation in sparse image sparse. The selection of layers in multiscale sampling is arbitrary and the gradient implementation can be easily defined. The principle of the hierarchical sampling of pixels is shown in Figure 1. Figure 1: Principle of hierarchical sampling strategy of pixels As shown in Figure 2, the feature extraction phase of the network uses a convolutional layer and four dense blocks, and the transition layer connects the dense blocks. The size of the image required in the input layer is 224 × 224 × 3; first, use a 7 × 7 convolution kernel, a step size of 2, and a complement size of 3. Shallow features are extracted and subsampled through the convolutional layer. there are four dense blocks with a growth rate of 32. Blocks are connected to the other blocks using a transition layer. The BN layer is used to normalize the data in the transition layer; then, the number of feature maps is reduced by 1×1, and average pooling is used. This operation reduces the dimensions of the feature map. The output feature map of the first convolutional layer and the four dense blocks are used in the hierarchical sampling of pixels. For each sampled pixel, the feature vector  iC p of the ith layer convolution output centered on p is calculated according to the pixel and American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 192 its four adjacent positions, and then the features sampled from each layer are sparsely arranged into a matrix that is easy to process later in the network. Finally, the sampled multilayer features are sent to the MLP for classification, and the organizational category to which the pixel belongs is obtained. Figure 2: The network architecture 3. Data preparation In order to obtain enough image data of egg embryos of virus strains, the egg embryos were photographed by using LED light source, HIKVISION industrial camera MV-CE013-50GC and HIKVISION lens MVL- HF0828M-6MP in the sterile dark box. 3.1. Data preparation The traditional image segmentation algorithm is used to segment a simple, processed egg image so that the egg embryo region is separated from the background (Figure 3). Finally, the air chambers, cracks, blood vessels and other tissues in the embryonic region of the egg are manually labeled. Manual labeling requires strict labeling standards. The data labeling work of the egg embryos is performed by experienced egg-picking workers. There are 1,500 artificially labeled hatching eggs in the dataset. The results of the labeling are shown in Figure 4. Red indicates blood vessels, light blue indicates gas chambers, blue indicates cracks, and black indicates the background. After the manual labeling is completed, the egg embryo tissue of each category in the annotation map is represented by an eigenvalue and converted into an index map. The correspondence between various colors and eigenvalues in the index map is shown in Table 1. Figure 3: Background coarse segmentation American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 193 Figure 4: Image segmentation Table 1: Relationship between color and eigenvalues in the index map Organization name Color Mark (RGB) Eigenvalue Background (0,0,0) 0 Blood vessel (255,0,0) 1 Crack (0,0,255) 2 Air chamber (0,162,232) 3 other (0,255,0) 4 3.2. Data enhancement On the basis of the original data, 150 images were randomly selected as the verification set, and the verification set does not perform data enhancement. The remaining 1,350 images were rotated by 45 degrees and 135 degrees, and the expanded dataset had 4,050 images. The number of crack samples is relatively small, and 200 crack samples from the expanded dataset were selected for image inversion. After balancing the data, there were 4,450 images in the dataset, of which 700 images were randomly selected for the test sets, and the remaining 3,750 images were used as training sets. Image rotation and image flipping do not change the spatial position of the pixels in the image, ensuring that the local features of the egg embryo image do not change. In the process of model training, a cost-sensitive method based on a cost-sensitive matrix is used to impose error penalties on different sample categories to achieve data enhancement. 4. Experimental 4.1. Analysis of the hyperparameter k in dense blocks The EggSegNet network uses four dense blocks, which have very narrow layers that increases the number of feature maps with a growth rate of k. Each layer can access all the previous feature maps in a block; that is, the collective knowledge of the network can be accessed. considering the feature map as the global state of the network, each layer adds its own k feature maps to this state. In this paper, experiments were carried out with different growth rates k. When the value of k is 16, 18, 24 or 32, the models were trained on the dataset and tested on the test set, and the MIoU values of different test results were calculated. As shown in Table 2, when k is 32 in the dense block, the MIoU value is the largest, so we select the hyperparameter to control the growth American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 194 rate of the feature map in the dense block. Table 2: MIoU of models generated by different sizes k k MIoU 16 88.9% 24 89.6% 32 90.5% 40 89.1% 4.2. Comparison with other networks To prove the effectiveness of dense blocks in EggSegNet in hatching egg image segmentation. This paper uses the method of reference [20] and reference [21] to conduct a comparative experiment with the network structure based on dense blocks. The generated training models under different basic network structures were tested on the test set, and the MIoU values of different test results were calculated. As shown in Table 3, the MIoU value of the test network based on dense blocks was higher than ResNet and SE-ResNet, demonstrating the effectiveness of the dense block based network in the segmentation of hatching egg images. Table 3: MIoU of different basic network models Network structure MIoU ResNet 88.5% SE-ResNet 89.1% The proposed method 89.3% 4.3. Comparison with other semantic segmentation methods To verify the effectiveness of EggSegNet in hatching egg image segmentation tasks relative to other methods, a comparative experiment was designed. In the experiment, the methods of reference [22], reference [23], reference [24] and EggSegNet were trained and tested, and the segmentation results were compared. Figure 5 shows the segmentation results of different networks. The segmentation result of the FCN is rough along the edges, and the small blood vessels cannot be segmented. The segmentation results of SegNet are significantly larger than the contour in the annotation map. The maximum pooled index is used in the decoder to perform upsampling on the low-resolution feature map so that SegNet performs better on the detailed features. The blood vessel contours in the DeepLab v3 segmentation results are closer to the annotation map than to the SegNet segmentation results. DeepLab v3 uses the ResNet-101 structure to produce a characteristic overlearning phenomenon, and the part of the egg embryo tissue that is closer to the blood vessel color is divided into blood vessels. The segmentation results of EggSegNet are closer to the annotation image, which alleviates the feature learning phenomenon in DeepLab v3. The models generated by the different training methods were tested on the test set, and the MIoU values of the different test results were calculated. As shown in Table 4, EggSegNet is superior to FCN, SegNet and DeepLab in terms of the MIoU. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 195 Figure 5: The segmentation results for the different semantic segmentation methods (the first row to the sixth row are the original image, the annotated image, the results of the FCN method, the results of the SegNet method, the results of the DeepLab method, and the results of the proposed method, respectively) Table 4: MIoU of different semantic segmentation methods Method MIoU FCN 84.7% SegNet 86.2% DeepLab 89.5% The proposed method 90.5% 4.4. Comparison with traditional image vessel segmentation methods The traditional image segmentation algorithm was compared with EggSegNet, and the traditional image American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 196 segmentation algorithm uses the methods of references [18,19]. The traditional image processing algorithm only divides the blood vessels. To show the difference between the traditional image segmentation algorithms and the proposed method, the MIoU value of the test results under different methods is calculated. As shown in Table 5, the MIoU of the proposed method reaches 90.5%, far exceeding the performances of the traditional image segmentation methods. Table 5: MIoU of different methods Segmentation method MIoU Shan, B and his colleagues [18] 83.5% Xu, Q. L and his colleagues [19] 85.7% The proposed method 90.5% 4.5. Comparison with other image vessel segmentation methods based on deep learning The image vessel segmentation algorithm based on deep learning was compared with EggSegNet, and the image vessel segment segmentation algorithm uses the methods of references [25]. Similarly, the image vessel processing algorithm only divides the blood vessels. The MIoU value of the test results is calculated. As shown in Table 6, Obviously, we can conclude that the proposed method in this paper has better performance. Table 6: MIoU of different methods on deep learning Segmentation method MIoU Geng and his colleagues [25] 86.6% The proposed method 90.5% 5. Fertility detection of hatching eggs The purpose of image segmentation of hatching eggs is to detect the fertility of hatching eggs. According to the area ratio of each tissue region in the segmentation result, the hatching eggs classification is carried out, and the fertility detection of hatching eggs is realized. On the basis of the segmentation results of the hatching egg image, the predetermined number of each tissue is counted. There should be no cracks in the hatching egg. Once there are cracks, the hatching egg will contaminate the hatching egg of the same batch. Therefore, cracks in the segmentation results will be removed; on the basis of removing cracks, hatching egg with larger air chamber are not suitable for cultivating avian influenza vaccine, so the larger air chamber eggs are removed on the basis of eliminating cracked eggs; finally, according to the blood vessel information obtained by the segmentation result, the hatching egg with a relative small blood vessel ratio has been eliminated. This is the process of Fertility detection of hatching eggs, as shown in Figure 6. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 197 Figure 6: A schematic diagram of fertility detection of hatching eggs There are two thresholds, namely α and β, in the schematic diagram of fertility detection of hatching eggs. In this paper, we experimented with multiple sets of values in the range of 0 to 1 for α and β, respectively. The experimental results show that the accuracy of fertility detection of hatching eggs is the highest when α and β are taken one third respectively. In order to illustrate the effectiveness of the hatching eggs image segmentation method based on dense blocks and the hierarchical sampling of pixels on the fertility detection of hatching eggs, the experiment compared the fertility detection of hatching eggs based on other image segmentation methods. The experimental results are shown in Table 7. The effect of this method on image segmentation of hatching eggs is better than the traditional image segmentation algorithms. The accuracy of fertility detection of hatching eggs is 95.5%, which is much better than traditional image algorithms. Table 7: The accuracy of different activity detection methods Method Segmentation method Classification method Detection accuracy Xu, Q. L and his colleagues morphology methods kNN 90.8% Shan, B and his colleagues morphology methods blood vessels area ratio 92.1 The proposed method egg embryos tissue area ratio 95.5% 6. Conclusions In this paper, we propose an image segmentation method based on dense blocks and the hierarchical sampling of pixels. This method can significantly improve the segmentation performance. The experimental objects are hatching egg images before 9 days, which is difficult to accurately segment. The experiment shows that the segmentation results for hatching egg images from this method are better than those from traditional image processing algorithms. The MIoU reached 90.5%. We also believe that the method proposed in this paper could American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 198 be used for other tasks, such as medical diagnosis and measuring tissue volumes. References [1]. Treanor J J, Campbell J D, Zangwill K M, Rowe T and Wolff M. “Safety and Immunogenicity of an Inactivated Subvirion Influenza A (H5N1) Vaccine.” New England Journal of Medicine, vol.354, pp. 1343-1351, 2006. [2]. Romanoff A L and Frank K. “High Frequency Conductivity and Dielectric Effect of Fresh Fertile and Infertile Hens'Eggs.” Experimental Biology & Medicine, vol.47, pp.527-530, 1941. [3]. Romanoff A L. “EVALUATION OF FUTURE PERISHABILITY OF INTACT FRESH EGGS BY RADIO‐ FREQUENCY CONDUCTIVITY.” Journal of Food Science, vol.14, pp. 310-313, 1949. [4]. Sinha D N, Johnston R G, Grace W K and Lemanski C. “Acoustic Resonances in Chicken Eggs.” Biotechnology Progress, vol.8, pp.240-243, 1992. [5]. Coucke P M, Room G M, Decuypere E M and De Baerdemaeker J G. “Monitoring Embryo Development in Chicken Eggs using Acoustic Resonance Analysis.” Biotechnology Progress, vol.13, pp.474-478, 1997. [6]. Cain J R, Abbott U K and Rogallo V L. “Heart rate of the developing chick embryo.” Proceedings of the Society for Experimental Biology & Medicine Society for Experimental Biology & Medicine, vol.126, pp.507-510, 1967. [7]. Suzuki Y, Musashi H and Tazawa H. “Noninvasive heart rate monitoring system for avian embryos based on the ballistocardiogram.” Medical & Biological Engineering & Computing, vol.27, pp.399- 404, 1989. [8]. Lewin R, Dörner M and Tönhardt H. “Pulse oximetry: a new way of determining the heart rate in chicken embryos.” Pflügers Archiv, vol.434, pp.639-641,1997. [9]. Mcquinn T C, Bratoeva M, Dealmeida A, Remond M, Thompson R P and Sedmera D. “High- frequency ultrasonographic imaging of avian cardiovascular development.” Developmental Dynamics, vol.236, pp.3503-3513,2007. [10]. Schellpfeffer M A and Kolesari G L. “Microbubble contrast imaging of the cardiovascular system of the chick embyro.” Ultrasound in Medicine & Biology, vol.38, pp.504-510,2012. [11]. Smith D P, Mauldin J M, Lawrence K C, Park B and Heitschmidt g w. “Detection of fertility and early development of hatching eggs with hyperspectral imaging.” XIth European Symposium on the Quality of Eggs and Egg Products, 2005, pp.176-180. [12]. Jones S T, Shattuck R E and Center A I. “Detection of Early Embryonic Development in hatehing eggs: a hyperspcctral imaging systems and neural network approach.” Johns Hopkins Apl Thchnical Digest, 2005, pp.67-73. [13]. Smith D P, Lawrence K C and Heitschmidt G W. “Fertility and embryo development of broiler hatching eggs evaluated with a hyperspectral imaging and predictive modeling system.” International journal of poultry science, vol.7, pp.1001-5101004, 2008. [14]. Liu L and Ngadi M O. “Detecting Fertility and Early Embryo Development of Chicken Eggs Using Near-Infrared Hyperspectral Imaging.” Food and Bioprocess Technology, vol.6, pp.2503-2513,2013. [15]. Zhu Z, Liu T and Ma M. “Hatching eggs nondestructive detection based on hyperspectral-imaging American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 74, No 1, pp 189-199 199 information and RVM.” Nongye Gongcheng Xuebao/Transactions of the Chinese Society of Agricultural Engineering, vol.31, pp.285-292, 2015. [16]. Das K and Evans M D. “Detecting Fertility of Hatching Eggs Using Machine Vision I. Histogram Characterization Method.” Transactions of the Asae, vol.35, pp.1335-1341, 1992. [17]. Das K and Evans M D. “Detecting Fertility of Hatching Eggs Using Machine Vision II: Neural Network Classifiers.” Transactions of the Asae, vol.35, pp.2015-2041, 1992. [18]. Xu Q L and Cui F Y. “Non-destructive detection on the fertility of injected SPF eggs in vaccine manufacture.” Proceedings of the 26th Chinese Control and Decision Conference, Changsha, PEOPLES R CHINA, 2014, pp.1574-1579. [19]. Shan B. “Fertility Detection of Middle-stage Hatching Egg in Vaccine Production Using Machine Vision.” Proceedings of the Second International Workshop on Education Technology and Computer Science, Wuhan, China, 2010, pp.95-98. [20]. He K, Zhang X, Ren S and Sun J. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp.770-778. [21]. Hu J, Shen L and Sun G. “Squeeze-and-Excitation Networks.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018. [22]. Long J, Shelhamer E and Darrell T. “Fully convolutional networks for semantic segmentation.” Proceedings of the 2015 Conference on Computer Vision and Pattern Recognition, Boston, MA, 2015, pp.3431-3440. [23]. Kendall A, Cipolla R and Badrinarayanan V. “Segnet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.39, pp.2481-2495, 2017. [24]. Chen L C, Papandreou G, Kokkinos I and Murphy K, Yuille A L. Semantic image segmentation with deep convolutional nets and fully connected crfs[J]. arXiv preprint arXiv:1412.7062, 2014 [25]. Geng L, Qiu L, Wu J and Xiao Z. “Blood vessels segmentation of hatching eggs based on fully convolutional networks.” Ninth International Conference on Graphic and Image Processing, 2018.