Image analysis for the sorting of brick and masonry waste using machine learning methods ACTA IMEKO ISSN: 2221-870X June 2023, Volume 12, Number 2, 1 - 5 ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 1 Image analysis for the sorting of brick and masonry waste using machine learning methods Elske Linß1, Jurij Walz1, Carsten Könke1 1 Materialforschungs- und -prüfanstalt at the Bauhaus-University of Weimar (MFPA), Coudraystraße 9, 99423 Weimar, Germany Section: RESEARCH PAPER Keywords: Optical sorting of building material; masonry waste; image analysis; classification; machine learning Citation: Elske Linß, Jurij Walz, Carsten Könke, Image analysis for the sorting of brick and masonry waste using machine learning methods, Acta IMEKO, vol. 12, no. 2, article 15, June 2023, identifier: IMEKO-ACTA-12 (2023)-02-15 Section Editor: Eric Benoit, Université Savoie Mont Blanc, France Received July 8, 2022; In final form February 27, 2023; Published June 2023 Copyright: This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: This work is supporting by the TMWWDG of the Free State of Thuringia, Germany. Corresponding author: Elske Linß, e-mail: elske.linss@mfpa.de 1. INTRODUCTION AND STATE OF THE ART Currently, 20-23 million tons of masonry building materials (mortar and plaster, lightweight concrete, aerated concrete, sand- lime bricks, and bricks) are produced annually in Germany. The quantity of bricks produced, including roof tiles, is approx. 10- 15 million tons [1]. Current guiding strategies and ambitious environmental policy goals increasingly call on manufacturers of mineral building products to introduce material cycles [2], [3]. Pure brick aggregates are currently used in sports field construction, in vegetation applications, in road construction as a proportionate component of frost protection and gravel base courses, and in building construction as recycled aggregate for concrete production [1]. Pure brick recycled aggregates (Figure 1), which are made from low-dense waste and can be obtained from masonry waste, can be returned to brick production as recycled material after being ground again. A prerequisite for this, however, is that no mortar adhesions or other impurities may be present [4]-[6]. Based on the investigations in the projects [5], [7], a leaflet on the use of recycled bricks in the brick industry was drafted. It recommends pure hard-fired material, pure/hard material, low- fired material, and masonry waste as suitable feed material for the production of roof tiles, facing bricks, and vertically perforated bricks. Depending on the type of brick, up to 25 wt.-% can be reused [6], [7]. After grinding, masonry waste can also be used as a cement composite material in the cement industry [8]. Here it is necessary to be able to distinguish and separate low-fired and high-fired brick types, mortar, concrete, and other components from each other. Figure 1. Masonry waste. ABSTRACT This paper describes different machine learning methods for recognizing and distinguishing brick types in masonry debris. Certain types of bricks, such as roof tiles, facing bricks and vertically perforated bricks can be reused and recycled in different ways if it is possible to separate them by optical sorting. The aim of the research was to test different classification methods from machine learning for this task based on high-resolution images. For this purpose, image captures of different bricks were made with an image acquisition system, the data was pre-processed, segmented, significant features selected and different AI methods were applied. A support vector machine (SVM), multilayer perceptron (MLP), and k-nearest neighbour (k-NN) classifier were used to classify the images. As a result, a recognition rate of 98 % and higher was achieved for the classification into the three investigated brick classes. mailto:elske.linss@mfpa.de ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 2 Optical single-grain sorting methods offer a chance for much differentiation even of very similar materials [10, 11, and 12]. Optical analysis and sorting methods are not yet used on a large scale in the sorting of construction and demolition waste. The optical sorting methods are essentially based on innovative detection routines that can be integrated into corresponding software (Figure 2). The aim of the investigations is the development of a recognition routine for the differentiation of different brick types for single-variety brick and masonry rubble. Features from the high-resolution image and spectrum and evaluation algorithms from machine learning will be used for the recognition task. The main focus is in a first step on particles with a size of 8/16 mm. Later it will be extended on particle sizes 2/4 and 4/8 mm. Very important is the sorting out impurities, like mortar, gypsum, concrete and so on. The fundamentals are being created in order to develop more effective sorting processes in the recycling of residual brick masses in the future. This paper investigates the recognition of types of brick by using RGB images and innovative methods of machine learning are described. This provides the basis for the development of optical sorting procedures. In the investigations, recognition routines and algorithms are to be tested for the correct differentiation and separation of various brick types based on image features. 2. INVESTIGATIONS Figure 3 gives an overview of the test procedure. 2.1. Materials and Categories At the MFPA, initial preliminary investigations were carried out to distinguish between brick types on the basis of visual image information. In the investigations, different brick products were distinguished, which can be assigned to the following brick categories: • Category II: Roof tiles and facing bricks, • Category III: Vertically perforated bricks and • Category V: Other. At first, a sample collection of a representative nationwide cross-section of brick varieties and adhering mineral building materials were done. The brick samples in the different categories are composed of different new (unused) and recycled (used) bricks without adherents. Furthermore, other relevant building materials such as aerated concrete, concrete, mortar, sand-lime bricks, etc. (new and recycled) have also been included as impurities. The measured material parameters water absorption after 24 h and bulk density according to DIN 4226-101 for all samples included in the investigations are shown in the diagram (Figure 4). A total of 7,765 image recordings of samples were made. Category I (high-fired material) is still missing from these investigations, as too little material was available for examination. Category IV (brick waste from recycling plant) was also excluded at this point, as the samples were not homogeneous and can therefore belong to different categories. In future works, both categories will be added to the data set. The particle size is 8-16 mm. The brick samples listed in Table 1 were divided into three classes, which differ in bulk density. Figure 2. Principle of optical single-grain sorting method. Figure 3. Overview of the experimental programme. Figure 4. Water adsorption versus bulk density for all samples. Table 1. Example images for the investigated three brick categories. Category Objects per category Example images II Roof tiles and facing bricks 3,203 III Vertically perforated bricks 2,314 V Others 2,248 ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 3 2.2. Image acquisition and used software A data set of high-resolution images of the brick particles was created. The brick samples were examined under different conditions. On the one hand, three different types of lighting and, on the other hand, different combinations of features were used. This allows the influence of the illumination on the analysis results to be investigated. Figure 5 shows the used image acquisition system “QUALILEO”. It consists of an RGB matrix camera with a 12 MP sensor and a step-less adjustable lighting system (Figure 6). 2.3. Investigations and Results The provided algorithms are analysed using the presented data set with individual parameters and the average recognition rate (RR) captured. The achieved RR and standard deviation (Stdev) are used to compare all of the results. All of the researches were carried out in the HALCON programming environment. The investigation’s classifier have different setting parameters. The distance-based nearest k-neighbours value has been set to 5 in the k-NN classifier. The rbf-kernel was used with SVM and the γ parameter was set to 0.02. Also the one-versus- all classification mode was used. With this setting a multi-class problem is reduced to a binary decision - each class is compared to all the other classes. For MLP is used a softmax activation function. It performs best at classification tasks with multiple independent classification outputs. In the hidden layer, the number of hidden units is set to 15. In this study, the data set is split into two parts: 80 % and 20 %. The classifier are trained with a major part of the data set, while the classifier are tested with a smaller fraction of the data. 2.3.1. Results before selection and combination of features At the beginning, the typical feature groups were analysed. Table 2 shows a summary of these results. The first result was that lighting 2 (half of a ring light that simulates light from the side) always produced significantly worse results, which is why the following only presented the results for lighting L1 (fully ring light on) and L3 (incident light). This data set showed that colour features alone are the most important feature for this recognition task. The recognition rates achieved are always above 95 % for all data sets and classifiers, as shown in Table 2. The region features have only little impact on recognition. Additionally, the texture features prove more effective with over 80 % recognition rates for SVM and MLP. The k-NN classifier had the lowest recognition results. Aside from colour features, all other accuracy is insufficient. This is because the k-NN algorithm is more vulnerable to redundant features than the robust classifiers such as SVM and MLP. This research found as well that using all of the possible number of features does not always result in a better accuracy, or that the improvement is minor because many of the features are redundant. Besides that, with a large number of features, the classifier’s training time increase. A further improvement is to be achieved with feature selection. Only the most important features for the study’s problem are calculated by this algorithm. 2.3.2. Results after selection and combination of features The best results by feature selection are shown in Figure 7 and Figure 8. The very high recognition rates for all classifiers are remarkable. Figure 5. Used image acquisition system QUALILEO. Figure 6. Step-less adjustable lighting system on the QUALILEO. Table 2. Recognition rates (RR) for different classifiers learned by various numbers and kinds of features for different lightings (L1 = Lighting 1, L3 = Lighting 3). Features SVM (RR in %) MLP (RR in %) k-NN (RR in %) L1 L3 L1 L3 L1 L3 18 colour 96.27 97.04 98.30 98.23 95.91 95.24 32 region 36.62 37.71 44.98 46.33 35.26 37.52 195 texture 86.16 83.78 84.94 81.92 65.83 64.74 382 gray 78.38 77.48 74.13 85.14 42.92 44.08 432 all 80.05 79.99 93.89 94.02 58.42 60.01 Figure 7. Results of the reached recognition rates for the different brick types using different classifiers on L1 data set. Category II Roof tiles and facing bricks Category III Vertically perforated bricks Category V Others ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 4 The very high recognition rates for all classifiers are remarkable. Also, there is no significant difference in the detection rate in the different lighting settings. Both ring light images and incident light images have very similar results. The best result was achieved by the MLP with 98.51 % for lighting 1. With SVM lighting L3 an average recognition rate of 97.92 % was reached. With a RR of 96.40 % a slightly lower result is achieved by k-NN classifier. However, the difference is marginal. A look at the individual categories also shows that category III is recognised with an over 99 % recognition rate with both SVM and MLP. Not far from this result, k-NN is just below. Category II is always recognised equally well by SVM and MLP (close to 98 %). Here k-NN is clearly worse in the recognition (about 96 %). Category V shows a lower recognition rate between 96 % and 97 % for all classifiers, but again k-NN shows the lowest results of all. Compared to the results without feature selection and combination, no large increase in the recognition rates was observed. With SVM and k-NN, the detection rate increased by only about 1 %, with MLP it remained at about the same rate. Finally, Figure 9 shows the importance of the feature selection procedure in classical classification classifiers. Because it is not useful to use as many features as possible for classification, but only the most important ones. This can be seen as the score progresses. The RR increases with the number of features until it reaches its maximum value. After that, the RR decreases as more characteristics are added. Because a lot of the features are redundant and have a lot of noise in them. As a feature selection method Halcon presents a greedy method. In this algorithm the currently most promising feature is added to the feature vector. After that, it is evaluated whether one of the newly added features is unnecessary. Summarised, it is important to note that the high recognition rates after feature selection were achieved with a relatively small number of features (Table 3). The average RR is over 95 % for all classifier and both data sets L1 and L3. Hence the different settings on light had only low effect on the result of the optimised classifier. Additionally, the standard deviation is low, indicating a stable classification process. This simplifies the future implementation of the application and reduces the required computing effort and time. 3. CONCLUSION AND FUTURE WORK In this work, a method for recognizing different kinds of the brick using image processing and machine learning was investigated. A very good differentiation of the selected brick categories roof tiles and facing bricks or vertically perforated bricks could be demonstrated. With an overall recognition rate of about 98 %, the three categories are separated. In the future, the data set will be extended by additional categories, and further classifier methods (also Deep Learning) will be tested. The algorithms obtained are to be further used for the development of optical sorting methods. ACKNOWLEDGEMENT The results were developed within the framework of the research group "Sensor Technology for Products and Processes" at MFPA Weimar, which is funded by the Free State of Thuringia. The investigations will be continued in an AiF-IGF research project. We would like to express our sincere thanks for the funding from the Thuringian Ministry of Economics, Science and Digital Society, and the Federal Ministry of Economics and Climate Protection. The responsibility for the research content lies with the authors. REFERENCES [1] A. Müller, I. Martins, Recycling of Building Materials: Generation - Processing – Utilization, Springer Vieweg (2022). DOI: https://doi.org/10.1007/978-3-658-34609-6 [2] Bundesverband der Deutschen Ziegelindustrie e. V., Roadmap for a greenhouse gas neutral brick and roof tile industry in Germany, FutureCamp Climate GmbH (2021). Online [Accessed 26 May 2023] Figure 8. Results of the reached recognition rates for the different brick types using different classifiers on L3 data set. Category II Roof tiles and facing bricks Category III Vertically perforated bricks Category V Others Figure 9. Mean recognition rate (RR) of the MLP classifier (on Lighting 1 data set) in dependence of the feature. Table 3. Mean value of Recognition rates (RR) and standard deviation (SD) for all classifiers and different lightings (L1 = Lighting 1, L3 = Lighting 3). Classifier L1 (Ring light) L3 (Incident light) SVM (RR and SD in %) RR SD Features 97.85 0.42 23 97.92 0.42 23 MLP (RR and SD in %) RR SD Features 98.51 0.16 20 98.37 0.24 19 k-NN (RR and SD in %) RR SD Features 96.40 0.54 18 96.05 0.31 13 https://doi.org/10.1007/978-3-658-34609-6 ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 5 https://cerameunie.eu/topics/cerame-unie- sectors/sectors/roadmap-for-a-greenhouse-gas-neutral-brick- roof-tile-industry-in-germany/ [3] Mehr Recycling von Bau- und Abbruchabfällen in Europa notwendig Online [Accessed 3 January 2020] [In German] https://www.Recyclingmagazin.de/2016/10/02/352716 [4] M. Landmann, A. Müller, U. Palzer, B. Leydolph, Leistungsfähigkeit von Aufbereitungsverfahren zur Rückgewinnung sortenreiner Materialfraktionen aus Mauerwerk – Teil 1 und 2, AT Mineral Processing, Heft 03 und Heft 04, ISSN 1434-9302, 55. Jahrgang (2014). [In German] [5] AiF- IGF Vorhaben 18889 BG: Charakterisierung sortierter Ziegel-Recycling-Materialien anhand physikalischer und chemisch-mineralogischer Eigenschaften für die Generierung neuer Stoffströme, Schlussbericht (2019). [In German] [6] S. Petereit, Ressourceneffizienz - Ziegel aus alternativen Rohstoffen. Vortrag zum IZF-Seminar, Essen, Germany, 19.-20. September 2019. [In German] [7] S. Sabath, Charakterisierung sortierter Ziegel-RC-Materialien, Vortrag zum IZF-Seminar, Essen, Germany, 19.-20. September 2019. [In German] [8] Verein Deutscher Zementwerke e.V., Brechsand als Zementhauptbestandteil – Leitlinien künftiger Anwendung im Zement und Beton: Die Potenziale der Recyclingbrechsande: Von der Aufbereitung mineralischer Bauabfälle bis zur Herstellung ressourcenschonender Betone. Information Betontechnik, 11 (2019). [In German] [9] MVTec Software GmbH, Halcon. Online [Accessed 26 May 2023] https://www.mvtec.com/doc/halcon/1712/de/toc_regions_fea tures.html [10] E. Linß, A. Karrasch, M. Landmann, Sorting of mineral construction and demolition waste by near-infrared technology, HISER Int. Conference, 21-23 June 2017, Delft, The Netherlands, ISBN/EAN: 978-94-6186-826-8, S. 29-32 [11] AiF-ZIM Vorhaben FKZ ZF 4144903GR6, Analyseverfahren zur automatisierten Qualitätssicherung für rezyklierte Gesteinskör- nungen auf Basis hyperspektraler Bildinformationen im VIS und NIR. Schlussbericht, (2018). [In German] [12] E. Linß, D. Garten, A. Karrasch, K. Anding, P. Kuritcyn, Automatisierte Sortieranalyse für rezyklierte Gesteinskörnungen, Tagungsbeitrag, Fachtagung Recycling R’19, Weimar, Germany, 25- 26 September 2019. [In German] [13] F. Rosenblatt, The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, Psychological Review, 6 (1958), p. 386-408. DOI: 10.1037/H0042519 https://cerameunie.eu/topics/cerame-unie-sectors/sectors/roadmap-for-a-greenhouse-gas-neutral-brick-roof-tile-industry-in-germany/ https://cerameunie.eu/topics/cerame-unie-sectors/sectors/roadmap-for-a-greenhouse-gas-neutral-brick-roof-tile-industry-in-germany/ https://cerameunie.eu/topics/cerame-unie-sectors/sectors/roadmap-for-a-greenhouse-gas-neutral-brick-roof-tile-industry-in-germany/ https://www.recyclingmagazin.de/2016/10/02/352716 https://www.mvtec.com/doc/halcon/1712/de/toc_regions_features.html https://www.mvtec.com/doc/halcon/1712/de/toc_regions_features.html https://doi.org/10.1037/H0042519