From survey to semantic representation for cultural heritage: the 3D modelling of recurring architectural elements ACTA IMEKO ISSN: 2221-870X March 2021, Volume 10, Number 1, 98 - 108 ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 98 From survey to semantic representation for cultural heritage: the 3D modelling of recurring architectural elements Valeria Croce1, Gabriella Caroti1, Andrea Piemonte1, Marco Giorgio Bevilacqua2 1 Civil and Industrial Engineering Department (DICI)/University of Pisa, Largo Lucio Lazzarino 1, 56122 Pisa, Italy 2 Department of Energy, System, Territory and Construction Engineering (DESTeC)/Univeristy of Pisa, Largo Lucio Lazzarino 1, 56122 Pisa, Italy Section: RESEARCH PAPER Keywords: Cultural heritage; 3D survey; H-BIM; point cloud classification; Calci Charterhouse Citation: Valeria Croce, Gabriella Caroti, Andrea Piemonte, Marco Giorgio Bevilacqua, From survey to semantic representation for cultural heritage: the 3D modelling of recurring architectural elements, Acta IMEKO, vol. 10, no. 1, article 13, March 2021, identifier: IMEKO-ACTA-10 (2021)-01-13 Section Editor: Carlo Carobbi, University of Florence, Italy Received April 30, 2020; In final form September 30, 2020; Published March 2021 Copyright: This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: The work is partly funded by the Vinci2019 Project, promoted by the Université Franco-Italienne. Corresponding author: Valeria Croce, e-mail: valeria.croce@unifi.it 1. INTRODUCTION Three-dimensional models are increasingly being used for the graphic representation of cultural heritage items. The development of three-dimensional (3D) surveying and representation systems promoted the birth of new forms of documenting, disseminating and analysing cultural heritage objects in digital environments. Reality-based acquisition systems provide dense and reliable 3D reconstructions of the object of study. Laser scanning and photogrammetry are widely applied methodologies [1], [2] for capturing the basic 3D data to be used as input for the creation and development of digital models. While the acquisition of 3D survey data is a well-established practice, recent studies have explored the subsequent phase, which involves the processing and elaboration of raw data obtained from the survey. This representation phase, involving the reconstruction of the current or past form of a building based on survey acquisitions, requires a logical process. It is necessary to decode and organise the raw information obtained from laser scanning or photogrammetric survey in order to represent the morphological complexity of heritage buildings. It is necessary to understand the shape and geometry of the elements to be represented as well as their mutual relationships in order to build complete and intelligible representations [3]. Only through this phase, which requires a logical and operational process of segmenting and classifying the raw data obtained from the survey, can an accurate and complete 3D model be reconstructed and used as a support for the documentation and analysis of the cultural heritage object. Once the model has been reconstructed in a 3D environment, the geometric elements can be associated with semantic data, i.e. information about the historical asset. ABSTRACT The digitisation of cultural heritage objects paves the way for new approaches for the surveying and restitution of historical sites. With a view to the management of integrated programmes of documentation and conservation, the research is now focusing on the creation of information systems that link the digital representation of a building to semantic knowledge. Through reference to an emblematic case study of the Calci Charterhouse, also known as Pisa Charterhouse, this contribution illustrates an approach to be followed for transforming 3D survey information derived from laser scanning and photogrammetric techniques into semantically enriched 3D models. The proposed approach is based on both the recognition, segmentation and classification of semantic/geometric elements and the manual mapping of non-uniform rational basis spline elements on the original raw point cloud. For this shape recognition process, reference to architectural treatises and the vocabulary of classical architecture is a key step. The created building components are finally imported into a heritage building information modelling environment, where they are enriched with semantic information related to historical knowledge, documentary sources and restoration activities. mailto:valeria.croce@unifi.it ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 99 It is thus possible to identify the following phases in the process that leads to the attainment of structured models from raw data (Figure 1): I) acquisition of survey data through laser scanning, photogrammetry or a combination of both techniques; II) first data processing, including point cloud registration, subsampling and noise removal; III) segmentation and classification of architectural elements; IV) construction of the 3D model and semantic enrichment. The aim of this contribution is to show how these different phases of the process develop and support one another based on an examination of studies involving the recovery and conservation of cultural heritage objects. The construction of semantic models using heritage building information modelling (H-BIM) technologies is also considered. This workflow is illustrated in detail through an emblematic case study of the Calci Charterhouse (Certosa di Calci), also known as Pisa Charterhouse, an extensive architectural complex dating back to the 14th century that has been the subject of an integrated restoration and maintenance plan. The paper is organised as follows: in Section 2, the state of the art for the process of acquiring semantically rich 3D models from survey data is illustrated and discussed, with a focus on scan-to-BIM acquisition systems. Section 3 illustrates the materials and methods of the proposed approach, using the above-mentioned case study as an example. Finally, Section 4 presents concluding remarks and possible future developments of the work. 2. RELATED WORK The use of digital technologies in cultural heritage documentation and preservation applications is increasingly gaining ground, thanks to the development of 3D surveying and modelling systems. Laser scanning and photogrammetry make it possible to describe heritage objects by acquiring reliable data in the form of point clouds [4]. These data, however, are raw and not directly usable as they contain purely geometric or radiometric information. A data processing phase is thus necessary in order to transform the original point cloud into a geometric 3D model or a parametric object characterised by geometrical, topological and semantic attributes [5]. Obviously, the appropriate processing method strongly depends on the type of model to be obtained. In addition to web- based approaches involving the on-line sharing of 3D models with respective semantic annotations [6]-[8], a technology that is becoming increasingly popular for cultural heritage documentation and dissemination is the generation of BIM objects, which have typically been used in the field of new construction [9]. The procedure for the construction of H-BIM objects from relevant data is a reverse engineering technique known as scan- to-BIM [10]. The basic approach, which is the same as for BIM, is to create a digital database of the building, in which all of the information related to the study and documentation of the asset can be linked to a digital representation. By doing so, data can be organised, archived and updated at any time by the various professionals involved because the geometric representation of the object is associated with a semantic value. The first studies related to the application of BIM techniques to cultural heritage date back to 2009 and are attributed to Dore and Murphy, from the School of Surveying and Construction Management of the Dublin Institute of Technology [11]-[13]. Since these first studies, the complexities of the application of BIM to cultural heritage have been highlighted. In historical buildings as opposed to new buildings, the transition to parametric modelling requires the performance of an architectural survey, the recognition and classification of characteristic elements and recurring architectural typologies and the interpretation of parts of the building that are not directly visible or inspectable. The transition from survey outputs to semantic models requires much more time and skill in reconstructions of heritage buildings than in traditional BIM [14]-[16]. Indeed, historical architectures are commonly characterised by complex geometries that are difficult to model: successive stratifications resulting from different temporal states, anthropogenic modifications and degradation or even loss of elements. In other words, built heritage has heterogeneity in terms of both geometry and semantics [17]. Figure 1. General process for the creation of semantically rich models starting from surveys. ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 100 Recurrent elements and morphological units thus need to be identified and distinguished within this heterogeneity in terms of form [18], [19] and terminology [20]-[22] in order to obtain semantic representations. Consequently, it is common to refer to classical architectural treatises [23]-[25] or to the construction of controlled vocabularies of cultural heritage [26], [27]. For the case of H-BIM modelling, the result of the classification is the construction of a parametric family made up of similar building components [28], [29]. Different methods exist for the construction of the parametric model based on the original point cloud. They can be distinguished according to the respective degree of automation. Most automated procedures are founded on the construction of parametric surfaces on the basis of shape detection algorithms: several studies have employed the RANSAC algorithm [30]-[33] or other robust estimators derived from it [34]. Moreover, many commercial software (e.g. Edgewise) or plug-ins directly integrated in BIM platforms (e.g. scan-to-BIM and FARO) are equipped with tools for the semi-automatic reconstruction of walls and floors and, in some cases, for recognising the position of the openings of the window elements. Although semi-automatic labelling procedures have been given a great deal of attention in research, they are currently only applicable to the recognition of regular elements, such as slabs, walls and ceilings. They are still difficult to apply in the case of more complex architectural elements, which are often found in historic buildings. For this reason, the more widely used procedure in the field of cultural heritage is the manual mapping of three-dimensional geometries, starting from the point cloud: the construction of parametric elements is predicted [17], [29] through reference to historical and archival documentation and conceptual forms contained in treatises or architectural vocabularies [35], [36]. The modelling is thus performed by manually tracing the geometries on the point cloud [37]. This can be done directly on BIM platforms, e.g. by importing the point cloud into Autodesk Revit [5], [19], [38], [39] or, in the case of elements with more complex geometry, by first moving to free-form modelling software. In the latter situation, non-uniform rational basis splines (NURBS) [39]-[41] or mesh models [17], [42] are created externally and imported into a BIM environment at a later stage. The use of mesh within BIM platforms, however, involves a significant increase in file size and is less compatible with the principles of BIM parametric modelling [38]-[39], which is why this contribution focuses on modelling using NURBS. Referring again to the diagram in Figure 1, the current treatment of survey data, which includes segmentation and classification of the point cloud, is not yet standardised and may involve more or less automatic construction processes starting from the dense point cloud. In any case, once the model has been reconstructed, it can be enriched with qualitative information pertaining to the knowledge and study of the asset, which can be directly associated with 3D geometries [41]. In this context, the aim of the following contribution is to present a possible approach to the data processing phase, with the general objective of building a semantic model. For the complex being studied, such a model should be able to contain, within an overall BIM perspective, the different multi-temporal data related to construction phases including transformations and modifications, original and degraded status and past and future maintenance or restoration interventions. 3. MATERIALS AND METHODS 3.1. A pilot case study 3D modelling for cultural heritage and scan-to-BIM modelling techniques in particular require the application of multidisciplinary knowledge for classifying, managing and sharing information [41], [43]. In this section, this principle is illustrated with reference to a pilot case study, which is that of the already mentioned Calci Charterhouse, located on the slopes of Pisan mountains, a few kilometres from the Italian city of Pisa in Tuscany. That monumental complex has been the subject of an integrated conservation and recovery plan funded by the University of Pisa, which has involved many professionals in a variety of cognitive studies and research activities. The complex was founded in 1366 by Carthusian monks at the behest of the Archbishop of Pisa, Francesco Moricotti. Towards the end of the 16th century, some of the main rooms were decorated and frescoed. The structure then underwent changes and transformations during the 17th and 18th centuries, with the restructuring of some areas and the creation of additional decorations and frescoes. The complex became state property in 1866 and was later declared a national monument. Since 1972, the spaces of the Charterhouse have housed both the Natural History Museum, under the management of the University of Pisa, and the National Museum belonging to the Tuscan Museal Pole, under the direction of the Italian Ministry of Heritage and Cultural Activities and Tourism (Ministero per i Beni e le Attività Culturali e per il Turismo) [44], [45]. Visiting the complex, it is possible to find the spaces that remain from its ancient use, articulated according to the principles of a monastic life: a large cloister overlooked by the cells of the choir monks, built according to a modular scheme; a church with a sacristy and chapels; a refectory; a guest house; a chapter house with an adjoining cloister; the dormitory of the lay brothers and adjoining spaces for agricultural activities and manual work. Due to the large size of the architectural complex, surveys must be carried out using integrated acquisition techniques (Section 3.2). The survey restitution is thus made on a larger or smaller scale depending on the required level of detail and representation. Among the various environments surveyed, it is worth noting the Grand Ducal Cloister, a large internal courtyard whose construction began in 1614 under the direction of the architect Matteo Nigetti, replacing a previous 14th century cloister. For its reconstruction, Nigetti was inspired by the principles defined in the Sixth Book of the 16th century treatise entitled The Idea of the Universal Architecture, by Vincenzo Scamozzi [46]. The cloister has a cistern in the centre and is closed on four sides. Along the closing perimeter, there is a walkway on the first level, covered by cross vaults. The columns, surmounted by Ionic capitals and resting on a continuous base, are made of pietra serena, a typical grey sandstone widely used in Italy. The architecture of the Grand Ducal Cloister, characterised by its classical style and the repetition of typological and stylistic elements, makes it a suitable case study for the creation of a parametric model as a basis for semantic enrichment. 3.2. Integration of survey techniques The survey campaigns were carried out by the Scientific and Topographic Applications for Operative Survey (ASTRO) ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 101 laboratory of the University of Pisa and have been ongoing since 2013. In these campaigns, integrated survey systems were needed to restore the complexity of the architectural organism, with the objective of providing the different groups involved in the study and conservation activities with reliable and metrically correct geometric information. Together with traditional topographic surveys, laser scanning, ground-based photogrammetry and indoor and outdoor drone photogrammetry were applied. The vast size and scope of the Charterhouse necessitated the use of these different techniques. For a detailed description of the different types of survey used, please refer to [47]; the choice of the type of technique depended on the level of detail and restitution required, as well as on the complexity of the environment to be surveyed. In general, laser scanning enriched the ground measurements for the accurate restitution of elevation walls, plans and sections of the complex at 1:100 scale and of some significant portions at 1:50 scale (Figure 2 and Figure 3). For the most valuable architectural and decorative elements and for the frescoed walls, whose documentation required a more detailed restitution scale, photogrammetric surveys were performed. This made it possible to complement the geometric precision of the laser scanner with the radiometric precision (in terms of colour) of a photographic campaign (Figure 4 and Figure 5). Aerial photogrammetry using unmanned aerial vehicles was undertaken to represent parts that were not directly visible or inspectable, such as roofs, wells and underground tanks, as previously described in [48], [49]. The different acquisitions were all georeferenced with respect to a single reference system, called 'Charterhouse reference system', which was set up through the construction of a network of benchmarks [47]. The integrated use of surveying techniques made it possible to create geometric and radiometric documentation of the building components at different scales of representation that can be used to provide information regarding: - the current state, including the dimensions and arrangement of architectural elements; - the mapping of materials; - the state of degradation, e.g. the extension and depth of degradation mechanisms. The above mentioned procedure of operative survey is of paramount importance for the generation of 3D models that could be used by many experts for the collection, archiving and continuous updating of data related to the historical and documentary analysis of cultural heritage objects. 3.3. Proposed methodology The proposed methodological approach applies to buildings whose structure can be reconducted to ideal and canonised shapes, often by referring to architectural treatises or construction rules. The method simplifies the process of transforming survey data into 3D informative models and is very effective when recurring architectural elements are present. It is based on the following steps: 1. Point cloud optimisation, including scan registration, noise/obstacle removal and subsampling; 2. Semantic segmentation and classification performed manually over the registered point cloud; Figure 2. Grand Ducal Cloister: portion of the laser-scanned point clouds. Figure 3. Registration of laser-scanned point clouds for the area of the Grand Ducal Cloister. Each sphere indicates a scan location. Figure 4. Spatial resection of images on a 3D point cloud for the photogrammetric processing of frescoed walls. Figure 5. Photogrammetric survey of a frescoed room. ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 102 3. NURBS-based modelling of typical and recurring building components characterised by morphological complexity; 4. Generation of a parametric model of the heritage object in a BIM platform with NURBS-based elements imported as mass components; 5. Semantic enrichment of the model, in which documentary or analytical sources are connected to the geometry of building components. In steps 2 and 3, reference is usually made to architectural treatises for element recognition, profile generation and shape modelling. In step 4, a NURBS-based reconstruction is used to bring the identified repetitive typological elements back to their ideal geometries. This approach offers several advantages for the creation of parametric geometries and for the improvement of the scan-to-BIM workflow: - Elements can be created in a faster and more accurate manner by making use of a free-form modelling software, e.g. Rhinoceros, by Robert McNeel & Associates. - In comparison with the mesh-based methods illustrated in Section 2, a greater control over construction parameters of the ideal geometry is ensured. In fact, the proposed approach allows a direct conversion of the created elements into parametric components that can be used in H-BIM platforms with no loss of meaningful information. - The different elements to be represented are modelled by making reference to an ideal geometry, that is the one defined by architectural treatises and building canons. This ideal geometry is a benchmark for all other elements with similar characteristics and is created through mathematical relationships in a unique way: parameter specifications and modelling commands are used to create the NURBS elements. In contrast to mesh-based methods, each ideal geometry can be easily managed, converted and integrated in a unique way [50]. This systematic and mathematical approach to digital heritage, based on the simplification of shapes and the reference to ideal geometries, presents an advantage in the parametric logic of H- BIM platforms, where series of smart objects, represented in 3D, are enriched with a large amount of technical information, related for example to geometry, materials, thickness, etc. [28], [40], [51]. The interoperability between NURBS modelling software and H- BIM platforms has also been verified [50], [52]. The reference to ideal shapes makes it possible to describe similar elements that are characterised by significant variations in terms of conservation states, e.g. medieval capitals or severely decayed and deformed architectural objects. In such cases, the information on the different conservation states can be attached to each single element to aid in the reconstruction of disparity and displacement maps between similar components [37]. 3.4. Semantic segmentation and creation of parametric elements The construction of semantically rich models from survey data, as specified in Section 2, requires a rational process in which the main building components are determined: in the multitude of detected points in the raw cloud, it is necessary to distinguish a finite number of parameters. The methodological approach proposed in Section 3.3 is illustrated hereafter with reference to the area of the Grand Ducal Cloister of the Charterhouse. The laser scanning surveys of the area were used to demonstrate the suitability of the approach following the completion of the scan registration, point cloud subsampling and cleaning (including the removal of obstacles and disturbing elements). A video showing the original point cloud that was used for the creation of the parametric model is available at [53]. Figure 6. Semantic segmentation of the 3D point cloud. ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 103 The case study is emblematic since the cloister has an architectural scheme that is common in religious complexes: it is made up of an open space, square or rectangular in shape, surrounded by vaulted corridors that open onto the central void with a series of arches and colonnades. It is therefore an area of the building characterised by the recurrence of basic elements, with typological characters, shapes and construction details intrinsically bound up with each other. In our case, for the recognition of the components, reference was made to architectural treatises and in particular to the already mentioned treatise The Idea of the Universal Architecture, by Vincenzo Scamozzi [46], which inspired the original design of the cloister. With reference to the classification and proportions of the architectural orders illustrated by Scamozzi, a manual semantic segmentation of the point cloud was performed. The identification and recognition of the main components thus followed the classification illustrated in Figure 6. Using this approach to the segmentation and subsequent classification of the cloud, each identified element could be modelled separately as a single building component. To parametrically reconstruct the classified typological elements, the steps are: 1. the creation of NURBS surfaces, mapped on the original point cloud; 2. the adoption of these surfaces as modelling components in BIM-based software. The use of NURBS surfaces is of fundamental importance since it simplifies the reconstruction process: once the profile of the architectural component has been manually traced over the point cloud (with reference to the canons of architectural treatises) the surface can be built by extrusion, loft, revolution or sweep along this profile, as illustrated in Figure 7 for a column shaft. The construction process was performed via the software Rhinoceros and its integrated graphical algorithm editor Grasshopper. Figure 8 shows the NURBS surfaces for the base and the Ionic capital of the column. The volutes of the Ionic capital were reconstructed by following the procedure illustrated by the 16th century architect Andrea Palladio in his treatise The Four Books of Architecture [54]. Complex-shaped building components were created as NURBS elements in a Rhinoceros environment and were only later imported into the BIM platform Revit, using the Standard ACIS Text (.SAT) extension. The creation of a SAT format file made it possible to use the created geometry as a mass component in the BIM environment, thus making it possible to perform other calculations and cuts over the imported 3D element. Using this procedure, it was possible to reconstruct a georeferenced H-BIM model that presents the conceptual and parametric forms of the Grand Ducal Cloister’s building components. In the 3D model, typological and recurring elements were originally represented in Rhinoceros, using NURBS modelling, and then imported to Revit. Reference to architectural treatises enabled the reconstruction of those specific elements following a univocal procedure with established proportions and shapes, regardless of which person built the parametric component. The model thus constructed, as displayed in Figure 9, provides the basis for the insertion of semantic information, i.e. documentary and analytical knowledge about the heritage asset. 3.5. Information storage and update An information system based on the approach to 3D reconstruction described in Section 3.4 is built in a BIM environment to manage both the geometry and the semantics of the Grand Ducal Cloister. The process is based on the connections between the parametric 3D representation of the building and the heterogeneous data derived from documentary and analytical sources (such as shape and dimensions and metric information). As for studies of the materials, the state of deterioration or possible interventions, a shared and univocal model of data insertion within the 3D model could be applied. The proposed scheme is illustrated in Figure 10: information on ‘material’, ‘state of degradation’, ‘diagnosis’ and type of performed or needed ‘intervention’ is associated with each typological element. Also, references to ‘documentary sources’, provided in the form of text files or images, can be linked to the element through the connection to a directory. Following the same principle, the external 3D survey files that were initially used for the construction of the parametric forms can be linked to the model using the ‘data acquisition’ toolbox. All of the toolboxes are implemented in the Revit platform as project parameters. Thus, the semantic structuring can take place in two distinct ways: the information can simply be linked to the individual building components, or, in a more refined process, it can be connected to only a part of a component (Figure 11). In the first case, the information is simply associated with a certain element in its totality; in the second case, however, the semantic data does not concern the whole element, but only a part of it. This happens, for example, in cases in which it is helpful to define a detailed map or annotate local characters, e.g. a phenomenon of superficial degradation that is not present along the whole extension of the element but concerns only a certain section. Figure 7. Creation of the column shaft by means of a loft operation applied to circular surfaces with the centre located on the shaft axis. ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 104 In this situation, the annotation is inserted by means of a trick: reference is made to the construction of a mass component representing the degradation along the surface of the element. This operation is performed directly on Revit and makes it possible to map 3D information onto the model. The database setting is structured in such a way that it is always possible to retrieve information about: - the professional or scholar who entered the annotation; - the time the information was entered; - the period to which the information refers. With this expedient, the different temporal states of construction and evolution of the model can also be displayed. The information system thus built, if customised according to the heritage building’s needs, allows for the management, recovery and updating of metric and semantic data. In other words, multiple representations and their associated information can be organised around the same semantic description model. In the future, cross-referencing capabilities are expected to improve, making it possible to establish correlations between levels of information, e.g. between materials and degradation status, decay level and improper interventions or diagnosis and repair and restoration interventions. The digitally enriched model is flexible and dynamic: the semantic base can be upgraded if new information of a different nature or from another research domain needs to be added. For instance, the envisaged future work on correlation queries or even mechanical or physicochemical properties of a material linked to a typological element may require the database entries to be updated by entering additional fields and tags onto the database structure. Different levels of database access and permissions (entry, modification of a field, read-only access, entry verification) are assigned to the different actors involved, based on their respective roles and responsibilities. This ensures greater control over the information entry and also makes it possible to keep track of the person who has modified, updated or added certain information within the 3D model. Further advantages of the proposed approach include: - the ability to track the time of updates and thus describe the evolution of a given annotation over time; - information overlay, meaning the ability to handle multiple levels of information. Different temporal states, for instance, can be displayed over the same 3D representation. In this multilayer information management system, it is also possible to overlap surface annotations (local components) with volume annotations (global components, describing a parametric element in its entirety). - Despite the huge potential of this information system, it is worth noting that the insertion of surface-type annotations within BIM platforms remains cumbersome and time consuming, especially when compared to the insertion of volume-type annotations. In addition, BIM commercial platforms were originally designed for new constructions, and their application to cultural heritage is still limited. Such platforms typically allow the performance of volume-type annotations and are not designed to manage local annotations on the elements. The inclusion of localised information within a certain parametric component, which is necessary for the analysis of existing structures, is still required. Figure 8. Generation of capital and base elements of the column via NURBS. The modelling phase was guided by reference to architectural treatises, [55]. ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 105 Figure 9. Recognition of ideal shapes over the point cloud for the creation of the 3D model. Figure 10. Proposal for the organisation of the information to be associated with parametric elements. ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 106 Indeed, limitations in the overlay of surface and volumetric annotations, in terms of file size, response time, functionality of the model, etc., have not yet been fully assessed in the current state of the research. These aspects are under study, and they will be the subject of future publications. Further developments of information modelling systems to support cultural heritage documentation should therefore aim at improving the system’s ability to handle local annotations to describe e.g. a degraded part, the change or loss of material, or a decorated element. 4. CONCLUSIONS AND FUTURE WORK Heritage studies involve a plurality of actors and resources and produce multiple and heterogeneous data sets. This leads to a broad range of disparate information and knowledge. To sort the different information in a logical way, it is increasingly essential to create digital systems localised within a 3D representation of the object of study in which different types of data can be entered, stored and displayed. This approach combines the acquisition, restitution and documentation phases of heritage studies. With the creation of semantically enriched 3D models, the qualitative information derived from studies of cultural heritage objects can be combined with quantitative information linked to the geometry of the represented elements and to their reciprocal spatial relations. To digitally represent heritage objects, however, a logical process of segmenting and classifying the existing elements is required. In Pisa Charterhouse, the size of the complex itself and the presence of multidisciplinary teams were the factors that determined the urgency of creating a shared system of information management and archiving. Tackling these issues, this contribution presents a method of manual segmentation and classification starting from the raw 3D survey data. The method is based on the study of architectural canons and on the use of NURBS surfaces for the modelling of complex building components. The final goal is the reconstruction of an enriched 3D model, where information related to the knowledge, maintenance and recovery of the heritage building is suitably stored in a 3D digital environment. The presented approach uses a manual reconstruction of the components over the point cloud and the subsequent importation and handling of the geometries in an H-BIM environment. Future developments follow the automation approach of the scan-to-BIM process via shape recognition algorithms. An interesting possible application could integrate modern machine learning and deep learning techniques in order to automate the classification of recurrent geometric elements and repetitive building components. Although these techniques are already widely used in the geospatial field, only a few studies refer to their application in the cultural heritage sector [56], [57]. The implementation of such technologies could greatly aid in the construction of semantic models from laser scanning or photogrammetric survey data and improve the implementation of H-BIM processes. ACKNOWLEDGEMENTS The work presented in this paper was undertaken within the project ‘Studi conoscitivi e ricerche per la conservazione e la valorizzazione del Complesso della Certosa di Calci e dei suoi Poli Museali’ (Cognitive studies and research aimed at the conservation and enhancement of the Calci Charterhouse Complex and its Museum Poles), funded by the University of Pisa. Additionally, this research was partially supported by the Vinci2019 Project, which was promoted by the Université Franco-Italienne. Section 3.5 includes the partial results of a thesis by the student Ilaria Grassi, entitled ‘Applicazione della metodologia HBIM al Chiostro Granducale della Certosa di Calci: restituzione semantica e mappatura tridimensionale del degrado’ and drafted under the supervision of M. G. Bevilacqua, M. Martino, S. Landi and A. Piemonte. REFERENCES [1] M. G. Bevilacqua, G. Caroti, A. Piemonte, A. A. Terranova, Digital technology and mechatronic systems for the architectural 3D metric survey, Mechatronics for Cultural Heritage and Civil Engineering 92 (2018) pp. 161-180. DOI: 10.1007/978-3-319-68646-2-7 [2] F. Remondino, Heritage recording and 3D modeling with photogrammetry and 3D scanning, Remote Sens. 3 (2011) pp. 1104-1138. DOI: 10.3390/rs3061104 [3] L. De Luca, Methods, formalisms and tools for the semantic-based surveying and representation of architectural heritage, Appl. Geomat. 6 (2014) pp. 115-139. DOI: 10.1007/s12518-011-0076-7 [4] M. G. Bevilacqua, G. Caroti, A. Piemonte, P. Ruschi, L. Tenchini, 3D survey techniques for the architectural restoration: the case of S. Agata in Pisa, ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLII-5/W1 (2017) pp. 441-447. DOI: 10.5194/isprs-archives-XLII-5-W1-441-2017 [5] F. J. López, P. M. Lerones, J. Llamas, J. Gómez-García-Bermejo, E. Zalama, A review of heritage building information modeling (H-BIM), Multimodal Technologies Interact. 2(21) (2018) pp. 1- 29. Figure 11. Project parameters linked to mass components. https://doi.org/10.1007/978-3-319-68646-2-7 https://doi.org/10.3390/rs3061104 https://doi.org/10.1007/s12518-011-0076-7 https://doi.org/10.5194/isprs-archives-XLII-5-W1-441-2017 ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 107 DOI: 10.3390/mti2020021 [6] Aïoli, A reality-based 3d annotation cloud platform for the collaborative documentation of cultural heritage artefacts, 2018. Online [Accessed 22 March 2021]. http://aioli.cloud [7] R. Quattrini, R. Pierdicca, C. Morbidoni, Knowledge-based data enrichment for HBIM: exploring high-quality models using the semantic-web, J. Cult. Herit. 28 (2017) pp. 129-139. DOI: 10.1016/j.culher.2017.05.004 [8] R. Garozzo, F. Murabito, C. Santagati, C. Pino, C. Spampinato, Culto: an ontology-based annotation tool for data curation in cultural heritage, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-2/W5 (2017) pp. 267-274. DOI: 10.5194/isprs-archives-XLII-2-W5-267-2017 [9] M. Lo Turco, M. Mattone, F. Rinaudo, Metric survey and BIM technologies to record decay conditions, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-5/W1 (2017) pp. 261-268. DOI: 10.5194/isprs-archives-XLII-5-W1-261-2017 [10] M. Pepe, D. Costantino, A. Restuccia Garofalo, An efficient pipeline to obtain 3D model for HBIM and structural analysis purposes from 3D point clouds, Appl. Sci. 10(1235) (2020) pp. 1- 21. DOI: 10.3390/app10041235 [11] C. Dore, M. Murphy, Current state of the art historic building information modelling, Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLII-2/W5 (2017) pp. 185-192. DOI: 10.5194/isprs-archives-XLII-2-W5-185-2017 [12] M. Murphy, E. McGovern, S. Pavia, Historic building information modelling - adding intelligence to laser and image based surveys of European classical architecture, ISPRS Journal of Photogrammetry and Remote Sensing 76 (2013) pp. 89-102. DOI: 10.1016/j.isprsjprs.2012.11.006 [13] M. Murphy, E. McGovern, S. Pavia, Historic building information modelling (HBIM), Structural Survey 27(4) (2009) pp. 311-327. DOI: 10.1108/02630800910985108 [14] H. Macher, T. Landes, P. Grussenmeyer, Point clouds segmentation as base for as-built BIM creation, ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. II-5/W3 (2015) pp. 191-197. DOI: 10.5194/isprsannals-II-5-W3-191-2015 [15] P. Tang, D. Huber, B. Akinci, R. Lipman, A. Lytle, Automatic reconstruction of as-built building information models from laser- scanned point clouds: a review of related techniques, Automation in Construction 19(7) (2010) pp. 829-843. DOI: 10.1016/j.autcon.2010.06.007 [16] R. Volk, J. Stengel, F. Schultmann, Building information modeling (BIM) for existing buildings—literature review and future needs, Automat. Constr. 38 (2014) pp. 109-127. [17] X. Yang, Y.-C. Lu, A. Murtiyoso, M. Koehl, P. Grussenmeyer, HBIM modeling from the surface mesh and its extended capability of knowledge representation, ISPRS Int. J. Geo-Inf. 8(7) (2019) p. 301. DOI: 10.3390/ijgi8070301 [18] F. Noardo, Architectural heritage semantic 3D documentation in multi-scale standard maps. J. Cult. Herit. 32 (2018) pp. 156-165. DOI: 10.1016/j.culher.2018.02.009 [19] N. Bruno, R. Roncella, HBIM for conservation: a new proposal for information modeling, Remote Sens. 11(15) (2019) p. 1751. DOI: 10.3390/rs11151751 [20] M. Previtali, R. Brumana, C. Stanga, F. Banfi, An ontology-based representation of vaulted system for HBIM, Appl. Sci. 10(4) (2020) p. 1377. DOI: 10.3390/app10041377 [21] G. Bacci, F. Bertolini, M. G. Bevilacqua, G. Caroti, I. Martínez- Espejo Zaragoza, M. Martino, A. Piemonte, HBIM methodologies for the architectural restoration. the case of the ex-church of San Quirico all’Olivo in Lucca, Tuscany, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-2/W11 (2019) pp. 121-126. DOI: 10.5194/isprs-archives-XLII-2-W11-121-2019 [22] T. Messaoudi, P. Véron, G. Halin, L. De Luca, An ontological model for the reality-based 3D annotation of heritage building conservation state, J. Cult. Herit. 29 (2018) pp. 100-112. DOI: 10.1016/j.culher.2017.05.017 [23] A. Baik, From point cloud to Jeddah Heritage BIM Nasif Historical House – case study, Digital Applications in Archaeology and Cultural Heritage 4 (2017) pp. 1-18. DOI: 10.1016/j.daach.2017.02.001 [24] M. Acierno, S. Cursi, D. Simeone, D. Fiorani, Architectural heritage knowledge modelling: an ontology based framework for conservation process, J. Cult. Herit. 24 (2017) pp. 124-133. DOI: 10.1016/j.culher.2016.09.010 [25] M. Doerr, Ontologies for cultural heritage, in: Handbook on Ontologies. S. Staab, R. Studer (editors). Springer, Berlin, 2009, ISBN 978-3-540-70999-2, pp. 463-486. DOI: 10.1007/978-3-540-92673-3_21 [26] P. Pauwels, S. Zhang, Y. Lee, Semantic web technologies in AEC industry: a literature overview, Autom. Constr. 73 (2017) pp. 145- 165. DOI: 10.1016/j.autcon.2016.10.003 [27] D. Simeone, S. Cursi, A platform for enriching BIM representation through semantic web technologies, Proceedings of the Joint Conference on Computing in Construction, Heraklion, Greece, 4 – 7 July 2017, pp. 423-430. DOI: 10.24928/JC3-2017/0323 [28] R. Brumana, D. Oreni, L. Barazzetti, B. Cuca, M. Previtali, F. Banfi, Survey and scan to BIM model for the knowledge of built heritage and the management of conservation activities, in: Transformation of the Design, Construction and Management Processes of the Built Environment. Research for Development. B. Daniotti, M. Gianinetto, S. Della Torre (editors). Digital Springer, Cham, 2020, ISBN 978-3-030-33570-0, pp. 391-400. [29] X. Yang, M. Koehl, P. Grussenmeyer, Parametric modeling of as- built beam framed structure in BIM environment, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-2/W3 (2017) pp. 651–657. DOI: 10.5194/isprs-archives-XLII-2-W3-651-2017 [30] M. A. Fischler, R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Commun. ACM 24 (1981) pp. 381-395. DOI: 10.1145/358669.358692 [31] I. Anagnostopoulos, V. Patrucean, I. Brilakis, P. Vela, Detection of walls, floors, and ceilings in point cloud data, Proc. of the Construction Research Congress 2016, San Juan, Puerto Rico, 31 May – 2 June 2016, pp. 2302-2311. DOI: 10.1061/9780784479827.229 [32] C. Thomson, J. Boehm, Automatic geometry generation from point clouds for BIM, Remote Sens. 7 (2015) pp. 11753-11775. DOI: 10.3390/rs70911753 [33] S. Hong, J. Jung, S. Kim, H. Cho, J. Lee, J. Heo, Semi-automated approach to indoor mapping for 3D as-built building information modeling, Comput. Environ. Urban Syst. 51 (2015) pp. 34-46. DOI: 10.1016/j.compenvurbsys.2015.01.005 [34] H. Macher, T. Landes, P. Grussenmeyer, From point clouds to building information models: 3D semi-automatic reconstruction of indoors of existing buildings, Appl. Sci. 7 (2017) p. 1030. DOI: 10.3390/app7101030 [35] A. Adan, X. Xiong, B. Akinci, D. Huber, Automatic creation of semantically rich 3D building models from laser scanner data, Autom. Constr. 31 (2013) pp. 325-337. DOI: 10.1016/j.autcon.2012.10.006 [36] Y. Arayici, Towards building information modelling for existing structures, Struct. Surv. 26(3) (2008) pp. 210-222. DOI: 10.1108/02630800810887108 [37] S. Bruno, M. De Fino, F. Fatiguso, Historic building information modelling: performance assessment for diagnosis-aided information modelling and management, Autom. Constr. 86 (2018) pp. 256-276. DOI: 10.1016/j.autcon.2017.11.009 https://doi.org/10.3390/mti2020021 http://aioli.cloud/ https://doi.org/10.1016/j.culher.2017.05.004 https://doi.org/10.5194/isprs-archives-XLII-2-W5-267-2017 https://doi.org/10.5194/isprs-archives-XLII-5-W1-261-2017 https://doi.org/10.3390/app10041235 https://doi.org/10.5194/isprs-archives-XLII-2-W5-185-2017 https://doi.org/10.1016/j.isprsjprs.2012.11.006 https://doi.org/10.1108/02630800910985108 https://doi.org/10.5194/isprsannals-II-5-W3-191-2015 https://doi.org/10.1016/j.autcon.2010.06.007 https://doi.org/10.3390/ijgi8070301 https://doi.org/10.1016/j.culher.2018.02.009 https://doi.org/10.3390/rs11151751 https://doi.org/10.3390/app10041377 https://doi.org/10.5194/isprs-archives-XLII-2-W11-121-2019 https://doi.org/10.1016/j.culher.2017.05.017 https://doi.org/10.1016/j.daach.2017.02.001 https://doi.org/10.1016/j.culher.2016.09.010 https://doi.org/10.1007/978-3-540-92673-3_21 https://doi.org/10.1016/j.autcon.2016.10.003 https://doi.org/10.24928/JC3-2017/0323 https://doi.org/10.5194/isprs-archives-XLII-2-W3-651-2017 https://doi.org/10.1145/358669.358692 https://doi.org/10.1061/9780784479827.229 https://doi.org/10.3390/rs70911753 https://doi.org/10.1016/j.compenvurbsys.2015.01.005 https://doi.org/10.3390/app7101030 https://doi.org/10.1016/j.autcon.2012.10.006 https://doi.org/10.1108/02630800810887108 https://doi.org/10.1016/j.autcon.2017.11.009 ACTA IMEKO | www.imeko.org March 2021 | Volume 10 | Number 1 | 108 [38] O. E. C. Prizeman, HBIM and matching techniques: considerations for late nineteenth- and early twentieth-century buildings, J. Archit. Conserv. 21(3) (2015) pp. 145-159. DOI: 10.1080/13556207.2016.1139852 [39] L. Barazzetti, F. Banfi, R. Brumana, M. Previtali, Creation of parametric BIM objects from point clouds using nurbs., Photogramm. Rec. 30 (2015) pp. 339-362. DOI: 10.1111/phor.12122 [40] F. Banfi, HBIM generation: extending geometric primitives and BIM modelling tools for heritage structures and complex vaulted systems, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-2/W15 (2019) pp. 139-148. DOI: 10.5194/isprs-archives-XLII-2-W15-139-2019 [41] R. Brumana, S. Della Torre, M. Previtali, L. Barazzetti, L. Cantini, D. Oreni, F. Banfi, Generative HBIM modelling to embody complexity (LOD, LOG, LOA, LOI): surveying, preservation, site intervention—the Basilica di Collemaggio (L’Aquila), Appl. Geomat. 10 (2018) pp. 545-567. DOI: 10.1007/s12518-018-0233-3 [42] X. Yang, M. Koehl, P. Grussenmeyer, Mesh-to-BIM: from segmented mesh elements to BIM model with limited parameters, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-2 (2018) pp. 1213-1218. DOI: 10.5194/isprs-archives-XLII-2-1213-2018 [43] D. P. Pocobelli, J. Boehm, P. Bryan, J. Still, J. Grau-Bové, BIM for heritage science: a review, Herit. Sci. 6 (2018) art. 30. DOI: 10.1186/s40494-018-0191-4 [44] M. A. Giusti, M. T. Lazzarini, La Certosa di Pisa a Calci, Pacini, Pisa, Italia, 1993, ISBN 978-88-7781-081-6. [45] M. L. Orlandi, Carte dell’Archivio della Certosa di Calci, Presentazione di Banti O., Pacini, Pisa, Italia, 2002, ISBN 978-88- 7781-457-9. [46] V. Scamozzi, L'idea della Architettura Universale, Venezia, 1615. [47] V. Croce, G. Caroti, A. Piemonte, M. G. Bevilacqua, Geomatics for cultural heritage conservation: integrated survey and 3D modeling, Proc. of the IMEKO TC4 International Conference on Metrology for Archaeology and Cultural Heritage, MetroArchaeo, Florence, Italy, 4–6 December 2019, pp. 271-276. Online [Accessed 22 March 2021] https://www.imeko.org/publications/tc4-Archaeo- 2019/IMEKO-TC4-METROARCHAEO-2019-50.pdf [48] G. Caroti, A. Piemonte, I. Martínez-Espejo Zaragoza, G. Brambilla, Indoor photogrammetry using UAVs with protective structures: issues and precision tests, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLII-3/W4 (2018) pp. 137-142. DOI: 10.5194/isprs-archives-XLII-3-W4-137-2018 [49] V. Croce, G. Caroti, A. Piemonte, Assessment of earthquake- induced damage level on buildings: analysis of two different survey methods for a case study, ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci XLII-2/W15 (2019) pp. 351-358. DOI: 10.5194/isprs-archives-XLII-2-W15-351-2019 [50] F. Diara, F. Rinaudo, Building archaelogy documentation and analysis through open source HBIM solutions via NURBS modelling, ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci XLIII-B2-2020 (2020) pp. 1381-1388. DOI: 10.5194/isprs-archives-XLIII-B2-2020-1381-2020 [51] D. Oreni, R. Brumana, F. Banfi, L. Bertola, L. Barazzetti, B. Cuca, M. Previtali, F. Roncoroni, Beyond crude 3D models: from point clouds to historical building information modeling via NURBS, Lecture Notes in Computer Science, 8740 (2014) pp.166-175. DOI: 10.1007/978-3-319-13695-0_16 [52] F. Diara, F. Rinaudo, From reality to parametric models of cultural heritage assets for HBIM, ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci XLII-2 (2019) pp. 413-419. DOI: 10.5194/isprs-archives-XLII-2-W15-413-2019 [53] The Grand-Ducal cloister of the Calci Charterhouse. Online [Accessed 28 March 2021] https://youtu.be/eNiiLaL-DFU [54] A. Palladio, I quattro libri dell’architettura (1570), libro I, capitolo XVI, Hoepli, Milano, 1945. [55] V. Croce, G. Caroti, G., L. De Luca, K. Jacquot, A. Piemonte, P. Véron, From the Semantic Point Cloud to Heritage-Building Information Modeling: A Semiautomatic Approach Exploiting Machine Learning. Remote Sens. 2021, 13, 461. DOI: 10.3390/rs13030461 [56] E. Grilli, F. Remondino, Classification of 3D digital heritage, Remote Sens. 11 (2019) p. 847. DOI: 10.3390/rs11070847 [57] F. Poux, P. Hallot, R. Neuville, R. Billen, Smart point cloud: definition and remaining challenge, Proc. of the 11th 3D Geoinfo Conference, Athens, Greece, 20 – 21 October 2016, pp. 119-127. DOI: 10.5194/isprs-annals-IV-2-W1-119-2016 https://doi.org/10.1080/13556207.2016.1139852 https://doi.org/10.1111/phor.12122 https://doi.org/10.5194/isprs-archives-XLII-2-W15-139-2019 https://doi.org/10.1007/s12518-018-0233-3 https://doi.org/10.5194/isprs-archives-XLII-2-1213-2018 https://doi.org/10.1186/s40494-018-0191-4 https://www.imeko.org/publications/tc4-Archaeo-2019/IMEKO-TC4-METROARCHAEO-2019-50.pdf https://www.imeko.org/publications/tc4-Archaeo-2019/IMEKO-TC4-METROARCHAEO-2019-50.pdf https://doi.org/10.5194/isprs-archives-XLII-3-W4-137-2018 https://doi.org/10.5194/isprs-archives-XLII-2-W15-351-2019 https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-1381-2020 https://doi.org/10.1007/978-3-319-13695-0_16 https://doi.org/10.5194/isprs-archives-XLII-2-W15-413-2019 https://youtu.be/eNiiLaL-DFU https://doi.org/10.3390/rs13030461 https://doi.org/10.3390/rs11070847 https://doi.org/10.5194/isprs-annals-IV-2-W1-119-2016