________________________________________________________________________________ Geoinformatics CTU FCE 2011 330 3DMADMAC|SPECTRAL: HARDWARE AND SOFTWARE SOLUTION FOR INTEGRATED DIGITIZATION OF 3D SHAPE, MULTISPECTRAL COLOR AND BRDF FOR CULTURAL HERITAGE DOCUMENTATION Robert SITNIK1, Grzegorz MĄCZKOWSKI1, Jakub KRZESŁOWSKI1, Tomasz GADZIŃSKI1 1Warsaw University of Technology, Faculty of Micromechanics and Photonics Boboli 8, 02-525 Warsaw, Poland r.sitnik@mchtr.pw.edu.pl g.maczkowski@mchtr.pw.edu.pl j.krzesłowski@mchtr.pw.edu.pl Keywords: 3D shape measurement, structured light, multispectral imaging, BRDF, 3D printing Abstract: In this article a new 3D measurement system along with the study on 3D printing technology is presented from the perspective of quality of reproduction. In the first part of the paper the 3DMADMAC|SPECTRAL system which integrates 3D shape with additional color and angular reflectance measurement capabilities is presented (see Figure 1). The shape measurement system is based on structured light projection with the use of a DLP projector. The 3D shape measurement method is based on sinusoidal fringes and Gray codes projection. Color is being measured using multispectral images with a set of interference filters to separate spectral channels. Additionally the set up includes an array of compact light sources for measuring angular reflectance based on image analysis and 3D data processing. All three components of the integrated system use the same greyscale camera as a detector. The purpose of the system is to obtain complete information about shape, color and reflectance characteristic of mea sured surface, especially for cultural heritage objects - in order to create high quality 3D documentation. In the second part of the paper the 3D printing technology will be tested on real measured cultural heritage objects. Tests allow to assess measurement and color accuracy of reproduction by selected 3D printing technology and shed some light on how current 3D printing technology can be applied into cultural heritage. 1. INTRODUCTION Preservation of cultural heritage is an important task for every modern society. Recently tools which employ new technology for scanning and digitization emerge. They facilitate precise, contactless measurement of artifacts‟ features such as shape, color or reflectance distribution. There are already several solutions which address this problem, for example Simon et. al. uses separate laser 3D scanner and custom made multispectral camera and develop a method of manual merging of shape and color data [1], whereas Mansouri et. al. propose an integrated system which uses structure light projection for shape measurement and multispectral camera for color acquisition [2]. None of these solutions address a problem of reflectivity measurement. An approach presented in this work is based on an integrated measurement system which is capable of automatic measurement of shape using structured light projection technique, color with the aid of multispectral acquisition and angular reflectance distribution. The presented measurement system utilizes a single CCD (charge-coupled device) detector for all above kinds of measurement, which makes it insensitive to alignment of data which come from different phases of the integrated measurement. The measurement device along with data processing path is presented in the first part of this paper, whereas the second part is dedicated to a 3D printing technology which gives completely new opportunities in this field. It allows for creation of real copies of the previously digitized objects for promotional and educational purposes. We conclude our work with examples of cultural heritage objects from Central Office of Measures in Warsaw, Poland which were scanned in order to create their digital models and then printed as a test of copying technology and the whole data processing path. 2. MEASUREMENT SYSTEM The measurement system used in this work consists of a developed 3D scanner based on structured light projection, multispectral camera and a specially designed device for angular reflectance measurement. All these components are ________________________________________________________________________________ Geoinformatics CTU FCE 2011 331 described in more details below, along with data processing procedures. Additional subsection approaches a 3D printing procedure which allows copying of registered objects. 2.1 3D shape measurement The 3D shape measurement method uses the (3D) Measurement with Algorithms of Directional Merging And Conversion (3DMADMAC) system [3]. This method of measurement is based on a structured light technique with digital sine patterns and Gray codes projection [4]. Collected images allow for calculation of absolute phase which corresponds to surface coordinates in the direction perpendicular to image plane that is along the depth of the measurement volume. On the other hand transverse surface coordinates are possible to calculate, because in the previous process of geometrical calibration each pixel of the detector has assigned a ray which intersects the image plane in specific point. The single measurement results in a cloud of points which represents shape of the scanned surface, where every point maps to a single pixel of the detector. The system consists of a Digital Light Projector (DLP) and a matrix detector (an industrial charged-coupled device (CCD) camera or a digital camera). Its components along with a measurement geometrical setup are shown in Figure 9. The 3DMADMAC system can be customized depending on end user requirements regarding size of measurement volume, amount of measurement points and duration of a single measurement. It also consists of a set of Software Development Kit tools which extend its functionality and automate all required measurement and data analysis algorithms. Figure 9. 3D shape measurement system concept. 2.2 Multispectral color measurement Color of a measured surface is captured with a multi-spectral approach. The aim of this solution was to estimate reflectance spectrum of the measured surface in every point, so that its color can be calculated independently of illumination conditions and appearance under different light sources can be simulated. This approach is especially important in case of cultural heritage objects, because it gives meaningful information for art conservators regarding storage and display of artefacts. The custom built camera was built to register images in 10 spectral bands with the aid of interference filters. The filter wheel is placed between the camera matrix and the lens. It has 11 slots, because additional empty window without a filter is necessary for performing shape measurement which uses the same detector ( Figure 10). The lens mount is located outside the case which allows for simple lens replacement according to required measurement conditions. The multispectral capture system uses analytic calibration procedure described in previous work [5]. It is based on capturing images of white reference plate for light source spectrum compensation and images of uniform background to compensate for illumination distribution and spectral filters' angular characteristic. Additionally transverse shifts of images due to positioning errors can be eliminated. ________________________________________________________________________________ Geoinformatics CTU FCE 2011 332 Figure 10. The concept of multispectral color measurement system and directional illuminators for BRDF measurement. 2.3 Angular reflectance measurement Another device is employed to measure angular reflectance distribution of surface. The result is a bi-directional reflectance distribution function (BRDF) modelled with Phong parameters [6]. The device comprises of a set of light sources distributed on a grid pattern and illuminating measurement volume. Each illuminator estimates lambertian source and allows directional illumination of the investigated surface ( Figure 10). Pictures are captured with all illuminators turned on sequentially which gives a collection of reflectance values in the function of illumination angle and serves as a BRDF estimation which leads to Phong parameters calculation [7]. 3. DATA PROCESSING Raw data coming from the measurement are simply black-and-white images and a set of operations is needed to extract meaningful data from them. For clearer description they can be divided in a similar way as measurement stages because the data coming from them are for the most of the processing path independent and they are combined into concise digital model at the end. 3.1 3D shape calculation The result of the shape measurement is a cloud of points which is a set of vertices distributed in a Cartesian space which resemble the location of surface of the measured object. In order to calculate it from measurement images a calibration data is necessary. It consists of straight lines‟ equations associated with every pixel of the detector, calculated in real units and a phase distribution interpolated over whole measurement volume, calculated from phase shifted sine fringes which are projected during the measurement procedure [Chyba! Záložka není definována.]. A cloud of points is a result of measurement from a single direction, so in order to scan the whole surface of arbitrary shape the measurement procedure should be repeated for multiple directions. When clouds of points are calculated the mask which can be applied to measurement images is known. It indicates which pixels from these images belong to the scanned surface. In other data processing procedures only these points are being calculated, which means that only the data associated with points in the cloud are further processed. In the next step the clouds of points from several directions are merged in order to estimate the whole measured surface. Because the object is placed in the measurement volume manually the individual clouds have different local coordinate systems and they need to be fitted together. Automatic algorithms in 3DMADMAC system are used to transform the clouds into a single, global coordinate system. They use two steps. In the first one a coarse fitting is achieved based on local curvature or texture distribution of the overlapping parts of the clouds. The second stage uses ICP algorithms for precise fit [8]. Once clouds are fit they are filtered and smoothed to eliminate noise and ill-calculated coordinates. Afterwards they are simplified to reduce the amount of data and finally a triangulation procedure [9] is applied to create a mesh which is possible to export in VRML format and display in standard 3D imaging and manipulation software. ________________________________________________________________________________ Geoinformatics CTU FCE 2011 333 3.2 Color calculation Images acquired during measurement correspond to different narrow spectral ranges and they estimate the spectral response of the measured surface combined with the characteristic of the light source. Proposed calibration with the use of white reference plate and uniform background helps to extract reflectance spectrum [Chyba! Záložka není definována.] in 10 intervals corresponding to interference filters used in the experiment. After that data are extrapolated over whole visible spectrum by the means of spline fitting between measurement points. The spectral reflectance calculated this way for every point in the cloud is used for color estimation in XYZ coordinates according to CIE rules [10], with previously chosen illuminant D65. Further processing employs color calculation in sRGB color space. Finally overlapping data from different directions are used to equalize lightness variations which exist because of non uniform illumination distribution during measurement. After above procedure it is possible to create texture for independently obtained triangle mesh. First the surface of the object is divided into separate areas with small variations in normal vectors directions and averaged normals are treated as independent texture mapping directions. This allows dividing color data into patches which are later mapped to different parts of the triangle mesh. This approach allows for precise texture mapping regardless of the measured shape. 3.3 BRDF calculation The set of images acquired with illumination from different directions allows estimation of bi-directional reflectance distribution function in every point of the surface. The fast algorithm [Chyba! Záložka není definována.] which calculates coefficients of the Phong model is used [Chyba! Záložka není definována.]. It assumes that the BRDF is symmetrical and does not depend on light wavelength, but nevertheless it gives interesting results for surfaces made from different materials, as expected. Three Phong parameters for diffuse, specular and shininess coefficients are stored as an additional texture which uses the same mapping as the color texture described previously and can be used as a material information in 3D rendering software. Data obtained this way are used for displaying in virtual environment only and does not take part in copying procedure which exploits shape and color. 4.3D PRINTING There are several 3D printing methods currently available on the market. Among many properties they differ in material used to create models, precision and speed. An interesting solution is 3D Printing (3DP) technology which is the only one which offers possibility to print in color. This gives a good opportunity for copying models of cultural heritage objects, because color reproduction is an important aspect of their appearance. Therefore this method was applied and evaluated in the presented research. 3D Printing is the fastest known method of producing 3D models in the rapid prototyping technology [11]. The use of the method is incremental (additive). The printing process is carried out in several stages. Initially, the created computer model is uploaded by a specialist into print management software. Then it is subdivided into hundreds of horizontal layers. The printer then applies a thin layer of powder on the ground in the working chamber with a roller. Then a layer of a color adhesive (glue) is spread on the powder surface by the printing head. A place where the cartridge is leaving the adhesive corresponds to the contours of the model in a given cross- section (layer). The procedure is shown in Figure 11. After applying all layers of the powder and glue, the printed model needs to be cleaned of non-glued plaster and to be hardened. Color on the surface is obtained thanks to separate heads spreading glue (resin) in different base colors: cyan, magenta, yellow, black and colorless which are mixed to create desired color similarly as in traditional ink-jet printing technology. 4.1 Shape reproduction In this study a Zcorporation ZPrinter 650 machine was used. According to the manufacturer, it can print models in 390,000 colors with a resolution of 600×540dpi. The size of the working chamber is 254mm × 3Ř1mm × 203mm. Single layer thickness lies between 0.089 and 0.102mm. Printed wall thickness may amount to 0.1mm. Many factors resulting from applied technology lead to spread in dimensions of created models. These include movement of the shaft scattering successive layers of plaster; positioning and movement of the printing head and platform in the print chamber; spreading of the glue; gravitational compaction of powder layers (a phenomenon of squash) as well as temperature and humidity. Conducted evaluation established several guidelines for more precise printing which makes a full use of the printer‟s capabilities. Among the most important ones; the longest dimension of the model should be placed in the horizontal plane. For the vertical planes projection is weaker. On the other hand long elements that extend beyond the main block of the model should be placed parallel to the acting forces of gravity which may be contradictory to the previous assumption. Alternatively long items that cannot be placed as suggested shall be printed with so called supports. This shows that individual decisions should be made according to specific printed shapes. The greatest distortions caused by the phenomenon of squash arise in the middle of the workspace. One way of minimizing this effect is to put the most massive elements of the model on the bottom so that the loose layer of plaster between the platform and the model was as small as possible. Other recommendations suggest that humidity and temperature in the ________________________________________________________________________________ Geoinformatics CTU FCE 2011 334 room where the printer is placed should be appropriate (according to the manufacturer's recommendations) and fixed. Additionally visible deterioration in surface reproduction is present due to wear of printing heads. Used heads cause weaker representation of the flatness of the surface. The models are rough; one can see places where each layer was stuck. In the following application these suggestions were taken into consideration with good results. Figure 11. Procedure of 3D printing. 4.2 Color reproduction Separate aspect of printing considers color reproduction. The manufacturer does not provide any automatic profiling tool for translating color into limited color space of the 3D printer. The only guideline is a set of 729 color samples uniformly distributed in RGB color space which are prepared to print as a reference map of colors possible to reproduce by the printer. The user is supposed to compare color patches and manually choose the one that best resembles the color which he intends to reproduce. Obviously this approach will be impossible to apply when printing an object with complex texture where every point may have different color. Therefore a profiling method for the printer was proposed. It uses mentioned color patches which were measured by the spectrophotometer to establish their CIE LAB coordinates. The acquired data constitute a 3D look up table (LUT) which provides transformation between RGB values sent to printer and device independent color space. The problem to solve is to transform an arbitrary LAB coordinates, not necessarily existing in the LUT into the printer color space (red ball in Figure 12). The proposed solution first finds the nearest LAB coordinates which belong to the LUT and its immediate neighbours (blue balls). All of found coordinate sets have their representations in printers RGB color space which are used to calculate unknown RGB coordinates as their weighted average. The weights are established as inverse of Cartesian distances between the required out of LUT LAB coordinates and its nearest neighbours in LAB color space. Figure 12. Construction of 3D printer color profile. Blue balls represent adjacent points in look up table between LAB and RGB color spaces, whereas red ball stands for out of LUT point which RGB coordinates are to be established. Performed analysis shows that suggested gamut mapping compresses the input color gamut which is usually larger to fit into the printer gamut by shifting out of gamut colors to the boundaries of the smaller gamut. Additionally conducted ________________________________________________________________________________ Geoinformatics CTU FCE 2011 335 tests show that final color appearance is very much influenced by after printing treatment which includes hardening the surface with resin and drying up the model. The acceptable repeatability is very hard to achieve because the process is made by hand and it would require very skilful operator. These shows that differences in achieved color can be considerable and even an application of a color profile may not produce good results in the face of significant technological inaccuracy. 5. DIGITIZATION RESULTS Courtesy of Central Office of Measures in Warsaw three test objects were investigated in order to create their digital models and copies with the aid of 3D printing technology. The original objects are shown in Figure 13, followed by Table 1 where their overall characteristics are provided. They were made from different materials and have different surface finish. The wooden volume reference cup is lacquered, the occasional medal is polished and the iron weight reference has worn, rusty surface. Figure 13. Measured objects: a) wooden cup serving as an old volume reference; b) occasional medal; c) old weight reference. Figure 14 shows virtual models of the digitized objects which were rendered in custom made software. The program allows for displaying color texture and reflectance properties as well as changing illumination and object‟s location with real time refreshing of its appearance. property object volume reference occasional medal weight reference material wood steel iron dimensions [mm] Ř0×60×60 70×70×5 73×Ř0×40 measurement directions 28 6 21 total points 0.8mln 1.55mln 4.78mln total triangles 90000 35000 60000 texture patches 16 4 16 texture resolution per patch 1024×1024 204Ř×2048 1024×1024 Table 1. Measured objects properties. ________________________________________________________________________________ Geoinformatics CTU FCE 2011 336 Figure 14. Models of measured objects: a) volume reference cup; b) occasional medal; c) weight reference. The last step presented step concerns copying the measured objects. Results of this procedure are shown in Figure 15. However shape in all the cases is preserved well, the color reproduction is poor because of limited gamut of the 3D printer and complicated handmade post printing treatment of copies. These results show that improvements can be made in color processing procedures as well as in printing technology. Figure 15. Printed copies of the measured objects: a) volume reference cup; b) occasional medal; c) weight reference. 6. CONCLUSIONS The integrated measurement system presented in this work facilitates digitization of cultural heritage objects. It performs automatic measurement of 3D shape, multispectral color and surface reflectivity with the aid of a single detector, therefore no manual data alignment is necessary. It can be applied to create precise digital models for the purpose of display or storing information about precious artefacts. Moreover it utilizes a 3D printing technology to build copies of the measured objects for promotion or education. As a result of conducted research sample measurements outcome is provided to show the possibilities of presented technology. However shape and color measurement techniques are already well developed there are still issues concerning unbiased merging data from different directions, precision and repeatability of color measurement procedure and printing technology which allows for objects reproduction with limited precision. Particularly 3D printer‟s color mapping is questionable. This gives a potential for further research, but even know it is very promising technology. ________________________________________________________________________________ Geoinformatics CTU FCE 2011 337 7. ACKNOWLEGDEMENTS This work was performed under the grant No. PL0097 financed by the Norwegian Financial Mechanism and EEA Financial Mechanism (2004-2009). 8. REFERENCES [1] Simon C. Huxhagen U. Mansouri A. Heritage A.Boochs F.Marzani F.: Integration of high resolution spatial and spectral data acquisition systems to provide complementary datasets for cultur al heritage applications, Proceedings of SPIE, 7531(1), 2010, 75310L. [2] Mansouri A.Lathuiliere A. Marzani F. Voisin Y. Gouton P.: Toward a 3D multispectral scanner: an application to multimedia, Proceedings of IEEE Multimedia , 14(1), 2007, 40-47. [3] Sitnik R. Kujawinska M. Woznicki J.: Digital fringe projection system for large-volume 360-deg shape measurement, Opt. Eng., 41, 2002, 443-449. [4] Osten W. Nadeborn P. Andrae P.: General hierarchical approach in absolute phase measurement, Proceedings of SPIE, 2860, 1996, 2-13. [5] Mączkowski G., Sitnik R., Krzesłowski J. Integrated Method for 3D Shape and Multispectral Color Measurement, J. Imaging Sci. Technol. 55(3), 2011, 030502-(10). [6] Phong B.T.: Illumination for computer generated pictures, Communications of the ACM, 18, 1975, 311-317 [7] Krzesłowski J. Sitnik R. Maczkowski G.: Integrated three-dimensional shape and reflection properties measurement system, Appl. Opt., 50, 2011, 532-541. [8] Besl P. McKay N.:, A method for registration of 3-D shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence, 14, 1992, 239-256. [9] Sitnik R. Karaszewski M.: Optimized Point Cloud Triangulation for 3D Scanning Systems, Machine Graphics & Vision 17, 2008, 349-371. [10] Wyszecki G. Stiles W. S.: Color Science: Concepts and Methods, Quantitative Data and Formule, New York, John Wiley&Sons, 2000. [11] Gibson I. Rosen D. Stucker B.: Additive manufacturing technologies. Rapid prototyping to direct digital manufacturing, New York, Springer, 2010.