ACTA IMEKO  ISSN: 2221‐870X  June 2015, Volume 4, Number 2, 4‐9    ACTA IMEKO | www.imeko.org  June 2015 | Volume 4 | Number 2 | 4  Single image geometry inspection using inverse endoscopic  fringe projection  Steffen Matthias, Christoph Ohrt, Andreas Pösch, Markus Kästner, Eduard Reithmeier   Institute of Measurement and Automatic Control, Leibniz Universität Hannover, Germany      Section: RESEARCH PAPER  Keywords: inverse fringe projection; fiberscopy; endoscopy; sheet bulk metal forming  Citation: Steffen Matthias, Christoph Ohrt, Andreas Pösch, Markus Kästner, Eduard Reithmeier, Single image geometry inspection using inverse endoscopic  fringe projection, Acta IMEKO, vol. 4, no2, article 2, June 2015, identifier: IMEKO‐ACTA‐04 (2015)‐02‐02  Editor: Paolo Carbone, University of Perugia, Italy  Received April 17, 2014; In final form February 19, 2015; Published June  2015  Copyright: © 2015 IMEKO. This is an open‐access article distributed under the terms of the Creative Commons Attribution 3.0 License, which permits  unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited  Funding: This work was funded by the German Research Foundation (DFG) within the Collaborative Research Center (CRC) / TR 73  Corresponding author: Steffen Matthias, e‐mail: steffen.matthias@imr.uni‐hannover.de    1. INTRODUCTION  The development of a new metal forming process demands several kinds of new technological approaches. Starting with new geometrical tool designs, fitted materials have to be found and analyzed. Additionally, process parameters such as forming speeds and forces have to be optimized in respect to a maximum output, while maintaining low tool abrasion and long durability. Concerning the new method of cold sheet bulk metal forming (SBMF) [3], which is currently being developed in an exemplary manufacturing process, assembly ready geometries shall be generated by a combined process of sheet and bulk forming. This results in both drawing and comprehensive forces on the tool in the same process step. The combination leads to early wearing effects on the SBMF-tool that may cause deficient workpieces. These need to be identified early in the manufacturing process in order to keep rejection costs at a minimum. Proper simulations for quality assessments can only be based on measurement data gathered from the experimental process. To generate datasets of the process, measurements obtained in a complete production cycle are necessary. These requirements can only be met with fast areal measurement devices. Desired cycle times are in the range of one second and less, while the standard deviation of the measured geometry data should be 100 µm or less compared to the reference. Experiences show that fringe projection fulfills these ABSTRACT  Fringe projection is an important technology for the measurement of free form elements in several application fields. It can be applied  to measure geometry elements smaller than one millimeter. In combination with deviation analysis algorithms, errors in fabrication  lines  can  be  found  promptly  to  minimize  rejections.  However,  some  fields  cannot  be  covered  by  the  classical  fringe  projection  approach. Due to shadowing, filigree form elements on narrow or internal carrier geometries cannot be captured. To overcome this  limitation, a fiberscopic micro fringe projection sensor was developed [1]. The new device is capable of resolutions of less than 15 µm  with uncertainties of about 35 µm in a workspace of 3x3x3 mm³.  Using standard phase measurement techniques, such as Gray‐code and cos²‐patterns, measurement times of over a second are too  long for in‐situ operation. The following work will introduce an approach of applying a new single image measuring method to the  fiberscopic system, based on  inverse fringe projection [2]. The fiberscopic fringe projection system employs a  laser  light source  in  combination with a digital micro‐mirror device (DMD) to generate fringe patterns. Fiber optical image bundles (FOIB) are used as well  as gradient‐index lenses to project these patterns on the specimen. This advanced optical system creates high demands on the pattern  generation algorithms to generate exact inverse patterns for arbitrary CAD‐modelled geometries. Approaches of optical simulations of  the  complex  beam  path and  the  drawbacks  of the  limited  resolutions  of  the  FOIBs  are  discussed.  Early  results  of  inverse  pattern  simulations using a ray tracing approach of a pinhole system model are presented. ACTA IMEKO | www.imeko.org  June 2015 | Volume 4 | Number 2 | 5  requirements very well [4]. However, inner geometries such as SBMF-tools are not measureable at optimum angle, which reduces the lateral resolution of the measurement with the cosine of the camera angle towards the surface normal. Common fringe projection sensors usually combine fringe generator and camera unit in a fixed housing with a predefined triangulation angle [5]. To overcome these limitations, a fiberscopic fringe projection system was developed [6], which can be positioned close to specifically stressed areas of a SBMF- tool. These, for example, filigree side form elements are exposed to drawing and bulk-forming forces at the same time. In these areas early abrasion is most likely and therefore needs to be taken care of at the very beginning of the development process. To reduce the time duration of measurements, the inverse fringe projection technique allows the detection of geometry deviations using a single projection of an object-specific inverse fringe pattern. An adapted projector pattern is derived from the specimen’s geometry and the desired camera pattern. Deviations in geometry lead to differences in the camera image which can be easily detected. Different approaches exist to calculate the inverse pattern. Li et al. [7] present an algorithm to calculate inverse patterns by processing measurements of a reference object. Following measurements will be able to detect deviations from the initially measured reference geometry. A different approach calculates the inverse pattern by the use of a CAD model of the geometry and a mathematical model of the fringe projection system [8]. The latter solution has the advantage of using a virtual reference defined by the tool designer instead of a manufactured reference. In the following sections the principle of the fiberscopic system will be presented together with the adaptation of a model based inverse fringe projection approach. Following an introduction of the optical setup, difficulties that arise from the image fibers and special lenses are discussed. In order to verify the pattern generation, parts of the system are simulated using ray-tracing software. The methods to reduce the measurement time to a minimum in order to keep it in the range of the cycle time of the process will be explained. 2. FIBERSCOPIC FRINGE PROJECTION  2.1. Principle setup  Figure 1 shows the basic design of the newly developed setup. To achieve a high depth of field (DOF) with FOIBs, a good fiber coupling at a sufficient intensity is required. By their nature, laser light sources offer high intensities as well as a collimated beam profile, which makes them almost ideal for fiber coupling. Only the coherent nature of lasers complicates the use in fringe projection since reflections on a diffuse reflecting specimen, as it is common in structured light measurements, create speckle contrast automatically. Sometimes the speckle contrast can be higher than the contrast of the projected fringe pattern and thereby prevents an accurate evaluation [6]. To overcome this, a rotating diffuser is installed in the system together with micro lens arrays that act as beam shapers. Since the diffuser changes the speckle formation faster than the frame rate of the CCD, it leads to an integration of randomly distributed patterns which results in a smooth light intensity image [6]. The beam shaper changes the Gaussian into a flat top distribution that is fitted to the size of the DMD, which is used for the pattern generation. The pattern is focused to the 1.7 mm input aperture of the FOIB and guided 1:1 to the measurement area. The image on the specimen is projected through gradient index (GRIN) rod lenses with high numeric apertures, which are directly attached to the FOIB. The distorted image of the fringe pattern is captured by a similar GRIN-FOIB arrangement in a defined triangulation angle and guided to a 5 megapixel CCD camera for computing. 2.2. Data processing  In structured light projection sequences of binary Gray-code followed by cos²-phase shift patterns are common in commercial fringe projection systems [4][5]. The method achieves high accuracies and is robust to background light and therefore can be used in a wide spectrum of measurement tasks. However, the projection and acquisition of the sequences take the major time of the whole measurement. Additionally, due to the significant loss of resolution in the 100,000 fiber image bundle, binary patterns start to fade on the cost of contrast in fiberscopic fringe projection. Additionally, for a robust phase measurement the Gray-code sequence needs to have twice the frequency of the continuous cos²-pattern to allow the compensation of errors in the phase-unwrapping process. It has to be noted, that only the information from the cos²-pattern is used in the later process, while the Gray-code patterns are only used for phase unwrapping of the continuous pattern. However, due to the limited resolution of the fiber bundles, the maximum pattern frequency is very limited. Thus, for the endoscopic system, unwrapping methods using lower frequency cos²-patterns are superior as the highest frequency of the cos²- patterns can be higher than by using Gray-code assisted unwrapping. Also, using the unwrapping technique described by Tao Peng [9], the number of patterns to project may be reduced, leading to a faster measurement process. To obtain 3D-measurement data from the 2D-phase maps, the system is calibrated using a pinhole camera model and a black box model for the projector [10]. 2.3. Resulting data  The generated data of the specimen’s geometry is saved in a point cloud in Cartesian coordinates. Sets of data in different geometries like plane, spheres and gearings were recorded and compared to data of other measurement systems, such as commercial fringe projection sensors and coordinate measurement machines (CMMs), as well as to the original CAD design data. For the task the commercial and widely accepted software Polyworks (InnovMetric Software Inc) was used. It Figure 1. Schematic of the fiberscopic fringe projection setup.   ACTA IMEKO | www.imeko.org  June 2015 | Volume 4 | Number 2 | 6  showed that the presented system achieves accuracies of ± 10 µm for a 3D-measurement area of 3x3x3 mm³ at standard deviations of 35 µm. This made it capable for the indicated task in SBMF. However, even with reduced pattern resolution and the hereby reduced number of images, measuring speeds are not sufficient for measurements in cycle time of the SBMF process for holistic measurements of each part. 3. INVERSE FIBERSCOPIC FRINGE PROJECTION  3.1. Inverse fringe projection  Inverse fringe projection enables the detection of 3D- geometry deviations utilizing only a single adapted pattern instead of a sequence of patterns [11]. This pattern is adapted to both the geometry of the specimen and the optical characteristics and positioning of the fringe projection system, as seen in Figure 2. For the inverse approach, the intended geometry needs to be known as a CAD model. The measurement setup, consisting of a camera, a projector and an ideal specimen, needs to be modelled in a virtual environment. This virtual setup must approximate the optical properties of the real setup with sufficient accuracy; therefore, a calibration procedure of the real setup must be undertaken to identify the model parameters of the system. A common approach is to approximate both the camera and the projector using the pinhole model. The pinhole model describes how a point of a 3D object in the camera reference frame is projected on the 2D camera image [12]. The pinhole model consists of a camera matrix K, seen in equation (1), which contains the so-called intrinsic parameters. 0 0 0 0 1 (1) The parameters and describe both pixel size and focal length, while and model the principal point. Additional intrinsic parameters may be used to approximate lens distortion effects [13]. Extrinsic camera parameters describe the transformation from the world coordinate system, which can for example be the coordinate system of a CAD-file, to the camera coordinate system. Three parameters describe the rotation, while three additional parameters model the necessary translation. The image of an object point X in the world coordinate system can be expressed in homogenous coordinates by equation (2), where is the camera matrix and the transformation matrix. (2) A complete model includes the intrinsic parameters of both camera and projector as well as a transformation matrix describing the relation of camera and projector and the transformation to the world coordinate system. The inverse pattern can be simulated by ray-tracing using the virtual system. In the first step of the simulation, the inverse projection pattern is calculated by inversion of the path of light propagation. Therefore, the camera is modeled as a projector and “emits” a straight, equidistant, structured light pattern onto the specimen (CAD model). The projector works in this context as a camera and is used to “capture” the diffuse reflection into a raster image with the DMD-pixels remodeled as sensor pixels. Of course, this is only possible utilizing the virtual ray-tracing-based system. When this inverse projection pattern, obtained by simulating the virtual system, is then projected by the real projector onto a real specimen (of the same shape and pose as the virtual specimen), the real camera will capture the beforehand-defined straight, equidistant light intensity pattern. Geometry deviations of the real specimen, however, will lead to distortions in the camera image which can be detected robustly using fast 2D image processing techniques. The amount of deviation of the 2D fringe pattern can be related to three dimensional geometry deviations by a linearized defect model called sensitivity map. Thus, quantitative information about the geometry defect is obtained. No reconstruction of point cloud data or processing of three dimensional data is required after acquisition of the measurement data resulting in a low latency time from measurement to result. The method is applicable to check for allowable geometry tolerances up to a µm scale. Poesch [11] showed the proof of principle on a macroscopic fringe projector. However, the optical path of a fiberscopic fringe projector as described above is far more complex to model for inverse simulation. New artifacts, such as pixelation effects due to the fiber bundle, have to be taken care of. Challenges arise from the simulation of the FOIB and the in- and out-coupling, but also from the beam shaping and speckle removal stage. Artifacts that arise from the optical elements which are unusual for classic fringe projection systems may complicate the scenario. 3.2. Simulation of a fiberscopic fringe projector with ray‐tracing  The simulation of a fiberscopic fringe projection system requires several uncommon components in the beam path, which not only increases the complexity of the projector design, but also extends the average calculation time of one complete simulation run significantly. With the applied simulation software FRED (Photon Engineering LLC) and a state of the art multicore computer one simulation cycle with a sufficient number of rays requires a processing time of about one hour. Since the optimization includes several kinds of optics and their respective positions numerous runs are inevitable. The setup was created component by component and connected afterwards in a combined simulation environment. In a first step, the telescope and rotating diffuser were optimized for minimal divergence of the beam and a highly randomized speckle distribution at each diffuser rotation. These elements mainly affect the available space in the fringe projection sensor. For the telescope the best simulation results were found with two 60 mm lenses which focus a laser with 2 mm diameter on the rotating diffuser plate. The rays detected behind the last lens are recorded and used as light source for the next simulation step. The flat top generator consists of the fly-eye micro lens arrays and a Fourier-lens. Figure 2. Inverse fringe projection applied to a specimen.  ACTA IMEKO | www.imeko.org  June 2015 | Volume 4 | Number 2 | 7  The pattern that is generated by the fly-eye optics is shown on “Analysis Surface 1” in Figure 3. The displayed section represents that the fly-eyes successfully break the wave front of the Gaussian distribution of the laser and generate a flat top light distribution. The several maxima flatten with increasing distance to the Fourier-Lens and equalize to a homogeneous level at the stage of the DMD. The optimization criteria were a highly consistent light pattern that exactly fits to the size of the 0.7” DMD. For the DMD in the first place a fixed micro mirror array was generated that projects half of the rays to a dump and the other half to the projecting FOIB. The pattern is shown on “Analysis Surface 2” in Figure 3. Again, the pattern is saved and used as the source for the next, most computation time intensive part of the simulation. The fiber coupling part consists of an in-coupling objective, which focusses the fringe image to a 1.7 mm diameter at the entrance of the FOIB. In a first attempt, the fiber bundles were simulated by modelling 100,000 individual fibers of 1 m length. The calculation of several hundred reflections in each fiber caused several days of simulation time. For comparison a 10 mm long version was designed. The simulated images of both fiber lengths were very similar, so that for simplification the 10 mm version was chosen for the following simulations (Fiber coupling in Figure 3). The fringe pattern is projected using an adapted GRIN-lens that is fitted to the diameter of the FOIB. Under the triangulation angle a second GRIN-lens acquires the image in the second FOIB to a second coupling optic, which projects the resulting fringe pattern to the last analysis surface that represents the CCD of the actual setup. With the finished model, the behavior of a projected pattern in the FOIB can be predicted. Especially the decrease of the 1 MP image to 100,000 pixels is very uncommon for fringe projection systems as well as the availability of the comparatively low resolutions. Using the image of the last analysis surface in the way described in section 3.1 enables first indications of possible qualities of an inverse fiberscopic fringe projection approach. Unfortunately, the number of parameters of this detailed model is too high to allow robust model identification. Thus, a simpler approach, similar to the pinhole model, needs to be found in order to calibrate the virtual system to the actual endoscopic fringe projection system. 3.3. Practical approach  For the continuous quality control of the abrasion within a SBMF-process it is necessary to compare the measured data to the reference. Figure 4 shows the measurement of a SBMF- deep drawing tool with the fiberscopic system. In the first step a reference measurement is obtained by projecting the common Gray-code and phase shift sequence which requires at least 12 images. The point cloud derived from the phase-map using the calibration data is calculated in the calibrated coordinate system for the system. To simulate the inverse pattern, it is crucial to know the pose and position of the tool in this system specific coordinate system. Both can be obtained by calculating a rigid body transform by aligning measured reference points to the corresponding points in the CAD-model using Polyworks. From the resulting transformation matrix and the system model an inverse pattern can be generated by the help of ray-tracing. After this initial pattern generation step, inspection can be performed by capturing just one image of the inverse pattern. Distortions in the camera image can only be seen at sections where abrasion (either welding on, or wear off effects) has taken place. The measuring procedure only takes several Figure 3. Complete ray tracing design simulation of the fiberscopic fringe projection system.  ACTA IMEKO | www.imeko.org  June 2015 | Volume 4 | Number 2 | 8  hundred microseconds, which is below the beforehand made requirement. From the data an easy “out-of-tolerance” study can be derived, providing information about necessary tool exchanges or process stability. Desired accuracies in the field of SBMF are in the range of 50 to 100 µm in order to be able to judge wear effects in terms of production quality. To verify the feasibility of this procedure, the system has been calibrated using the common pinhole model for both camera and projector. It was necessary to replace the projector micro lens with a standard GRIN lens as the adapted lens can only be described with this model with low accuracy. Unfortunately, this decreases the depth of field of the endoscopic fringe projection system. As a first proof of principle and verification of the calibrated model parameters, an inverse pattern was simulated for a step calibration standard. The calibration standard is a diffuse plane with a 1 mm deep and 2 mm wide cutout. The standard was positioned arbitrarily in the fringe projection system’s coordinate system. Pose and position were estimated by capturing a point cloud with the standard fringe projection technique and fitting it to the CAD- model using Polyworks. The simulated inverse projector pattern can be seen in the top left image in Figure 5. Simulation of the pattern has been performed in a non-optimized MATLAB ray-tracing environment and required a processing time of about 2 minutes with a model consisting of about 30 triangles. As this simulation needs to be performed only initially, the processing does not limit the measurement time of the system. More advanced software is capable of calculating ray-tracing with complex models in less than one minute. The cutout can be seen in the center of the inverse pattern, while the slight deformation of the pattern towards the outer areas of the geometry is a result of the calibrated camera and projector distortion. In order to assess the quality of this approach, the inverse pattern was projected back onto the calibration standard using the endoscopic system. The grayscale image on the right in Figure 5 shows the obtained camera image. It can be seen that the fringe patterns are parallel throughout the camera image, as it has been defined in the inverse pattern generation step. Slight deviations are visible near the step in the standards geometry, resulting from slight positioning errors. The used step standard makes these misalignments visible due to its non-continuous geometry. With continuous geometries, such as the gearings on the bulk-sheet metal forming tools, the inverse pattern is less sensitive to misalignments. To enable an automated inspection of the image, a phase map was calculated by processing the fringe pattern in the marked rectangular part of the camera image. As expected, the phase-map shows only small deviations along the vertical axis. The cutout leads to a narrow shadowed area in the camera image, at which the phase unwrapping fails. This leads to the artifact seen in the phase image. 4. CONCLUSIONS AND FUTURE ACTIVITY  The work showed that the principle of fiberscopic fringe projection works with common pattern generation approaches such as Gray-code and phase shift or encoded phase shift. With highly collimated light sources and beam shaping a sufficient depth of focus can be achieved even with micro GRIN optics attached to the FOIBs. The working system has been mapped into a computer model for the simulation software FRED. This model is the basis for the evaluation of virtual model based inverse fringe projection. We showed that the generation of inverse patterns works for the fiberscopic system. For practical evaluations, the simulation model has been simplified by using the pinhole model for projector and camera. Arbitrary CAD geometry can be used for pattern generation after pose and position of the sensor have been calibrated. The next steps of the work on the fiberscopic fringe projection sensor will be the installation in a SBMF-machine with a high precision positioning system to examine the results of the inverse fiberscopic fringe projection with more complex geometries. Possible enhancements of the technique include improvements of the 2D-image processing as well as an improved lighting model for the ray-tracing simulation. Apart from lens distortion modelling for the projector GRIN lens, the Figure 4. Measurement of a SBMF deep drawing tool (top) with following analyses of the point cloud in respect of abrasion effects (bottom).  Figure 5. Inverse measurement of a calibration standard.  ACTA IMEKO | www.imeko.org  June 2015 | Volume 4 | Number 2 | 9  largest limitation of the demonstrated approach is currently its sensitivity to positioning errors and vibrations, as the position of the sensor head to the desired measurement area is defined during the simulation of the inverse patterns. ACKNOWLEDGEMENT  The authors would like to thank the German Research Foundation (DFG) for funding the project B6 "Endoscopic geometry inspection" within the Collaborative Research Center (CRC) / TR 73. REFERENCES  [1] C. Ohrt, S. Matthias, M. Kästner, E. Reithmeier, “Fast endoscopic geometry measurement with fiber-based fringe projection for inner geometries”, Journal of the CMSC, Vol. 7, no. 2, 2012, pp. 10-14. [2] A. Pösch, T. Vynnyk, M. Kästner, O. Abo-Namous, E. Reithmeier, “Virtual Inverse Fringe Projection”, CMSC Conference, Reno, 2010. [3] M. Merklein, J.M. Allwood, B. Behrens, A. Brosius, H. Hagenah, K. Kuzman, K. Mori, E. Tekkaya, A. Weckenmann, “Bulk forming of sheet metal”, Annals of the CIRP, Vol. 61, no. 2, 2012, pp. 725-745. [4] M. Kästner, “Optische Geometrieprüfung präzisions- geschmiedeter Hochleistungsbauteile”, Shaker, Aachen, 2010, ISBN 3832294430. [5] G. Frankowski, M. Chen, T. Huth, “Real-time 3D Shape Measurement with Digital Stripe Projection by Texas Instruments Micromirror Devices (DMD)”, Proceedings of the Conference on Three-Dimensional Image Capture and Applications III, San Jose, CA, USA, 2000. [6] C. Ohrt, M. Kästner, E. Reithmeier, “Endoscopic geometry inspection by modular fiber optic sensors with increased depth of focus”, Proc. SPIE 8082, Optical Measurement Systems for Industrial Inspection VII, 2011. [7] W. Li, T. Bothe, W. Osten, M. Kalms, “Object adapted pattern projection—Part I: generation of inverse patterns”, Optics and lasers in engineering, 41(1), 2004, pp. 31-50. [8] A. Pösch, T. Vynnyk, E. Reithmeier, “Using inverse fringe projection to speed up the detection of local and global geometry defects on free-form surfaces”, SPIE Optical Engineering+ Applications, International Society for Optics and Photonics, 2012, pp. 85000B-85000B. [9] T. Peng, “Algorithms and models for 3-D shape measurement using digital fringe projections”, University of Maryland, PhD- Thesis, 2007. [10] J. Vargas, M.J. Terrón-López, J.A. Quiroga, “Flexible calibration procedure for fringe projection profilometry”, Optical Engineering, Vol. 46, no. 2, 2007, p. 023601. [11] A. Pösch, T. Vynnyk, E. Reithmeier, “Fast detection of geometry defects on free-form surfaces using inverse fringe projection”, Journal of the CMSC, Vol.8, no. 1, 2013. [12] R. Hartley, A. Zisserman, “Multiple View Geometry in Computer Vision”, Cambridge University Press, 2004, ISBN 0521540518. [13] C. Brown, “Close-range camera calibration”, PHOTOGRAMMETRIC ENGINEERING, Vol. 37, No. 8, 1971, pp. 855-866.