Acta Polytechnica CTU Proceedings https://doi.org/10.14311/APP.2023.40.0123 Acta Polytechnica CTU Proceedings 40:123–128, 2023 © 2023 The Author(s). Licensed under a CC-BY 4.0 licence Published by the Czech Technical University in Prague USE OF LIGHTWEIGHT FFT-BASED ALGORITHMS FOR REMOTE MEASUREMENTS USING DIC Slávek Zbirovský∗, Václav Nežerka Czech Technical University in Prague, Faculty of Civil Engineering, Department of Physics, Thákurova 7, 166 29 Prague 6, Czech Republic ∗ corresponding author: slavek.zbirovsky@fsv.cvut.cz Abstract. Abrupt collapse events that result in casualties due to failure of load-bearing structures have been reported in recent years. Continuous remote monitoring of these structures, assessing the behavior under operational load, could provide a reliable prevention, as the developing damage manifests itself through a loss of stiffness, and therefore excessive deflections. Digital image correlation (DIC) algorithms based on fast Fourier transform (FFT) were implemented in a small computer unit Raspberry Pi, capable of remote operation in a low-energy demand mode. The unit is fully autonomous, capable of localizing QR code targets and extracting encoded information. This strategy allows us to instantly evaluate displacements of QR codes attached to any surface and send the data to a server. The software was developed in Python, allowing for two modes: the online mode for current data evaluation with information transfer via the MQTT protocol and the offline mode used for retrospective evaluation. Keywords: DIC, FFT, Raspberry Pi, MQTT, QR code. 1. Introduction Digital image correlation (DIC) is an optical non- contact method for measuring surface deformations, operating on the principle of comparing a reference image with an image after event. Due to the pos- sible interchangeability of individual pixels, groups of pixels (hereafter subsets) are compared, in which the grayscale of individual pixels is evaluated. The correlation function is used as a comparison criterion. To ensure sufficient reliability of the correlation of the individual subsets, the sample must be provided with a random, high contrast isotropic pattern, which is called a “pattern” [1]. The displacement of the subset is often described by a first-order shape function that includes the dis- placement, rotation, and deformation of the observed subset. This method is often used due to its precision and low computational complexity [2]. As can be seen in Figure 1, the coordinates of the point P (xf, yf) in the distorted image can be obtained as follows: xf = x0 + u + ∂u ∂x (x0 − x0,c) + ∂u ∂y (y0 − y0, c) (1) yf = y0 + v + ∂v ∂x (x0 − x0,c) + ∂v ∂y (y0 − y0, c), (2) where xf and yf represent the end coordinates of the subset, x0 and y0 the initial coordinates of the subset, u and v are the displacement vectors in the direction of the coordinate axes and ∂u/∂x, ∂u/∂y, ∂v/∂x, ∂v/∂y are components of the displacement gradient. DIC can be divided into two groups for 2D and 3D correlation of digital image. 2D DIC is designed to monitor displacements and deformations in the plane. It should be without significant spatial deformations and parallel to the plane of the sensing device (CCD sensor). In contrast, 3D DIC is designed to monitor displacements and deformations in a spatial sample using a pair of cameras and subsequent stereotriangu- lation. The 2D DIC method was used for the work. 1.1. Algorithms The two-dimensional correlation algorithms can be categorized into two main groups: (1.) The first evaluates the matches between the refer- ence and the image after the event in the grayscale intensities of the individual pixels. The minimum or maximum of the correlation criterion is calcu- lated [3]. (2.) The second types of algorithm use fast Fourier transformation (FFT). They can be divided into two groups, cross-correlation in the frequency domain and phase correlation. These calculation algorithms make it possible to track large displacement of the subsets, and unlike the other mentioned algorithms they search for an optimal match between the refer- ence and deformed image in the frequency domain. The Fourier transform F (α) converts the signal from the time domain f(t) to the frequency domain (α). The signal can be discrete or continuous. The transformation can decompose the signal into eas- ily describable sine and cosine waves. The Fourier transform can be expressed as follows: F {f(t)} = F (α) = ∫ ∞ −∞ f(t)e−2πiαt · dt, (3) 123 https://doi.org/10.14311/APP.2023.40.0123 https://creativecommons.org/licenses/by/4.0/ https://www.cvut.cz/en Slávek Zbirovský, Václav Nežerka Acta Polytechnica CTU Proceedings Figure 1. Subset (QR code) deformation and shift. where α is the frequency, t is the time, i = √ −1 and −2πα is the angular frequency. The signal f(t) is usually not continuous, but is measured as a sequence f(x), containing N consecutive samples. This set can be transformed into a field of frequencies using the Discrete Fourier Transform (DFT), which can be defined as follows: D {f(t)} = F (ξ) = N−1∑ x=1 f(x)e−2πi xξ N . (4) For a two-dimensional N × N matrix, N2 operations are needed. Therefore, in 1965, J. W. Cooley and J. W. Tuckey developed the fast Fourier transform FFT method [4]. This can reduce the complexity to N log2 N operations. 1.2. Phase correlation Phase correlation which is one of the algorithms using FFT can be described as follows; let functions f1(x, y) a f2(x, y) be the considered images before and after deformation. Their relationship can be expressed as f2(x, y) = f1(x − x0, y − y0). The relation of the corresponding Fourier transforms F1(ξ, η) and F2(ξ, η) is defined as: F1(ξ, η) = F2(ξ, η)e−2πi(ξx0+ηy0). (5) The normalized cross-power spectrum of images f1(x, y) and f2(x, y) with their Fourier transform F1(ξ, η) and F2(ξ, η) can then be expressed as: Zf1,f2(ξ, η) = F1(ξ, η)F ∗ 2 (ξ, η) |F1(ξ, η)F2(ξ, η)| = e2πi(ξx0+ηy0), (6) where F ∗ 2 (ξ, η) represents the complex conjugate form F2(ξ, η). The inverse Fourier transform of the normal- ized cross-power spectrum gives the discrete impulse function. δf1,f2(x, y) = D−1 {Zf1,f2(ξ, η)} (7) It is zero in all coordinates, except at the coordinates corresponding to the displacement (−x0, −y0) : δ(x,y) = { 1 if (x, y) = (−x0, −y0) 0 otherwise. (8) To obtain the displacement vectors u and v, the max- imum is found at the point [x, y], which is subtracted from the initial coordinates [x0, y0] [5]. 2. Hardware The hardware set was adapted to real-time measure- ments, the design emphasized the low energy consump- tion and mobility of the entire set. The computing unit consists of a single-board computer Raspberry Pi model 4B, which is equipped with the Raspbian oper- ating system. The main advantages of this computer are small size and price/performance ratio. Power is supplied via a USB-C cable with the Thunderbolt 3 standard from the mains, or from a backup source connected to the solar panel. Figure 2 shows two options for recording an image, a DSLR camera and a Picamera. The device’s connectivity to the Internet is enabled using a built-in Wi-Fi module with the 802.11ac specification. 3. DSLR camera The accuracy of displacement and deformation eval- uated by the DIC method depends on the number of pixels in the captured image. For this and other reasons, a DSLR (Digital Single Lens Reflex) cam- era was used in this work for data collection. The basic element of digital SLR cameras is a charge- coupled device (CCD) or complementary metal-oxide- semiconductor (CMOS) sensor that captures image information. These sensors convert captured light into an electrical signals. 124 vol. 40/2023 FFT-based algorithms for DIC measurements Figure 2. Hardware set consisting of a Raspberry Pi 4B computer, a DSLR camera, and a Pi camera. 4. Software The program that records the image and calculates the displacements of the observed subset is written in the Python version 3 programming language. The OpenCV (Open Source Computer Vision) function library is used to locate and read QR codes in the im- age. To work with the Fourier transform, the NumPy function library is used, which defines n-dimensional arrays of numbers [6]. The program flow chart in the online version is described in Figure 3. The program has the following definable input categories: (1.) DIC characteristic settings, (2.) MQTT protocol related settings, (3.) program settings. 4.1. Detect and decode QR code An important factor in DIC is the pattern of the observed surface. It should always be high-contrast, nonrepetitive, and isotropic. All these requirements are met by the QR code, and therefore during the experiment, it served as an orientation element on which the observed subset was placed. The OpenCV function library is used to locate and read the QR code. The DetectAndDecodeMulti function used has as input a captured image and as output data contained in QR codes, bbox = output field of vertices of found quadrangles containing the codes, and rectified image = corrected binarized QR codes [7]. 4.2. Image acquisition Two options for image acquisition were cre- ated in the program. For the first solution, they used a function from the OpenCV package cv2.VideoCapture(cam_number) that controls the attached Pi camera. For measurements where higher accuracy is required, a DSLR camera will be connected to the hardware assembly; for its control, the gPhoto function library is implemented, which is launched externally via the command line, and in the program the function is ensured by the subprocess command. 4.3. Sending data Data is sent via the message queueing telemetry trans- port (MQTT) protocol. The central element of this system is the broker through which all messages are sent between clients. The messages are sent using the subscribe and publish commands. In this work, the open source MQTT broker Eclipse Mosquitto is used [8]. This service is inserted into the Python programming language through the paho.mqtt function package [9]. From it, the publish.single func- tion was used for sending, which has the following optional inputs: mqtt_topic = address to store on the server, mqtt_message = message, qos = quality of service, host_name = server address. The message sent saved into a variable mqtt_message has the following format: speci- men_name; qr_code_numbers; absolute time [s]; rel- ative time [s]; subset shifts (qr code), dx and dy [mm]; image in Base64 code Example of a sent message: Specimen A; [4]; 2022-04-18-16:10:45; 34.1; [(-0.9194, 0.1313)]; /9j/4AAQSkZJRgABAQAAAQABA.... 4.4. Displacement calculation The phase correlation algorithm described above was used to calculate the shift. It was implemented into the program by the skimage.registration function pack- age [10]. The phase_cross_correlation function from the package was used. Its basic inputs are: (1.) subset_ref = reference image, (2.) subset_cur = image after deformation, (3.) upsample_factor = subpixel registration. The resulting shift vector [px] must be transformed into the Cartesian system and converted to millime- ters. 4.5. Subset size and scale of the image To obtain the scale of the image, i.e. the value of the number of pixels per mm, it is necessary to define the size of the QR code as an input. This dimension is then written into the QR code at the third position in the string and inserted into the program as a variable qr_code_width. The variable scale_factor [px/mm] is calculated from the photographed dimensions of the QR code divided by the real size of the QR code, and then this value will be used to convert the calculated displace- ments in pixels into millimeters. Therefore, the QR code serves not only as a random speckle pattern for the DIC method but also as an information carrier and scaling tool. 5. Website interface for real time measurements An online interface was created for a clear display of results when measuring displacements in real time. 125 Slávek Zbirovský, Václav Nežerka Acta Polytechnica CTU Proceedings Figure 3. Flow chart diagram of the program. It is available on the website https://rm.fsv.cvut. cz/. Depending on the settings, it allows to display the images and the time evolution of the displacement. 6. Validation To verify the functionality of the device, a simple mechanism consisting of two sliding wooden plates was constructed. A 50 × 50 mm QR code was pasted onto the upper plate, allowing the localization of the subset. A millimeter ruler was also attached to the assembly to prescribe displacements. A Panasonic LU- MIX DMC G85 DSLR camera equipped with a Penta- con auto 50 mm f/1.8 Multi Coating lens was placed on a tripod at a height of 45 cm above the prepared mechanism. During the measurement, the top plate with the attached QR code was moved horizontally by 2 mm each time. The measurement was performed in six repetitions. The comparison of prescribed and measured values during validation is shown in Fig- ure 4. The evaluation revealed a standard deviation from the prescribed displacements of 0.179 mm. The accuracy and reliability of the FFT algorithms used to calculate the displacement was previously tested in a laboratory measurement [11]. 7. Experimental measurements In order to test the functionality of the designed device, an experimental measurement was carried out in the field. During the Hall of the Year Academic Student Competition, vertical displacement was measured and monitored on the bottom flange of the wooden truss beam model of the hall structure. 126 https://rm.fsv.cvut.cz/ https://rm.fsv.cvut.cz/ vol. 40/2023 FFT-based algorithms for DIC measurements Figure 4. Comparison of prescribed and measured values during validation. Figure 5. Model with QR code for subset registration. During loading, the value of the vertical displace- ment dy of the lower flange of the truss was monitored. A QR code was placed on the side of the lower flange (Figure 5). A Panasonic LUMIX DMC G85 DSLR camera with a Panasonic Lumix G 25mm f/1.7 ASPH lens was used as the recording device. The camera worked in offline mode and pictures were taken at a frequency of 1 picture per second, the evaluation was done retrospectively. Before collapse, the deflec- tion of the bottom flange of the truss reached a value of 17.97 mm. The development of the structure deflec- tion can be seen in Figure 6. 8. Conclusion The presented solution for continuous remote measure- ments of QR code displacements can provide a real- time assessment of structural health. The Python code implemented in a compact Raspberry Pi com- puter enables fully autonomous localization of QR codes and monitoring of their relative shifts. The low energy demands of this solution enable off-grid opera- tion if powered by a battery pack connected to a solar panel. Unit-to-server communication is accomplished using the MQTT protocol. This economical solution Figure 6. Evolution of the beam deflection in time. proved to be accurate and reliable in our preliminary testing; however, a few requirements must be met: a mobile data signal must be available at the measure- ment site (currently designed with a 4G module), the QR codes must be illuminated, and the camera view cannot be obstructed. 127 Slávek Zbirovský, Václav Nežerka Acta Polytechnica CTU Proceedings List of symbols DIC Digital image correlation F F T Fast Fourier transformation DSLR Digital single-lens reflex camera MQT T Message queuing telemetry transport QoS Quality of service Acknowledgements This work was supported by the Czech Technical University in Prague (grant number SGS22/029/OHK1/1T/11). References [1] V. Sebera, J. Tippner. Optické měření deformace metodou korelace digitálního obrazu. Mendelova univerzita, Brno, 2012. [2022-03-14]. https://bit.ly/fraxinus_mendelu_cz_vyuka [2] B. J. Li, Q. B. Wang, D. P. Duan. Displacement measurement errors in digital image correlation due to displacement mapping function. Experimental Techniques 43(4):445–456, 2019. https://doi.org/10.1007/s40799-018-00301-z [3] J. Blaber. DIC Algorithms. Ncor. [2022-04-06]. http://www.ncorr.com/index.php/dic-algorithms [4] J. W. Cooley, J. W. Tukey. An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation 19:297–301, 1965. https: //doi.org/10.1090/S0025-5718-1965-0178586-1 [5] B. S. Reddy, B. N. Chatterji. An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE transactions on image processing 5(8):1266–1271, 1996. https://doi.org/10.1109/83.506761 [6] C. R. Harris, K. J. Millman, S. J. van der Walt, et al. Array programming with NumPy. Nature 585(7825):357–362, 2020. https://doi.org/10.1038/s41586-020-2649-2 [7] OpenCV. cv::qrcodedetector class reference, 2022. [2022-04-16]. https://bit.ly/OpenCV_QRcodes [8] R. A. Light. Mosquitto: server and client implementation of the MQTT protocol. Journal of Open Source Software 2(13):265, 2017. https://doi.org/10.21105/joss.00265 [9] Paho-mqtt 1.6.1 project description, 2021. [2022-04- 17]. https://pypi.org/project/paho-mqtt/#single [10] S. van der Walt, J. L. Schönberger, J. Nunez-Iglesias, et al. scikit-image: image processing in Python. PeerJ 2:e453, 2014. https://doi.org/10.7717/peerj.453 [11] V. Nežerka, P. Havlásek. A lightweight DFT-based approach to the optical measurement of displacements using an open-source Python code. Experimental Techniques 46(3):485–496, 2021. https://doi.org/10.1007/s40799-021-00488-8 128 https://bit.ly/fraxinus_mendelu_cz_vyuka https://doi.org/10.1007/s40799-018-00301-z http://www.ncorr.com/index.php/dic-algorithms https://doi.org/10.1090/S0025-5718-1965-0178586-1 https://doi.org/10.1090/S0025-5718-1965-0178586-1 https://doi.org/10.1109/83.506761 https://doi.org/10.1038/s41586-020-2649-2 https://bit.ly/OpenCV_QRcodes https://doi.org/10.21105/joss.00265 https://pypi.org/project/paho-mqtt/#single https://doi.org/10.7717/peerj.453 https://doi.org/10.1007/s40799-021-00488-8 Acta Polytechnica CTU Proceedings 40:123–128, 2023 1 Introduction 1.1 Algorithms 1.2 Phase correlation 2 Hardware 3 DSLR camera 4 Software 4.1 Detect and decode QR code 4.2 Image acquisition 4.3 Sending data 4.4 Displacement calculation 4.5 Subset size and scale of the image 5 Website interface for real time measurements 6 Validation 7 Experimental measurements 8 Conclusion List of symbols Acknowledgements References