14th International Symposium on Particle Image Velocimetry – ISPIV 2021 August 1–5, 2021 GANs-based PIV resolution enhancement without the need of high-resolution input Alejandro Güemes1, Carlos Sanmiguel Vila1, Stefano Discetti1∗ 1 Universidad Carlos III de Madrid, Aerospace Engineering Research Group, Madrid, Spain ∗ sdiscett@ing.uc3m.es Abstract A data-driven approach to reconstruct high-resolution flow fields is presented. The method is based on ex- ploiting the recent advances of SRGANs (Super-Resolution Generative Adversarial Networks) to enhance the resolution of Particle Image Velocimetry (PIV). The proposed approach exploits the availability of in- complete projections on high-resolution fields using the same set of images processed by standard PIV. Such incomplete projection is made available by sparse particle-based measurements such as super-resolution par- ticle tracking velocimetry. Consequently, in contrast to other works, the method does not need a dual set of low/high-resolution images, and can be applied directly on a single set of raw images for training and estimation. This data-enhanced particle approach is assessed employing two datasets generated from direct numerical simulations: a fluidic pinball and a turbulent channel flow. The results prove that this data-driven method is able to enhance the resolution of PIV measurements even in complex flows without the need of a separate high-resolution experiment for training. 1 Introduction The enhancement of the spatial resolution in Particle Image Velocimetry (PIV) is a long-standing line of research. Many efforts have been directed towards increasing both the Dynamic Spatial Range (DSR) and the Dynamic Velocity range (DVR), defined as the ratio between the largest and smallest measurable length/velocity, respectively (Adrian, 1997). An increase of the DSR can be achieved either by technological advances (enlarging the sensor size) or by algorithms capable of reducing the size of the smallest resolvable scale. Along this line, it has been assumed for long that the mean interparticle spacing was the lowest bound for resolution, unless temporal information is used in the process. For time-resolved PIV, this process has led already to several successful applications in 2D and 3D (see for instance Hain and Kähler 2007). The challenge of achieving higher resolution flow fields is critical in turbulent flows in which the spatial scales have a wide spectrum, being the ratio between the integral scale and the Kolmogorov scale propor- tional to Re−3/4 (Pope, 2000), where Re is the Reynolds number based on the large-scale size and velocity. Due to their presence in a large number of relevant fluid-flow problems, in recent years, applications based on ensemble-correlation (Westerweel et al., 2004; Scharnowski et al., 2012) and ensemble-particle-averaging (Cowen and Monismith, 1997; Kähler et al., 2012; Agüera et al., 2016) approaches have been employed to increase the resolution of flow statistics in standard two-frame applications. Unfortunately, the outcome of these approaches is restricted to obtaining high-resolution turbulence statistics since, when it comes to obtaining instantaneous fields, the resulting outcome is a sparse sampling of the velocity field that does not provide enough information to study the flow features. To overcome these limitations, recently, a model-free fully-data-driven method to enhance the spatial resolution of two-frame PIV fields has been proposed by Cortina-Fernández et al. (2020). The method is based on using sparse Particle-Tracking-Velocimetry (PTV) data to build a high-resolution dictionary of Proper Orthogonal Decomposition (POD) modes. The POD modes are then used to reconstruct velocity fields using a POD temporal basis obtained from low-resolution PIV fields. This method has the main limitation of being based on linear projection on temporal modes obtained from low-resolution PIV fields. Consequently, it cannot resolve non-linear dependencies, especially for what regards features that are not resolved in the original low-resolution PIV fields, which are recurrent in turbulent flows. Convolutional neural networks (CNNs) and generative adversarial networks (GANs) are now offering new opportunities for super-resolution imaging, and have already been used for resolution enhancement of turbulent flow fields (Fukami et al., 2019; Deng et al., 2019; Liu et al., 2020; Fukami et al., 2021). GANs for image super-resolution (SRGANs), in particular, have show promising resolution upsampling also for PIV experiments. While the success of these works is undeniable, they are all limited by the need for a high-resolution target for training, i.e. a set of pairs of fields with low and high resolution are needed, which limits their applicability. It is indeed difficult to obtain from experimental data, and somewhat implies that high-resolution measurements are already available. While it is true that in PIV experiments there is not a pair of low and high resolution fields, we always have available at least an incomplete projection of the low-resolution fields onto an high-resolution one. Indeed, while PIV fields are obtained from cross-correlation of groups of particle images, PTV provides vectors at the level of the single particle, without filtering effects. This corresponds to sampling the field in scattered location, thus providing an incomplete (“gappy”) view of the high-resolution field. The ansatz of the work presented here exploits the capability of GANs to be trained with conditional data to generate high-resolution fields from PIV fields using only this scattered information. Being the great advantage of this work, the ability to achieve high-resolution fields without having a priori any high-resolution field. 2 Methodology 2.1 PIV-tailored GAN Generative adversarial networks are composed of two competing networks, a generator, in charge of pro- ducing an artificial output which mimics reality, and a discriminator, which oversees distinguishing between reality and artificial outputs. For this work, the architecture proposed in Ledig et al. (2017) has been used. However, following the findings of Wang et al. (2019) the batch-normalization layers have been removed to ensure a proper fitting of the deep layers in the architecture. The in-plane PIV velocity components are fed to the generator, which applies a convolutional layer with filter size 9× 9 and 64 feature maps, followed by a parametric ReLU (PReLU) activation function. After these initial layers, 16 residual blocks are applied with the layout proposed by Gross and Wilber (2016), i.e., a convolutional layer followed by a PReLU activation function and a second convolutional layer, with 64 feature maps of filter size 3× 3 for both convolution operations. Before increasing the resolution, a skip- connection sum is performed between the output of the residual blocks and the output of the initialization layers. The subpixel convolution layer proposed by Shi et al. (2016) is used to increase the resolution up to the targeted-PTV one. Finally, a convolution operation with linear activation function and filter size 9×9 is applied to recover the in-plane PTV velocity components as outputs. Both the real and generated PTV fields are fed into the discriminator network. It must be noted that the real fields are sparse, while the entire information is provided by the GAN approach. The initialization of the network is carried out by a convolution operation of filter size 3× 3 and 64 feature maps, followed by Leaky ReLU (LReLU) activation function. After this, 7 discriminator blocks of successive increasing feature maps and are applied, where the discriminator blocks are composed of convolutional layers of 3×3 size and LReLU activation function. Note that odd blocks applied a stride step larger than one to reduce the height and width of the feature maps. Finally, the feature-map tensor is flattened into a vector, followed by a fully-connected layer of 1024 neurons with LReLU activation function and output fully-connected layer with one neuron and sigmoid activation function. This output gives a probability of whether the input is real (0s) or fake (1s). The generator loss is defined with weighted mean-squared error of the predicted output with respect to its target. The target is built from the scattered vector discretized in windows, from now on called “bins”. Binned-PTV distributions are obtained by setting to 0 the velocity in empty bins, and to the average velocity of the vectors within the bin when one or more vectors are founds. The error is weighted to consider only those bins in the target where there are vectors. Moreover, an adversarial contribution is added to the error. This adversarial loss, weighted with a factor of 0.001, is defined as the binary cross-entropy of the predicted fields, i.e., to check if the discriminator has defined as fake the generated fields. The discriminator loss is defined as the mean values between the binary cross-entropy of the predicted fields and the target ones. For the latter, the error checks if the discriminator has recognized those fields as reals. It is worth to mention that to increase the stability of the GAN training all the cross-entropy errors have been perturbed with a random fluctuation of standard deviation 0.2. The different cases have been trained during 100 epochs with 8 samples per batch. The generator and discriminator states have been stored each 5 epochs to find later the optimum point before the GAN starts overfitting. Both networks’ weights have been updated using Adam optimizer with learning rate 0.0001. A schematic view of the architecture is presented in figure 1. XLR YHR ỸHRGenerator (G) Discriminator (D) D(ỸHR) D(YHR) Figure 1: Schematic illustration of GAN architecture (adapted from Kim et al. (2021)). The novel element is the use for training and testing of gappy high-resolution fields provided directly by binned PTV data. 3 Validation The present method has been tested employing two different Direct Numerical Simulation (DNS) dataset that includes a fluidic pinball and a turbulent channel flow. Using these numerical simulations, a combined set of pseudo-PIV and psuedo-PTV fields have been generated and used to train the presented GAN architecture. 3.1 Dataset description The first test case is the fluidic pinball (Deng et al., 2020) which is a two-dimensional wake flow around a cluster of three equidistantly-spaced cylinders with equal radius R = D/2, whose centres form an equilateral triangle with side length equal to 3R. The triangle is oriented with an upstream vertex and with the down- stream side orthogonal to the freestream flow, located at x = 0 and centred with respect to the y axis. The wake of the pinball includes the interaction of the wakes of three bodies, a region of development and the final merging in a large-scale unique shedding wake. The DNS data at Re = 130 (referred as chaotic regime Deng et al. (2020)) are used to generate synthetic PIV images. The details of the simulation settings and flow behaviour can be found in Deng et al. (2020). A set of 4737 synthetic PIV images are generated with a custom-made code. The first 4000 are used for training, while the rest is used for testing. Gaussian blobs with a maximum intensity of 100 counts and 3 pixels diameter are generated on 8-bit images. A domain ranging from −5D to 8.04D in the streamwise direction and from −3.84D to 3.84D in the spanwise direction is discretized with 25pix/D, thus resulting in images with 576× 192 pixels. Particles are randomly distributed to achieve an image density of 0.02 particles per pixel. The images are processed with a custom-made multi-pass image deformation (Scarano, 2001) PIV algorithm developed at the University of Naples (Astarita, 2008). The interrogation window is set to 32×32 pixels with 50% overlap. For this first test case, the SRGAN seeks to achieve upsampling levels ranging from fu = 2−8, with fu being the upsampling factor. The window used to bin the vector fields has size equal to b = DI/ fu, with b,DI being the bin size of the binned PTV and the interrogation window size of the PIV process, respectively. Furthermore, the grid is refined by the same amount, thus maintaining the overlap between adjacent windows. It is important to remark that selecting a smaller bin increases the levels of gappyness of the distributions to be reconstructed. The level of gappyness is approximately 1%,29%,73% for fu = 2,4,8, respectively. The second test case is the turbulent channel flow available at the Johns Hopkins Turbulence Database (http://turbulence.pha.jhu.edu/). The channel has a dimension of 2 half-channel-heights h from wall to wall, 3πh in the span-wise direction and 8πh in the stream-wise direction. The DNS database covers one channel-flow through time 8πh/Ub (where Ub is the channel bulk velocity) with a DNS time step of δt = 0.0014h/Ub. The details of the simulation settings and flow behaviour can be found in Li et al. (2008). For our simulated experiments, subdomains are taken of size 2h× h in the streamwise and wall-normal directions, respectively. Such domains have been discretized with 512pix/h and seeded with particles with a density of 0.01 particles per pixel. The particles have been randomly distributed in the subdomain and tracked for 10 time steps of the simulations to generate the position in the second frame. Snapshots are http://turbulence.pha.jhu.edu/ −4 0 4 y / D −5 0 5 10 15 x/D −4 0 4 y / D −5 0 5 10 15 x/D −5 0 5 10 15 x/D −5 0 5 10 15 x/D −1.5 0.0 1.5 −0.7 0.0 0.7 Figure 2: Contour maps of instantaneous velocities for fluidic pinball case with upsampling factor fu = 8. Top and bottom row denote horizontal and vertical velocity components respectively. From left to right, columns refers to PIV, binned PTV, GAN prediction and DNS reference field. generated with time separation of 1 convective time to reduce correlation between snapshots. In order to obtain a sufficient number of snapshots, data are extracted in subdomains at different locations in the streamwise and spanwise direction. The streamwise and spanwise separation between domains was equal to 2h and 0.25h respectively. A total of 11856 snapshots has been generated, with 10000 of them used for training. The particle images are generated with the same parameters of the previous test case. The interrogation window is set to 64× 64 pixels with 50% overlap. Here a relatively large window is selected on purpose to challenge the GANs in presence of a modulated input with severe lack of information in the near-wall region. As for the case of the pinball, for this test case, upsampling factors up to 8 have been tested. This corresponds to a gappyness percentage of approximately 8%,47% for fu = 4,8 respectively, while for fu = 2 the percentage of gaps is negligible. 3.2 Results For both validation cases, GAN performance is assessed in terms of instantaneous visualizations of velocity fields. Although complete high-resolution DNS data is not used during the training procedure, it is shown for the sake of comparison. Moreover, an instantaneous velocity-based squared error is computed for the GAN predictions with respect to the DNS data. It must be recalled that this error is not the same that has been used during the training, which was referenced with respect to the incomplete high-resolution PTV data. Figure 2 reports an instantaneous flow visualization of the fluidic pinball case with upsampling factor fu = 8. It can be observed that in the near-cylinder region the GAN performance allows to recover most of the small-scale details. For instance, the vertical-velocity details in the front of the cylinders are perfectly recovered, while this information was barely seen in the PIV input field. Similar performance is observed in the cylinder wake, where all velocity patterns are recovered. Figure 3 shows the same instantaneous velocity visualization for the turbulent channel case with upsam- pling factor fu = 8. For this case, GAN improves the small-scale details with respect to the PIV input. It is remarkable the recovery of wall-attached vertical velocity fluctuations in the near-wall region. Nonetheless, the comparison between GAN prediction and DNS reference shows that the smallest-scale details are not perfectly recovered. The instantaneous squared error for the fluidic pinball case is reported in fig. 4. The error is normalized with the bulk displacement (in this case set equal to 1). Together with the error for upsampling factors fu = [2,4,8], the original PIV error is reported with respect to the DNS reference. It is shown that the PIV error is significantly larger than in any of the GAN predictions, thus proving that the proposed methodology is providing a resolution improvement. This error is more significant in the near-cylinder region, where the velocity shear rate is larger, and the small spatial wavelenght of the shear layers released by the cylinders cannot be capture with the resolution of the original PIV analysis. For the GAN, it is observed that a fu = 2 is already sufficient to recover with very high accuracy the fluctuations in the developed wake. However, small scale details in the near wake are still missed. With increasing upsampling, the GAN progressively recovers also such details, without suffering for quality reduction due to the high level of gappyness of the 0 1 y / h 0 1 2 x/h 0 1 y / h 0 1 2 x/h 0 1 2 x/h 0 1 2 x/h 0.4 0.8 1.2 −0.1 0.0 0.1 Figure 3: Contour maps of instantaneous velocities for turbulent channel case with upsampling factor fu = 8. Top and bottom row denote horizontal and vertical velocity components respectively. From left to right, columns refers to PIV, binned PTV, GAN prediction and DNS reference field. −4 0 4 y / D PIV fu = 2 fu = 4 fu = 8 −5 0 5 10 15 x/D −4 0 4 y / D −5 0 5 10 15 x/D −5 0 5 10 15 x/D −5 0 5 10 15 x/D 0.00 0.01 0.00 0.01 Figure 4: Contour maps of instantaneous error, defined as the squared difference between the DNS reference and PIV/GAN prediction, for fluidic pinball case. Top and bottom row denote horizontal and vertical veloc- ity components respectively. From left to right, columns refers to PIV, and predictions with fu = [2,4,8]. binned PTV distributions used for training. For the turbulent channel flow case, the instantaneous squared error is shown in fig. 5. For this case, the larger error values are bounded in the near-wall and logarithmic regions. It is observed a clear error reduction between fu = 4 and fu = 2 with respect to the original PIV fields. This was expectable, due to the artificially large interrogation window used for PIV. However, there is not a clear improvement for case fu = 8 with respect to fu = 4, which confirms the visual inspection of fig. 5. With respect to the difference between wall-normal and streamwise velocity components, it seems that the latter suffers from larger errors. However, this worse performance can be ascribed to the normalization of the data, since the wavelength range present in wall-normal velocities is smaller. In any case, the comparison between the instantaneous error and velocity fields shown in fig. 3 indicates that the error is located in regions with high velocity shear, which points out the interface between high- and low-momentum regions. 4 Conclusions A novel method to enhance the resolution of PIV snapshots based on sparse particle-based measurements has been proposed. The working principle is based on exploiting the ability to retrieve information of the Generative Adversarial networks. Pairs of standard “low-resolution” PIV and high-resolution sparse particle-based snapshots are used for the training process, being not necessary to have a complete high- resolution field as target. In this study, the high-resolution target is generated naturally by binning the scattered PTV vector distribution on a regular mesh, and withdrawing empty bins during the training. The results show that flow features that are not captured in the PIV snapshots can be recovered by the GAN 0 1 y / h PIV fu = 2 fu = 4 fu = 8 0 1 2 x/h 0 1 y / h 0 1 2 x/h 0 1 2 x/h 0 1 2 x/h 0.00 0.01 0.00 0.01 Figure 5: Contour maps of instantaneous error scaled with the channel bulk velocity Ub, defined as the squared difference between the DNS reference and PIV/GAN prediction, for turbulent channel case. Top and bottom row denote horizontal and vertical velocity components respectively. From left to right, columns refers to PIV, and predictions with fu = [2,4,8]. through the incomplete projections provided by the PTV. A reasonably good reconstruction error can be achieved even at an upsampling factor fu = 8, which normally corresponds to large levels of gappyness of the binned distributions, with minimal effect of noise or discontinuities in the reconstructed distributions. The results presented here highlight the promising margin of improvement of the spatial resolution of PIV using machine learning, without the need of additional high-resolution data for training. Acknowledgements This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 949085). This document reflects only the author’s view and the Agency and the Commission are not responsible for any use that may be made of the information it contains. References Adrian R (1997) Dynamic ranges of velocity and spatial resolution of particle image velocimetry. Meas Sci Technol 8:1393 Agüera N, Cafiero G, Astarita T, and Discetti S (2016) Ensemble 3D PTV for high resolution turbulent statistics. Meas Sci Technol 27:124011 Astarita T (2008) Analysis of velocity interpolation schemes for image deformation methods in PIV. Exper- iments in Fluids 45:257–266 Cortina-Fernández J, Sanmiguel Vila C, Ianiro A, and Discetti S (2020) From sparse data to high-resolution fields: ensemble particle modes as a basis for high-resolution flow characterization. Exp Therm Fluid Sci 120:110178 Cowen E and Monismith S (1997) A hybrid digital particle tracking velocimetry technique. Experiments in Fluids 22:199–211 Deng N, Noack BR, Morzyński M, and Pastur LR (2020) Low-order model for successive bifurcations of the fluidic pinball. Journal of Fluid Mechanics 884:A37 Deng Z, He C, Liu Y, and Kim KC (2019) Super-resolution reconstruction of turbulent velocity fields using a generative adversarial network-based artificial intelligence framework. Phys Fluids 31:125111 Fukami K, Fukagata K, and Taira K (2019) Super-resolution reconstruction of turbulent flows with machine learning. J Fluid Mech 870:106–120 Fukami K, Fukagata K, and Taira K (2021) Machine-learning-based spatio-temporal super resolution recon- struction of turbulent flows. J Fluid Mech 909 Gross S and Wilber M (2016) Training and investigating residual nets. Facebook AI Research 6:3 Hain R and Kähler C (2007) Fundamentals of multiframe particle image velocimetry (PIV). Exp Fluids 42:575–587 Kähler CJ, Scharnowski S, and Cierpka C (2012) On the resolution limit of digital particle image velocime- try. Experiments in Fluids 52:1629–1639 Kim H, Kim J, Won S, and Lee C (2021) Unsupervised deep learning for super-resolution reconstruction of turbulence. Journal of Fluid Mechanics 910 Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al. (2017) Photo-realistic single image super-resolution using a generative adversarial network. in Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4681–4690 Li Y, Perlman E, Wan M, Yang Y, Meneveau C, Burns R, Chen S, Szalay A, and Eyink G (2008) A pub- lic turbulence database cluster and applications to study lagrangian evolution of velocity increments in turbulence. Journal of Turbulence 9:N31 Liu B, Tang J, Huang H, and Lu XY (2020) Deep learning methods for super-resolution reconstruction of turbulent flows. Phys Fluids 32:025105 Pope S (2000) Turbulent flows. Cambridge University Press Scarano F (2001) Iterative image deformation methods in PIV. Measurement science and technology 13:R1 Scharnowski S, Hain R, and Kähler CJ (2012) Reynolds stress estimation up to single-pixel resolution using PIV-measurements. Experiments in Fluids 52:985–1002 Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, and Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. in Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1874–1883 Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, and Loy CC (2019) ESRGAN: Enhanced Super- Resolution Generative Adversarial Networks. in L Leal-Taixé and S Roth, editors, Computer Vision – ECCV 2018 Workshops. pages 63–79. Springer International Publishing, Cham Westerweel J, Geelhoed P, and Lindken R (2004) Single-pixel resolution ensemble correlation for micro-PIV applications. Exp Fluids 37:375–384 Introduction Methodology PIV-tailored GAN Validation Dataset description Results Conclusions