14th International Symposium on Particle Image Velocimetry – ISPIV 2021 August 1–5, 2021 On the determination of 3D position and orientation of spheroidal particles using defocusing and deep learning M. Rossi1∗ 1 Technical University of Denmark, Department of Physics, DTU Physics Building 309, DK-2800 Kongens Lyngby, Denmark ∗ rossi@fysik.dtu.dk Tracking the 3D position of tracer particles or small objects like cells or unicellular organisms in minia- turized lab-on-a-chip or biomedical devices is complicated since it is often not possible in these setups to use multi-camera approaches. Most successful single-camera approaches for these applications are based on holography or defocusing. Holographic methods have been used to track complex objects such has bac- teria (Bianchi et al. (2019)) and even to estimate their orientation (Wang et al. (2016)). However, these methods require a complex and expensive experimental setup which is not always available in research laboratories. On the other hand, defocusing methods work with conventional microscopic optics, are easy to implement, and have shown excellent results in 3D PTV experiments (Qiu et al. (2019)). One main drawback is that they normally work only with spherical and mono-dispersed tracer particles. A defocus- ing method that has potential to measure non-spherical particles is the General Defocusing Particle Tracking (Barnkob and Rossi (2020)) which is based on pattern recognition and can be conceptually extended to more complex tasks by extending the reference library of particle images, including not only spherical particles at different depth positions, but also non-spherical particles at different orientations. However, whether this approach could work in practice is still unknown. First, is the information contained in simple defocused images sufficient to reconstruct depth and orientation of non-spherical particles, and eventually under which circumstances? Second, how to practically collect the labelled reference images? In this work we address the first question using synthetic images of defocused, non-spherical parti- cles generated by the synthetic image generator MicroSIG (Rossi (2020)), based on ray-tracing (Figure 1). Specifically, we consider spheroidal fluorescent particles (prolates or oblates), randomly placed at different depth positions (z) and orientations. Due to rotational symmetry, the orientation is here fully determined only by two Euler angles, α and β. The in-plane position (x,y) can be obtained with conventional segmen- tation procedures and is not analyzed here. Therefore, the problem reduces to the determination of z, α, and β for a set of single particle images. For pattern recognition, we use a ResNet-50 convolutional neural network (He et al. (2016)), a well-known architecture used for complex image recognition tasks, adapted to address a regression problem (i.e. we have here three continuous outputs z, α, and β). The neural network is programmed in the Python language using the Keras/TensorFlow platform. Figure 1: Working principle used in MicroSIG to generate synthetic images of spheroidal particles. Black dots on the surface of the spheroid represent uniformly-distributed point-sources of light. A ray-tracing approach is used to reconstruct the image on the sensor. Due to the rotation symmetry of the spheroid, the orientation is fully determined by two Euler angles α and β. Figure adapted from Rossi (2020). zx y zx y zx y zx y zx y zx y 0.0 0.5 1.0 z/h 0.0 0.5 1.0 z′ /h 0 2 0 1 2 3 ′ 0 1 0.0 0.5 1.0 1.5 ′ (a) (b) Figure 2: (a) Synthetic images of a prolate spheroid with different positions and orientations. (b) Mea- sured versus true values of particles’ depth position, z, and Euler angles α and β. The average normalized uncertainty for the three output variables is 1.5% for z, 3.5% for α, and 1.5% for β. Results for the case of prolate spheroids with an equatorial radius a = 2 µm and a polar radius c = 8 µm, simulated assuming a 20× magnification lens are shown in Figure 2. The particles are randomly ori- ented with 0 ≤ α ≤ 2π and 0 ≤ β ≤ π and randomly placed along a total depth h = 40 µm. The ResNet-50 is trained on 5000 labelled images for 110 epochs with a batch size of 64 and Adam as optimizer. After training, the neural network is tested on 1000 new images giving an average normalized uncertainty for the output variables of σz/h = 0.015, σα/2π = 0.035, and σβ/π = 0.015. Further results obtained on different shapes (prolates and oblates spheroids) and different simulated optics will be presented and discussed in the presentation. In conclusion, this work provides a first proof-of-principle of this method on synthetic images and opens up possible applications in fields such as swimming of micro-organisms, or non-spherical colloids. On-going research is planning to apply this method to study the motion of the micro-organism Euplotes vannus and preliminary results will be presented in the conference. The author acknowledges financial support by the Villum Foundation under the Grant No. 00022951. References Barnkob R and Rossi M (2020) General defocusing particle tracking: fundamentals and uncertainty assess- ment. Experiments in Fluids 61:1–14 Bianchi S, Saglimbeni F, Frangipane G, Dell’Arciprete D, and Di Leonardo R (2019) 3D dynamics of bacteria wall entrapment at a water–air interface. Soft matter 15:3397–3406 He K, Zhang X, Ren S, and Sun J (2016) Deep residual learning for image recognition. in Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770–778 Qiu W, Karlsen JT, Bruus H, and Augustsson P (2019) Experimental characterization of acoustic streaming in gradients of density and compressibility. Physical Review Applied 11:024018 Rossi M (2020) Synthetic image generator for defocusing and astigmatic PIV/PTV. Measurement Science and Technology 31:017003 Wang A, Garmann RF, and Manoharan VN (2016) Tracking E. coli runs and tumbles with scattering solu- tions and digital holographic microscopy. Optics express 24:23719–23725