Acta Polytechnica CTU Proceedings https://doi.org/10.14311/APP.2022.39.0001 Acta Polytechnica CTU Proceedings 39:1–5, 2022 © 2022 The Author(s). Licensed under a CC-BY 4.0 licence Published by the Czech Technical University in Prague AUGMENTED REALITY TO SUPPORT PARCEL HANDLING IN LAST-MILE LOGISTICS Dragos Datcua,∗, Jelmer Winkelb, Leon Rothkrantzc a InteliWatch, César Franckstraat 103, 2625 CB Delft, Netherlands b University of Groningen, 9712 CP Groningen, Netherlands c Delft University of Technology, Mekelweg 5, 2628 CD Delft, Netherlands ∗ corresponding author: email@dragosdatcu.eu Abstract. We have developed an augmented reality (AR) based system which keeps track of events during the parcel handling process for last-mile logistics. The system can retrieve and highlight the location of parcels in large piles with AR on the user’s smartphone. A camera array automatically detects the parcels laid down manually by an operator. New parcels are scanned and parcel fingerprints are generated semi-automatically. The system can detect and track the known parcels by fingerprint and can further highlight the location of the parcel using 3D visual clues, directly on the smartphone of the operator. Keywords: Augmented reality, parcel tracking, logistics. 1. Introduction The rapid growth in e-commerce challenges the last- mile logistics and further demands for more effective technical solutions and procedures [1]. Last-mile lo- gistics concerns the transport of goods from the final warehouse to the customers’ home and can be sepa- rated into three main phases: (1) loading the truck at the warehouse, (2) transport from the warehouse to the delivery area, (3) the final delivery to the cus- tomers [2, 3]. Augmented reality is expected to be one of the enabling key technologies in the age of industrial digi- talization [4]. Recent studies show warehouse opera- tions can benefit from augmented reality technology through increasing the efficiency [5, 6] and reducing the costs [7]. Several technical challenges reported in the litera- ture [7, 8] limit the adoption of augmented reality at the industrial level. So far, little work has been done to investigate the barriers of integrating augmented reality into logistics processes adapted to last-mile delivery. This paper presents a technical solution which in- tegrates augmented reality to support indoor as well as outdoor logistics processes for last-mile delivery of parcels. In order to show the field delivery agent the correct location of the parcel, it is important that the sensing system works successfully. The camera sys- tem tracks the parcel’s precise location and presents this in augmented reality on the user’s smartphone. The visual tracking uses parcel fingerprints with the characteristics of parcels, such as the dimensions, the shape and the texture or the bar-codes. We have implemented the AR system and further conducted a small-scale targeted experiment for testing the func- tional system prototype. This is the first approach to couple a camera sensor array which detects parcels in storage racks with a visualization in augmented reality on smartphones, to help localize parcels for parcel picking and delivery. The technical solution is called SOLiD and means Self Organized Logistics in Distribution. The research project represents the joint efforts of PrimeVision BV, DPD and the University of Groningen in The Netherlands. The next section presents relevant research projects and related work on the automatic input sensing al- gorithms, tracking parcels and displaying information for building situational awareness in operational envi- ronments. Section 3 discusses the aspects related to the ar- chitecture of the parcel tracking system. Section 4 presents and discusses the results of the experiments with the first functional system prototype applied in a scaled-down version of the first operational scenario. The last section addresses the conclusions of our re- search and pinpoints aspects regarding the future work. 2. Related work The topic of augmented reality has been explored and found to be an interesting technology with the potential to improve the efficiency in logistics opera- tions [3, 9, 10]. The use of situation-specific informa- tion technology generally saves a significant amount of time due to the fact that less paperwork or manual checks have to be performed [2]. Augmented reality technologies are usually offered in the form of apps for mobile phones, because these hand-held projectors are the best practice for introducing augmented reality to the market due to the low production costs and the ease of use [10, 11]. 1 https://doi.org/10.14311/APP.2022.39.0001 https://creativecommons.org/licenses/by/4.0/ https://www.cvut.cz/en D. Datcu, J. Winkel, L. Rothkrantz Acta Polytechnica CTU Proceedings Figure 1. System architecture. Recent laboratory experiments have shown that augmented reality can boost efficiency and accuracy for order picking in warehouse operations [5]. Aug- mented reality graphic-based user interfaces can re- duce job completion time and number of errors by 13 % and 59 % respectively, as concluded from a simulated warehouse job involving order picking and part as- sembly [6]. Also, earlier work reported on the benefit of augmented reality over the traditional routines for warehouse operations [12]. In recent years, several system designs and archi- tectures have been created to support the use of aug- mented reality. Essential elements in those archi- tectures are a smart device (smart glasses or mobile phone), a projecting application system, camera vision, internet connection, privacy gates, command controls, modeling tools, back-end services and databases [13]. This model has been used to develop the system design. Over the last years, a large number of open-source aug- mented reality frameworks (e.g. ARCore, ARkit, VR Juggle, Unity 3D) have been developed with tool kits that can be used easily by computer engineers [14, 15]. These tool-kits provide the computer engineers the tools to develop 2D or 3D content in an augmented form (e.g. arrow, text, cross etc.). The aforementioned frameworks help the computer engineers create the front-end part of the system, the part that is visible for the users and where the interaction with the system takes place. There are also scholars that doubt the actual bene- fits that augmented reality can bring to the logistics operations because of multiple challenges during the implementation. Rejeb [8] conducted an extensive literature review of 43 papers from academic jour- nals on the challenges of augmented reality in logistics and 74 % of these papers mentioned that technical chal- lenges are an obstacle for implementing augmented re- ality systems. One key challenge is object recognition, because the digital world has to be in harmony with the real world in order to achieve high accuracy [16]. Robustness of the system is continually improving de- spite computational costs are still high, but real-time tracking for pure vision-based approaches looks very promising [10]. Furthermore, software challenges follow no stan- dardization of the programming language and this makes it difficult for practitioners to experiment and create the correct algorithm [17]. Next to the integration of augmented reality as a means to enhance the visualization, the integration of various sensors [18] and vision models [19, 20] has been the focus in order to recognize, validate, and optimize the activities of workers in logistics and for warehouse automation. In our research, we have integrated and built upon previous findings regarding input sensing models for activity recognition with augmented reality oriented user interface for supporting last-mile logistics opera- tors and delivery agents. 3. System architecture The system architecture consists of a front end and back end (Figure 1). The user interacts with the front- end part of the system, where the operator loads the truck at the distribution center. The system is using cameras that are installed inside the truck, for registering and tracking the parcels inside the truck by using camera vision. The cameras provide video streams which extract knowledge about the layout of the parcel after loading and for rendering the AR layer. The algorithms for the detection and tracking of parcels are replicated on each camera of the camera array. The back end fuses together all the camera observations at the high semantic level so as to build global situational awareness on the parcels at the storage rack. When the field delivery agent arrives at the cus- tomer, he/she runs the augmented reality application and sees a pop-up screen with the details about the customer and the parcel. The 3D user interface displays the animated visual clue in the form of an animated arrow pointing to the parcel to be handled next by the field delivery agent. The 2D user interface displays information about the parcel to be handled. Moreover, the interface includes buttons to update the handling status of the parcel (see Figure 2). 2 vol. 39/2022 Augmented Reality to Support Parcel Handling in Last-mile Logistics Figure 2. Highlighting in AR the location of the parcel to be delivered next. The detection of 2D markers from one or multiple cameras is used to reconstruct the 3D geometry of the physical shelves and storage rack. Following this, the camera parameters are computed. A second cal- ibration relates to a manual alignment and is done through the mobile application by adjusting the 3D position of specific key-points of the shelf. This step is necessary for fine tuning the 3D rendering. The pattern match algorithm has a multi-threaded imple- mentation and uses scale-invariant feature transform (SIFT) descriptors for visual pattern matching. The bar-code module relies on ZXing [21], an open-source, multi-format 1D/2D bar-code image processing library. In scan mode, the sensor component uses all infor- mation about the detected parcels to generate parcel fingerprints. The segmentation of the side images of a parcel uses the algorithm of Wang and Guo [22] which reconstructs the RGB-D indoor scene with plane prim- itives. The algorithm estimates the entire scene by adaptive planes without losing geometry details and preserves sharp features in the final mesh. In addition to the sensor component, the SOLiD system consists of an augmented reality-based visualization compo- nent. This component is a mobile application running on the smartphone of the field delivery agent (Android device). Figure 3. Results of the lab experiment. 4. Results PrimeVision and the University of Groningen in The Netherlands have run a first experiment on a scaled- down version of the first scenario with the fixed storage rack at the premises of the parcel delivery company. Figure 2 illustrates a part of the experimental setup together with the AR layer highlighting the position of the parcel to be delivered next. The experiment has shown that the combined ac- curacy of parcel detection and the precision of 3D parcel presentation in augmented reality for a small- scale scenario with the delivery agent and with tens of parcels in the parcel database, is 95 % (Figure 3). A limitation of the current system is the parcel detection by image recognition. The keypoint-based detector fails when a parcel has a few or no key points at all. On the other hand, the key point-based approach for parcel detection and tracking proved to be robust against variation in scale, illumination, rotation or oc- clusion. Two different tracking algorithms were tested in the lab test, namely state-based tracking with one single key frame and state-based tracking with multi- ple key frames. Our findings showed that when the augmented reality system used a multiple key-frame state-based algorithm was more robust compared to a single key-frame state-based algorithm because it was able to track 95 % of the parcels compared to 80 % respectively. Increasing the number of key frames per state has a positive impact on the performance of the system. The factors relate to lowered illumination and the associated noise sensed in the video frames. One of the drawbacks of a single state-based track- ing algorithm is that it only has one opportunity to find a parcel in contrast to a multi-frame state-based tracking algorithm. The camera vision is subjected to the dynamic en- vironment such as shadows, incidence of light or tur- bulence, and this creates noise for the algorithm to track the parcel. The system must be robust since it has to operate in a dynamic environment due to the fact that the cameras are installed inside a truck. Another reason for the lower success rate, was that parcels were only partially in the active Field Of View (FOV) of the camera. This made it more difficult for the algorithm to recognize the unique fingerprints of a parcel. Because the system only uses one key frame it sometimes was not able to find the parcel when it was partially in the active FOV. However, the higher 3 D. Datcu, J. Winkel, L. Rothkrantz Acta Polytechnica CTU Proceedings success rate was achieved through a higher number of key frames that the tracking algorithm can use to find the parcels and thus increases the chances to find a parcel. The multiple key-frame state-based algorithm stored ten key frames every time a parcel was loaded into the truck. This increased the chances of finding the parcels by tenfold. The results of the preliminary study can be found in Figure 3. 5. Conclusions and future work The first results from a small-scale targeted experi- ment of PrimeVision and the University of Groningen in The Netherlands has shown very promising results. The research has been carried out on a scaled-down version of the storage rack to comply with the new imposed governmental Covid-19 oriented regulations forbidding access to the research facilities. Our exper- iments have revealed that the SOLiD prototype can automatically determine the location of parcels during the parcel handling process in last-mile logistics. A follow-up step of the current research project has already started by testing the one version of the system in a real-life production environment. We want to see the extent at which the system can support the parcel-handling activities in the storage location of a local store which has dedicated service for parcel handling. Acknowledgements This work is done as part of project SOLiD (Self- Organising Logistics in Distribution) and is supported via a national NWO grant with Grant Number 439.17.551. The project was facilitated by TNO Den Haag that enabled the collaboration between the different project partners. References [1] W. Wang, F. Wang, W. Song, S. Su. Application of augmented reality (AR) technologies in inhouse logistics. E3S Web of Conferences 145:02018, 2020. https://doi.org/10.1051/e3sconf/202014502018 [2] R. Gevaers, E. Van de Voorde, T. Vanelslander, et al. Characteristics of innovations in last-mile logistics-using best practices, case studies and making the link with green and sustainable logistics. Association for European Transport and contributors 1:21, 2009. [3] G. Holger, J. Kai, M. Johannes, T. Björn. Augmented reality in logistics: Changing the way we see logistics – a DHL perspective, 2014. [2023-01-12], https://discover. dhl.com/content/dam/dhl/downloads/interim/full/ dhl-csi-augmented-reality-report.pdf. [4] T. Masood, J. Egger. Adopting augmented reality in the age of industrial digitalisation. Computers in Industry 115:103112, 2020. https://doi.org/10.1016/j.compind.2019.07.002 [5] A. Gialos, V. Zeimpekis. Testing vision picking technology in warehouse operations: Evidence from laboratory experiments. International Journal of Industrial Engineering and Management 11(1):1–12, 2020. https://doi.org/10.24867/IJIEM-2020-1-249 [6] S. Kim, M. A. Nussbaum, J. L. Gabbard. Influences of augmented reality head-worn display type and user interface design on performance and usability in simulated warehouse order picking. Applied Ergonomics 74:186–193, 2019. https://doi.org/10.1016/j.apergo.2018.08.026 [7] N. Ilanković, D. Živanić, A. Zelić. Augmented reality in order-picking processes – advantages and disadvantages. Lim Folyóirat 5(1):4–12, 2020. https://doi.org/10.29177/LIM.2020.1.4 [8] A. Rejeb. The challenges of augmented reality in logistics: A systematic literature review. WSN 134(2):281–311, 2019. https://ssrn.com/abstract=3446102. [9] R. Reif, W. Günthner, B. Schwerdtfeger, G. Klinker. Evaluation of an augmented reality supported picking system under practical conditions. Computer Graphics Forum 29(1):2–12, 2009. https: //doi.org/10.1111/j.1467-8659.2009.01538.x [10] R. van Krevelen, R. Poelman. A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality 9(2):1–20, 2010. https://doi.org/10.20870/IJVR.2010.9.2.2767 [11] L. Ranieri, S. Digiesi, B. Silvestri, M. Roccotelli. A review of last mile logistics innovations in an externalities cost reduction vision. Sustainability 10(3):782, 2018. https://doi.org/10.3390/su10030782 [12] H. Baumann, T. Starner, H. Iben, et al. Evaluation of graphical user-interfaces for order picking using head-mounted displays. In Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI ’11, p. 377–384. Association for Computing Machinery, New York, NY, USA, 2011. https://doi.org/10.1145/2070481.2070553 [13] B. Zobel, L. Berkemeier, S. Werning, et al. Towards a modular reference architecture for smart glasses-based systems in the logistics domain 1 smart glasses for logistics services functions. In 9th International Workshop on Enterprise Modeling and Information Systems Architectures (EMISA), pp. 95–99. 2018. [14] G. Lampropoulos, E. Keramopoulos, K. Diamantaras. Enhancing the functionality of augmented reality using deep learning, semantic web and knowledge graphs: A review. Visual Informatics 4(1):32–42, 2020. https://doi.org/10.1016/j.visinf.2020.01.001 [15] M. Sisto, N. Wenk, N. Ouerhani, S. Gobron. A study of transitional virtual environments. In Augmented Reality, Virtual Reality, and Computer Graphics, pp. 35–49. Springer International Publishing, 2017. https://doi.org/10.1007/978-3-319-60922-5_3 [16] S. Uma. Latest research trends and challenges of computational intelligence using artificial intelligence and augmented reality. In Computational Intelligence and Sustainable Systems: Intelligence and Sustainable Computing, pp. 43–59. Springer International Publishing, 2019. https://doi.org/10.1007/978-3-030-02674-5_3 [17] M.-H. Stoltz, V. Giannikas, D. McFarlane, et al. Augmented reality in warehouse operations: Opportunities and barriers. IFAC-PapersOnLine 50(1):12979–12984, 2017. 20th IFAC World Congress. https://doi.org/10.1016/j.ifacol.2017.08.1807 4 https://doi.org/10.1051/e3sconf/202014502018 https://discover.dhl.com/content/dam/dhl/downloads/interim/full/dhl-csi-augmented-reality-report.pdf https://discover.dhl.com/content/dam/dhl/downloads/interim/full/dhl-csi-augmented-reality-report.pdf https://discover.dhl.com/content/dam/dhl/downloads/interim/full/dhl-csi-augmented-reality-report.pdf https://doi.org/10.1016/j.compind.2019.07.002 https://doi.org/10.24867/IJIEM-2020-1-249 https://doi.org/10.1016/j.apergo.2018.08.026 https://doi.org/10.29177/LIM.2020.1.4 https://ssrn.com/abstract=3446102 https://doi.org/10.1111/j.1467-8659.2009.01538.x https://doi.org/10.1111/j.1467-8659.2009.01538.x https://doi.org/10.20870/IJVR.2010.9.2.2767 https://doi.org/10.3390/su10030782 https://doi.org/10.1145/2070481.2070553 https://doi.org/10.1016/j.visinf.2020.01.001 https://doi.org/10.1007/978-3-319-60922-5_3 https://doi.org/10.1007/978-3-030-02674-5_3 https://doi.org/10.1016/j.ifacol.2017.08.1807 vol. 39/2022 Augmented Reality to Support Parcel Handling in Last-mile Logistics [18] A. Diete, L. Weiland, T. Sztyler, H. Stuckenschmidt. Exploring a multi-sensor picking process in the future warehouse. UbiComp ’16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct pp. 1755–1758, 2016. https://doi.org/10.1145/2968219.2968270 [19] R. Yoneyama, A. J. Duran, A. P. del Pobil. Integrating sensor models in deep learning boosts performance: Application to monocular depth estimation in warehouse automation. Sensors 21(4):1437, 2021. https://doi.org/10.3390/s21041437 [20] X. Liu, J. Cao, Y. Yang, S. Jiang. CPS-based smart warehouse for industry 4.0: A survey of the underlying technologies. Computers 7(1):13, 2018. https://doi.org/10.3390/computers7010013 [21] ZXing. ZXing. [2023-01-12], https://github.com/zxing/zxing. [22] C. Wang, X. Guo. Plane-based optimization of geometry and texture for rgb-d reconstruction of indoor scenes. 2018 International Conference on 3D Vision (3DV) pp. 533–541, 2018. https://doi.org/10.1109/3dv.2018.00067 5 https://doi.org/10.1145/2968219.2968270 https://doi.org/10.3390/s21041437 https://doi.org/10.3390/computers7010013 https://github.com/zxing/zxing https://doi.org/10.1109/3dv.2018.00067 Acta Polytechnica CTU Proceedings 39:1–5, 2022 1 Introduction 2 Related work 3 System architecture 4 Results 5 Conclusions and future work Acknowledgements References