International Journal of Interactive Mobile Technologies – ISSN: 1865-7923 – Vol. 12, no. 7, 2018 Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality Study of the Hand Anatomy Using Real Hand and Augmented Reality https://doi.org/10.3991/ijim.v12i7.9645 Poonpong Boonbrahm(*), Charlee Kaewrat, Prasert Pengkaew, Salin Boonbrahm Walailak University, Nakorn si Thammarat, Thailand poonpong@gmail.com Vincent Meni Grenoble Institute of Technology, Grenoble, France Abstract—Anatomy is considered one of the foundation studies for all of the health science students especially medical and nursing students. Anatomy of the hand is complicated. It composes of bones, nerves, blood veins, muscles, and tendon. Memorising all the details about all those parts is tedious work and need much imagination. With the advances in computer graphics and human- computer interaction techniques, understanding how those body parts move is easy to understand in a visual presentation. Augmented Reality (AR) is the technique that allowed the computer-generated objects to overlay on top of the real world. In this study, we concentrate on studying the bones only. We have selected the Leap Motion, which is the device that can detect the hands and fin- gers, like a tracking device, and marker-based AR technique for displaying the computer generated bones on top of the real hand. Since the Leap Motion de- tects the hands and shows the bone in real time, so when a user moves the hands such as waving, all the 3D virtual bones move to the new position just like the real hand. Besides using this tool as the educational tool to help the students have better learning about anatomy, it can also be used as an assessment tool for anatomy class as well. Results from testing this tool with volunteer students in- dicate that it helps them to understand the hand anatomy better and faster than traditional ways. Keywords— augmented reality, hand anatomy, leap motion, bones 1 Introduction Human anatomy is considered one of the foundation studies for all of the health science students especially nursing and medical students. In the past, cadaver dissec- tion was used in teaching and learning anatomy in every medical school since it helps students learn about human body ranging from skin layer, muscles, and bones that constructed into the human body. Even though learning from cadaver dissections is useful and give a good experience to the students, but there is no clear evidence on the effectiveness of this kind of learning. Besides that, there is some inadequacy to ca- daver specimens, so the students will have some limitation to access the specimens. iJIM ‒ Vol. 12, No. 7, 2018 181 Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality To solve the problems, some schools start using animal specimens as part of the learn- ing experiences in the laboratory, but with the shape and size, it cannot replace the cadavers. Study of the anatomy can also be done using the figures from the textbook. Usually, trying to figure out the 3D image from the 2D image is not difficult if the object is not so complex such as symmetric–shaped or geometric shape object. In the case of the complex object such as human organs, imagination may not fulfill this purpose. During the past decade, there were tools available to help students under- stand human anatomy easily such as video or even computer-generated 3D models of the human organs, but the interaction with those media is still limited. With the ad- vances in technologies especially in computer hardware and computer graphics, new techniques for displaying 3D models is possible such as hologram [1], virtual reality and augmented reality. Holography system is the technology that captured the light scattered from the object and present them as a 3D object. This technique requires complex setup and unique viewing device, so it may not be practical to work within the near future. On the contrary, virtual reality and augmented reality are technologies that are available today, and they can apply to many areas of usage ranging from, advertising, marketing, architecture, construction, entertainment, military, travel, medicine and education [2]. By definition, Virtual Reality (VR) is the computer- generated environment that simulates real-life situation. Augmented Reality (AR), on the other hand, is the situation where the computer-generated object is overlaid atop of the real world. Even though AR is considered a form of VR, but they are quite different. In VR, users immersed in the virtual environment but in AR, the user can experience the virtual objects or situations while still in the real world. As mentioned earlier, AR and VR are different, so applications for AR and VR will cover different aspects. For VR, all applications must be in virtual space, so the users needed the devices that put them in the virtual world. This condition can be done by making the environment around the user to look virtual such as by projecting comput- er-generated image on the wall around the user as in the CAVE (Cave Automatic Virtual Environment) or by covering the user’s eyes to let them see only the virtual world as in the case of using the head-mounted display system. The different between the two is that, in CAVE, the user can look around and see the different view, but in the head-mounted display, the tracker is needed to control the view of the environ- ment when the user turns their heads. For Augmented Reality, the device must display the virtual world and real world in the same scene. Examples of VR applications can be seen in Figure 1 (a) where human anatomy can be displayed, and the user can in- teract with them in CAVE. Figure 1 (b) shows the AR application in which user with the head mounted device in real space can see and interact with the model of human anatomy that appeared in the virtual space. 182 http://www.i-jim.org Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality Fig. 1. Human Anatomy in VR (a) and in AR using head-mounted display (b) 2 Literature review Even though, many research papers indicated that the use of cadaveric dissection (dissection of a dead human body) is mandatory as a fundamental part of medical training, but due to the limitation of the cadaveric specimen, the expensive in opera- tion, time consuming and potentially hazardous, many medical school in some coun- tries such as Australia and the UK, have stopped using cadavers altogether and move to computer-simulated anatomy instead [3]. Some medical schools also try to balance the use of digital technologies with cadaver specimens in anatomy classrooms. Re- search on using Computer-Assisted Learning and computer animation in supporting anatomy learning are available for more than 25 years [4]. With the advance in com- puter graphic technology and the powerful processor, the simulation of human anato- my can replace the cadaveric dissection in many aspects. Besides computer simulations and computer-assisted learning, the new technology such as AR and VR can be used for anatomy teaching and learn as well. In Virtual Reality (VR), students can study the anatomy of the human body by peeling off the components layers by layers starting from the skin to bones, which make them under- stand clearly about the human body. Kažoka. D. et al [5] had investigated the useful- ness and user satisfaction of using the Anatomage Table (Virtual Dissection Table) in teaching and learning of Human Anatomy course at Riga Stradis University, Riga, Latvia, and found that virtual dissection technology seems to have a promising role in future anatomy training. Fairén et al. [6] had set up the VR system that allowed the nursing students to see and interact with the 3D model of human organs. After the session, the evaluation of the performance along with the student satisfaction of the system was investigated. The results confirm that VR is a powerful tool that helps the students to understand the anatomy of human body. For AR, Chien et al. [7] used an interactive AR technology to help medical students to understand and memorize the 3D anatomy structure. Results from the experiment indicated that the system could help medical students to learn the complex anatomy structure better and faster than the traditional way. Research on AR in medical education [8], has also mentioned the work on visualizing human anatomical structure with AR and concluded that the technology is promising because of its strong visualization. Moro et al. [9] found that iJIM ‒ Vol. 12, No. 7, 2018 183 Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality both AR and VR, not only give immersive experiences to the students but also in- creased learner immersion and engagement on the anatomy topics as well. In the last few years, there has been a lot of research related to using AR in teach- ing and learning anatomy of the human body but almost all of them related to the models that are static, just like in cadaver dissections laboratory. In some experi- ments, the student may be able to rotate, enlarge or shrink the model but they cannot experience with the moving part of the body or dynamic model and see how the bones or muscle behaved when they are moving. For the real dynamic movement of the model, Kugelmann et al. [10] used Augmented Reality Magic Mirror to check with 880 first-year medical students to see whether AR can be an additional resource for anatomy course beside cadaver lab. For magic mirror concept, the augment 3D organs or skeletons can be projected on the body of the user at the precise location, so when the body moves, the 3D virtual organs or skeleton will move as well. The evaluation from the students indicated that the system is a beneficial addition to the anatomy class except for the limitation on the visual resolution. In this research, we are attempting to find the alternative way for the students to learn hand anatomy effectively using AR and their own hands. 3 Conceptual framework Fig. 2. The flowchart diagram showing the relationship between the 3 criteria for dynamic AR of hand anatomy To set up the AR experiment on hand anatomy learning that can replace or at least reduce some process in training or learning with the cadaver, three criteria need to be considered. The first criteria are how to capture the image of user’s arm and hand. The second criteria are how to map the virtual image of the bones, muscle, etc. in place of the user’s arm which is in the real world and the third criteria are how to 184 http://www.i-jim.org Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality track and monitor the augmented system when the arm is moving. The three criteria are related to each other as can be seen as a flowchart in Figure 2. 4 Experimental setup The experimental setup for studying hand anatomy requires both hardware and software to fulfil the requirement. For the hardware, besides the computer which re- quired high processing power with extra RAM, the setup required webcam for detect- ing the hand of the user. The webcam is used as a video camera that streams the cap- tured image in real time and sent them to a computer. The specification of the webcam can be varied depending on the requirement of the system, i.e., how far the webcam is apart from the hand, etc. In our experiment, the distance between the cam- era and the hand is around 1 meter, so we select Logitech c920, which is perfect for our purpose. The other hardware device required for this setup is the “Leap Motion.” The Leap Motion is a USB peripheral sensor device that supports hand and finger motions as input but requires no contact. The device consists of two cameras and three infrared LEDs which track infrared light outside the visible light spectrum. The detecting range of the “Leap Motion” is from 0 to around 1 meter from the object. Detail appearance of the Leap Motion is displaying in Figure 3, and the hardware setup for the study of the hand anatomy is showing in Figure 4. For the software, besides the application program for creating the 3D image of the human body and organs which in this case is Maya, we also need Unity 3D and Qual- comm’s Vulforia. Unity 3D is a cross-platform game engine, used for developing 2D and 3D video games and simulations for computers and Vuforia is an Augmented Reality Software Development Kit (SDK) for mobile devices that enables the creation of Augmented Reality applications. Fig. 3. Leap Motion sensor devices iJIM ‒ Vol. 12, No. 7, 2018 185 Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality Fig. 4. Details of the hardware setup 5 Testing the setup To check whether this educational tool performs as it should be, we have set up the system and test it on various conditions and the results are shown in Figure 5-7. Fig- ure 5 shows the visualization of the dynamic process of the bone skeleton of user’s hand. This same technique can also be used for the study of the arm and hand muscle and blood vessel as well, as can be seen in Figure 6. Figure 7 shows the ability to interact with the system and get the results in the form of data which can be used for other teaching and learning. For example in student training, information about the hand anatomy can be displayed if the condition in displaying is fulfilled such as when touching part of the bone. Using this condition, we can create a self-tutorial session for the students, or we can generate the quiz for learning assessment, checking whether the students can remember the name of each bone or not. 186 http://www.i-jim.org Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality Fig. 5. Visualization of the dynamic bone anatomy Fig. 6. Results from the study of hand anatomy for muscle and blood veins Fig. 7. Information about some piece of the hand can be displayed 6 Results and discussion In order to verify that this new setup is useful and can help medical students to learn about hand anatomy, we have arranged the usability evaluation test to check how well users can learn and use a product to achieve their goals. Since the usability iJIM ‒ Vol. 12, No. 7, 2018 187 Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality is a combination of many factors such as ease of learning, ease of understanding, efficiency in use and etc., so we conduct this usability based on these factors. We have selected the health sciences students, i.e., medical students, nursing students and environmental health students as our target group since they have experience in anat- omy learning. Eighteen volunteers participate in the activity which covered, listen to the brief explanation of the purpose, participate in using the system and then finally answering the questionnaires. The Likert scale was used for the closed end questions. A Likert Scale is a type of rating scale used to measure attitudes or opinions of the respondents on a level of agreement. The highest score is five, and the lowest score is one. The number of students in each group that participate in the evaluation is dis- playing in Table 1. Results from the questionnaires for usability and users satisfaction are showing in Table 2. Table 1. Number of the Participants from Various Health Science Programs Students n % Medical Students 3 17 Nursing Students 9 50 Environmental Science Student 6 33 Total 18 100 Table 2. User Satisfaction and Usability of the experiments Factors Score (5) % Ease of Learning 4.61 92.2 Ease of Understanding 4.44 88.8 Ease for Reviewing 4.56 91.2 Textbook and Figure Replacement 4.39 87.8 Efficiency in Use 4.56 91.2 Overall User Satisfaction 4.63 92.6 From the results, it is quite clear that the students participate in the experiment are satisfied with the system (4.63/5). In term of usage, students seem to enjoy how easy to learn the anatomy of their hands (4.61/5) plus they can understand the anatomy of hand in action at every angle in any levels from bones to muscle to skin with the blood vessel and nervous system in between (4.44/5). They also prefer using the sys- tem for reviewing the parts that are not clear to them at any time and as many times as they wanted (4.56/5). Comparing the system with the textbook or other media, the participants indicated that the system could be used as the substitute for those media (4.39/5). For the efficiency of the system, the result indicates that it is suitable for learning hand anatomy (4.56/5). 188 http://www.i-jim.org Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality 7 Conclusion From the results, we can conclude that the system for studying hand anatomy using AR seemed to be the best choice for cadaver laboratory replacement or partial re- placement which can save both time and budget for anatomy learning. Even though the system is useful for studying the anatomy of arm and hand, but anatomy learning must cover the whole body, so this limitation needs to be solved. Since this limitation is due to the ability of Leap Motion in sensing, so a new sensor is required. The best candidate of the sensor that can cover the whole body is Microsoft’s Kinect. Kinect already has skeleton tracking of the whole body, so using them for studying anatomy in action is possible. The only drawback may be the resolution which may not satisfy the users. With the advance in technology both in hardware and software, learning anatomy with AR may soon replace cadaver completely. 8 References [1] Pearson. (2017). Holographic Anatomy Lessons: 'Inventing a Whole New Way of Teach- ing'. Retrieved March 6, 2017 from http://www.pearsonlearningnews.com/holographic- anatomy-lessons-inventing-a-whole-new-way-of-teaching/. [2] Yuen, S. C., Yaoyuneyong, G., & Johnson, E. (2011). Augmented reality: An overview and five directions for AR in education. Journal of Educational Technology Devel-opment and Exchange (JETDE), 4 (1), 119-140. https://doi.org/10.18785/jetde.0401.10 [3] Nichols, H. (2014). The digital age of medicine: cadavers still best choice for learning anatomy. MedicalNewsToday. Retrieved March 4, 2017 from https://www.medicalnewstoday.com/articles/284057.php. [4] Walsh, R.J. & Bohn, R.C. (1990). Computer assisted instructions: a role in teaching human gross anatomy. Medical Education, 24(6), 499-506. https://doi.org/10.1111/j.1365- 2923.1990.tb02665.x [5] Kažoka, D., & Pilmane, M. (2017). Teaching and learning innovation in present and fu- ture of human anatomy course at RSU. Papers on Anthropology, 26(2), 44–52. https://doi.org/10.12697/poa.2017.26.2.05 [6] Fairén, M., Farrés, M., Moyés, J., & Insa, E. (2017). Virtual reality to teach anatomy. Eu- rographics, 51-58. [7] Chien C.H., Chen, C.H., & Jeng, T.S. (2010). An Interactive Augmented Reality System for Learning Anatomy Structure. Proceeding of the International Multiconference of Engi- neers and Computer Scientist, March 17-19, 2010, Hong Kong. [8] Kamphuis, C., Baron, E., Schijven, M., & Christoph, N. (2014). Augmented Reality in Medical Education. Perspect Med Educ, 3(4), 300-311. https://doi.org/10.1007/s40037- 013-0107-7 [9] Moro, C., Štromberga, Z., Raikos, A., & Stirling A. (2017). The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ., 10(6), 549- 559. https://doi.org/10.1002/ase.1696 [10] Kugelmann, D., Stratmann, L., Nuhlen, N., Bork, F., Hoffmann, S., Samarbarksh, G., Pferschy, A., Von der Heide, A.M., Eimannsberger, A., Fallavollita, P., Navab, N., & Waschke, J. (2018). An augmented reality magic mirror as additive teaching device for gross anatomy. Annals of Anatomy, 215, 71-77. https://doi.org/10.1016/j.aanat. 2017.09.011 iJIM ‒ Vol. 12, No. 7, 2018 189 Short Paper—Study of the Hand Anatomy Using Real Hand and Augmented Reality 9 Authors Dr. Poonpong Boonbrahm is an Associate Professor and Dean of School of In- formatics, Walailak University, Thailand. Charlee Kaewrat and Prasert Pengkaew are graduate students in Ph.D. and Master Programs in Management of Information Technology, School of Informatics, Walailak University. Dr. Salin Boonbrahm is a lecturer in Information Technology Department, School of Informatics, Walailak University. Vincent Meni is an exchange graduate student from Grenoble Institute of Tech- nology, Grenoble, France. Article submitted 04 October 2018. Resubmitted 18 October 2018. Final acceptance 27 October 2081. Final version published as submitted by the authors. 190 http://www.i-jim.org