Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6, 3782-3795 2024 Publisher: Learning Gate DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate © 2024 by the authors; licensee Learning Gate * Correspondence: rjhkwon@koreatech.ac.kr Object manipulation using haptic feedback based six hand areas in virtual reality Chulwoo Ha1, Sanghun Nam2, Joungheum Kwon3* 1Department of Culture and Technology Convergence, Changwon National University, Changwon, Korea; hachulwoo5@gs.cwnu.ac.kr (C.H.). 2Department of Culture Technology, Changwon National University, Changwon, Korea; sanghunnam@changwon.ac.kr (S.N.). 3Department of Future Technology, Korea University of Technology and Education, Cheonan, Korea; rjhkwon@koreatech.ac.kr (J.K.). Abstract: Human hands are capable of performing various tasks through precise finger movements. If realistic finger movements are implemented in a virtual reality (VR) environment with the addition of haptic feedback, more precise interaction with virtual objects would become possible, leading to enhanced user immersion. This study proposes an object manipulation technique in a VR environment that utilizes five fingers along with haptic feedback. In particular, the technique allows users to manipulate objects using not only the fingertips but also the palms and finger phalanges. During manipulation, force and vibrotactile feedback are provided to ensure that the hand and objects do not penetrate each other, while allowing users to intuitively perceive the stiffness of objects. The study involved three tasks—stacking cubes, matching shapes, and throwing balls—which were used for both quantitative and qualitative evaluation. Results showed that users found the physical interactions with objects to be realistic and reported high levels of immersion. Although wearable devices imposed some physical strain and slightly increased the difficulty of object manipulation, mental load, stress, and distractions were relatively minimal. While the impact of combining free finger movements with haptic feedback on task performance has yet to be fully confirmed, the positive effects on user perception of realism and immersion were evident. The findings of this study could contribute to the development of advanced technologies that improve task performance and reduce fatigue in virtual environments. Keywords: Hand interaction, Haptic feedback, Object manipulation, Presence, Virtual reality. 1. Introduction Humans lead more convenient lives because they can move freely and use their hands to perform complex tasks. For example, when manufacturing or repairing a car that enhances mobility, it is necessary to assemble screws and small components. In this process, it is important to precisely position the screws using fingers and to apply the appropriate force with a screwdriver. If tasks requiring precise hand movements are simulated in VR content, users can train repeatedly without burden and effectively enhance their skill proficiency. This would allow them to adapt easily to how to move their hands and how much force to apply in real tasks. Recently, various studies have been conducted on hand interactions that influence users' sense of realism and immersion in VR. For example, if users can manipulate an object freely using all five fingers, it can enhance their sense of realism. Additionally, if haptic feedback (such as force or vibration feedback) is delivered to the hand to allow users to intuitively feel the object's physical properties, it would also enhance immersion. When manipulating objects, humans use not only the tips of their fingers but also the palms, the sides of the fingers, and the finger phalanges. The sides of the fingers help to grip thin objects more stably, while the palms and the finger phalanges assist in gripping objects more securely and lifting heavier objects. However, in VR the tips of the fingers are primarily used for manipulating objects. If the palms, the sides of the fingers, and the finger phalanges could also be 3783 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate actively utilized in object manipulation, the sense of realism would be significantly enhanced. Furthermore, by utilizing physics engines and constraints in VR, it is possible to visually prevent the fingers from penetrating objects upon contact. However, since real fingers can bend further if physical constraints are not applied, discrepancies between virtual and real finger movements may arise, which can cause users to feel a sense of dissonance. By directly delivering force feedback to the user's hand, physical forces can prevent the fingers from advancing further. Furthermore, generating vibration feedback signals that the fingers are penetrating an object, prompting the user to stop, can reduce the sense of dissonance. To express the stiffness of an object in VR, vibration is transmitted to the user's hand. For rigid objects, strong vibrations are delivered, while for soft objects, weaker vibrations are transmitted, allowing the user to perceive the stiffness of the object based on the intensity of the vibrations. Force feedback can also be effectively utilized to represent stiffness. Rigid objects can provide maximum resistance upon contact, while soft objects can gradually increase resistance as the fingers penetrate further, allowing users to intuitively sense the object's stiffness. In VR, simulating non-penetration between the hand and objects, along with expressing the stiffness of objects, mimics real-world physical phenomena and allows users to indirectly perceive the characteristics of the objects. This enables more realistic hand-based interactions and enhances user immersion in the process. In this study, the SenseGlove DK1, a wearable exoskeleton device, was used to enable users to freely utilize all five fingers when manipulating objects. Additionally, haptic feedback was delivered to the user's hand to enhance the sense of realism and immersion in object manipulation tasks. To implement physically accurate interactions, a physical model of the virtual hand was designed, incorporating the palm, the sides of the fingers, and the finger phalanges. Additionally, a module was developed to enable the manipulation of objects using various grabbing techniques. The object manipulation module provides force and vibration feedback to the user during the manipulation process, representing both the non-penetration between the hand and the object and the object's stiffness. Three tasks (stacking cubes, fitting shapes, and throwing balls) were tested. Task completion times were measured for quantitative assessment, and surveys on immersion and task load were conducted for qualitative assessment. 2. Related Work In VR, manipulating objects with hand movements influences the user's sense of realism. If a virtual environment is designed to allow users to utilize all five fingers for object manipulation in a manner similar to the real world, users may experience a sense of ownership, perceiving the virtual hand as if it were their real hand. If users can actively control their fingers, it enhances the sense of ownership. The stronger this sense of ownership, the more realistic the user perceives the experience in VR [1],[2]. The more realistic the experience, the greater the user's immersion in the virtual world, leading to higher satisfaction and more positive reactions [3]. Implementing detailed finger movements to enable complex and precise object manipulation can further enhance the user's sense of realism [4]. Providing accurate haptic feedback during the manipulation of objects can enhance the user's sense of immersion. Haptic feedback can offer tactile experiences similar to those in the real world, improving the accuracy and efficiency of interactions [5]. By applying vibrations or physical forces to the user's hand, the characteristics of the object such as surface texture, resistance, elasticity, and weight can be indirectly perceived, allowing for the representation of non-penetration and stiffness in object manipulation, which in turn facilitates more intuitive immersion [6],[7]. Without precise haptic feedback, users may unintentionally drop objects or fail to adjust the appropriate finger strength when gripping or releasing objects [8]. Hand-based interactions that allow for free use of fingers while delivering haptic feedback enable natural, realistic interactions both visually and tactually, significantly enhancing the sense of immersion and realism in VR experiences [9],[10]. Several studies have explored the development of hardware that provides haptic feedback and supports immersive hand interactions, functionally similar to existing commercial VR controllers. The CLAW is a handheld controller designed to allow free movement of the index finger and deliver haptic feedback through sub-motors and vibration actuators. It provides appropriate vibrations or force feedback based on the force exerted by the index finger, thereby enhancing the user's sense of realism and immersion during object manipulation [11]. The TORC is another handheld controller that enables 3784 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate precise manipulation by actively engaging the thumb, index, and middle fingers to grab or rotate objects. It is designed to convey resistance and vibrations to the fingers during interaction, thereby improving the perception of texture and properties of objects and enhancing the realism of object manipulation [12]. In addition, there are studies that utilize camera-based hand tracking technology to enable the free use of all five fingers in object manipulation. Johnson, C employed a VR headset with an integrated camera capable of recognizing and tracking the hand to allow users to manipulate virtual objects with five fingers during tasks such as sorting virtual balls [13]. Similarly, Rantamaa, H utilized a VR headset with hand tracking capabilities to facilitate natural hand-based interactions during virtual surgery training [14]. Furthermore, research evaluating the naturalness and intuitiveness of using five fingers to perform tasks, such as placing differently colored balls into corresponding boxes and inputting numbers on a virtual keyboard, has been conducted [15]. However, camera-based hand tracking technology cannot provide any feedback to the hand and may suffer from accuracy issues due to lighting conditions, hand position, and latency problems, potentially diminishing the user's sense of realism and immersion [16]. xTouch combines Leap Motion, a camera-based hand tracking technology, with a wrist-worn device capable of generating electrical signals at the fingertip. This system allows users to move their hands freely while providing tactile feedback through electrical nerve stimulation, based on the force exerted by the fingers on objects. This setup enhances the sense of realism and immersion by delivering realistic tactile feedback [17]. Moon, H combined a custom glove, which generates vibrations, with an integrated camera in a headset to provide both free hand movement and haptic feedback, thus improving the sense of realism and immersion experienced by users during VR rhythm games [18]. Scheggi, S integrated Leap Motion with a wearable device that envelops the fingertips and delivers force feedback. By adjusting the position of a platform at the fingertip upon contact with an object, this system applies pressure, resulting in a more natural and realistic physical interaction between the fingers and objects [19]. E. Amirpour argued that a wearable exoskeleton device, which allows for the free movement of all five fingers while delivering more intuitive haptic feedback, can significantly enhance the user's sense of realism and immersion [20]. Topini, A demonstrated that in VR rehabilitation training, wearing an exoskeleton device to manipulate objects allows the detection of the user's strength and movements, adjusts the resistance applied to the hands, and supports the force required for lifting objects. This approach maximized rehabilitation effects by facilitating more natural interactions [21]. A study introducing a social VR clinic for knee osteoarthritis patients employed the SenseGlove exoskeleton device to enable the movement of the user's five fingers and provide force feedback at the fingertips. This setup aimed to increase user immersion during training with virtual injection tools and to offer effects similar to actual therapy [22]. In a virtual finger rehabilitation training system, users of the SenseGlove engaged in tasks involving grasping and releasing objects of various shapes and sizes. The provision of force feedback allowed for more realistic manipulation of objects, enhancing training immersion and enabling accurate assessment of each finger's extension capabilities [23]. Kuling, I's research found that participants wearing the SenseGlove, while remotely controlling a robotic arm to move blocks to different containers, experienced enhanced realism and performance in virtual tasks when vibrations or force feedback were provided to the fingers [24]. However, these studies predominantly focus on the fingertips' involvement in object manipulation, with limited application of the palm, finger phalanges, and sides. The use of the palm and finger phalanges could enable more natural lifting of heavy objects, and if the sides of the fingers could support objects like thin cards or pens, users would likely perceive the experience as more realistic. Furthermore, providing more sophisticated haptic feedback that adapts to the characteristics and constraints of the object, rather than simple feedback when the hand contacts the object, could further enhance immersion. Therefore, this study proposes an object manipulation technique using the SenseGlove DK1 that allows comprehensive use of the entire hand while delivering precise haptic feedback to simulate non- penetration between the hand and the object and to express the object's stiffness. 3785 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate 3. Haptic Based Six Hand Areas 3.1. Virtual Hand Physical Model A physical model of a virtual hand, designed to enable users to freely move all five fingers and receive haptic feedback, was constructed with multiple layers (Physics, Manipulation, Feedback Layer). The wearable exoskeleton device, SenseGlove DK1, was employed to track finger movements and deliver vibration and force feedback. The base of the virtual hand model utilized the SGHand model from the holographic texture included in the SenseGlove SDK. The Physics Layer implements simple physical collision effects, such as the displacement of objects when colliding with the hand, based on Unity engine’s rigidbody. As shown in Figure 1(a), capsule colliders are arranged in a shape that conforms to the hand, preventing the hand and objects from passing through each other. This setup allows for actions such as catching a falling ball with the palm and rolling the ball without any constraints. The Manipulation Layer enables the grabbing and releasing of objects. As illustrated in Figure 1(b), numerous small sphere colliders are distributed throughout the hand. These sphere colliders detect other objects and facilitate grabbing and releasing based on the constraints defined by the object manipulation algorithm. To support various grabbing methods that can be executed with five fingers, four colliders are placed on the front surface of each finger and two pairs of two colliders on the side surfaces of each finger. Additionally, to support the manipulation of objects requiring palm strength, seven sphere colliders are placed on the palm. The Physics Layer ensures that the hand and object do not penetrate each other during collisions. However, when the Manipulation Layer is used to grab an object, continuous collisions between the fingers and the object may increase computational load and lead to unstable physical phenomena in the engine. To mitigate this, collisions between the object and the hand are temporarily disabled during the grabbing process to prevent mutual repulsion. Although this allows the fingers to pass through the object, the Feedback Layer employs haptic feedback to prevent the fingers from penetrating the object while grabbing. As shown in Figure 1(c), the Feedback Layer incorporates slightly larger capsule colliders at the tips of each finger, compared to the object-detecting colliders. When these colliders contact the object, haptic feedback is provided to the user’s hand based on algorithmic constraints and the characteristics of the object, thereby preventing penetration and representing the object's rigidity. This virtual hand model allows for the free manipulation of objects with all five fingers and the palm, and the haptic feedback enhances the realism of interactions. Figure 1. Layers comprising the virtual hand model. (a) Physics Layer, (b) Manipulation Layer, (c) Feedback Layer. 3.2. Object Manipulation Module Module was developed to facilitate smooth interaction between the virtual hand model and objects while delivering appropriate force and vibration feedback to the user, according to the algorithm's 3786 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate constraints. The interaction with objects consists of grasping and releasing, and for an object to be grasped, the fingers or palm must make contact with the object. As shown in Figure 2(a), the Manipulation Layer of the hand model consists of six detection areas, one for each finger (five areas) and one for the palm. Each detection area is activated when a sphere collider, placed within the area, comes into contact with the object, turning the area into a detected area. If contact with the object is lost, the detected area is deactivated. Detected areas are used for object grasping, and the number of detected areas in contact with the object is calculated upon interaction. As shown in Figure 2(b), when the sphere collider on the index finger touches the object, one detected area is calculated. If the index, middle, and ring fingers make contact, three detected areas are calculated. For typical object manipulation, at least two detected areas, including either the thumb or palm area, are required, and for larger objects, a greater number of detected areas is necessary. As shown in Figure 3(a) and (b), small objects can be grasped with just two finger areas detected, while larger objects, such as a ball, require detection from both the fingers and the palm. Multiple sphere colliders are placed on the fingertips, sides, and phalanges of the fingers, allowing for various grasping techniques. This allows for object lifting using only the finger phalanges and palm, or for grasping smaller, thinner objects such as cards or pens with the sides of the fingers, as shown in Figure 3(c). Figure 2. Detection areas for hands. (a) six detection area, (b) example of detected finger area. Figure 3. 3787 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate Grasping of various objects. (a) small volume object, (b) large volume object, (c) Small and thin object. An object is released when the thumb or palm areas disengage from the object, or when the number of detected areas required for grasping falls below the minimum threshold. For instance, as shown in Figure 3(a), if an object like a small box is grasped between the thumb and middle finger, and either the thumb or the middle finger disengages, causing the detection to be lost, the box will be released. In cases where the object is large, or due to manipulation issues such as penetrating the object with the hand while grasping, the object may unintentionally remain within the detection areas even when the hand is fully open, preventing the object from being released. To avoid such situations, finger angle values are continuously tracked to ensure proper release. The finger angle values are tracked within a range of 0 to 1 (0 = fully extended, 1 = fully bent) and stored in an array at the moment of a successful grasp. During the grasp, the difference between the stored finger angle values and the current tracked angle values is continuously calculated. If the difference exceeds the release threshold of 0.07, the corresponding finger is excluded from the detected area, indicating it can no longer maintain the grasp. For example, if the index finger angle at the moment of a successful grasp was 0.3, and the current tracked angle is 0.2, the difference of 0.1 exceeds the threshold of 0.07, leading to the exclusion of the index finger from the detected area. As shown in Figure 3(b), when releasing a large ball, even if the user fully extends their fingers, unintended contact between the spheres on the lower phalanges and the object may maintain enough detected areas to prevent the ball from being released. In such cases, adjusting the number of detected areas through the finger angle difference calculations ensures that the object is properly released. The direction and velocity of the object at the moment of release are calculated based on the position values from the 10 frames prior to release and the object's weight. This object manipulation module delivers force and vibration feedback to the user's hand during interaction, ensuring non-penetration between the hand and the object, as well as representing the stiffness of the object. Force feedback is delivered using the force feedback properties defined for each object, applying force in the opposite direction of the finger's movement. These properties include a distance value, which sets the maximum depth the finger can penetrate into the object from the point of contact, and a maximum force feedback value that can be applied to the finger. As the finger moves deeper into the object, the force feedback gradually increases until the maximum force is applied at the specified maximum distance. The maximum force feedback value can be set within a range of 0 to 1, where a value of 1 represents the maximum force. When the hand contacts an object, the module applies maximum force feedback to prevent the fingers from further penetration, thus physically controlling the movement and ensuring non-penetration. Additionally, the distance values are adjusted for each object to represent varying degrees of stiffness. For example, in the case of a hard steel object, the distance value is set to 0, meaning that from the moment of contact, the maximum force feedback value of 1 is applied, preventing any further penetration of the finger. On the other hand, for an object like a stress ball, with a distance value of 0.5 cm (as shown in Figure 4), the force feedback starts at 0 upon contact and gradually increases as the finger penetrates deeper, allowing the finger to penetrate up to the maximum distance of 0.5 cm. To prevent the user's fingers from ignoring the force feedback and penetrating the object, additional vibration feedback is applied. When the capsule collider in the Feedback Layer touches the object, and the maximum force feedback is being applied to the finger, vibrations are triggered. These vibrations serve as a signal that the finger has reached the maximum penetration distance, and further attempts to penetrate will result in additional vibration feedback, indicating that the hand is attempting to pass through the object. at the point where maximum force feedback is applied, the finger angle values are stored, and as the finger angle increases beyond the stored value (i.e., as the finger bends further, penetrating the object), the intensity of the vibrations also increases, allowing the user to intuitively recognize how much the hand is penetrating the object. 3788 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate Figure 4. Manipulation of the stress ball. 4. Experiment & Result 4.1. Experimental Overview This study conducted an experiment using the developed object manipulation technology that incorporates five-finger control and haptic feedback. The experiment consisted of three tasks: stacking cubes, fitting shapes, and throwing balls. In the stacking cubes task, as shown in Figure 5, participants had to stack six colored cubes on a black pad, matching the color and shape of six transparent cubes displayed on the pad. Task completion time and cube displacement were measured. Cube displacement was used to determine how accurately the cubes were placed, calculated by summing the distance differences at the eight vertices between the participant's cube and the target cube. The distance difference for each cube ranged from 0 to 30, where 0 indicated perfect alignment with the target cube, and 30 indicated maximum displacement, meaning no overlap. The total displacement value for the six cubes ranged from 0 to 180. The target cubes appeared after pressing the start button, and their colors were assigned randomly at the beginning.In the fitting shapes task. As shown in Figure 6, participants were required to fit five different shapes into appropriately sized holes in a large box on the left. Only task completion time was measured. In the throwing balls task, as shown in Figure 7, participants had to throw balls to knock down three targets. The targets were placed at distances of 1.0m, 1.5m, and 2.0m from the throwing position, starting with the closest . Task completion time and the number of throws were recorded, with the number of throws representing how many attempts were made to hit all the targets. 3789 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate Figure 5. Stacking cubes task process. Figure 6. Fitting Shapes Task Process. Figure 7. Throwing Balls Task Process. After participants equipped the haptic device and HMD, they were asked to confirm their hand movements. One of the three tasks was randomly assigned, and participants received a brief explanation of the assigned task. They then pressed a red button on the table to begin the experiment. The time taken to complete each task was recorded, and the remaining tasks were performed in the same manner. Upon completing all three tasks, participants filled out a questionnaire and concluded the experiment. A 3790 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate total of 10 participants took part in the study, including 8 males and 2 females. Among them, 3 participants had no prior VR experience, 4 participants had 1 to 5 experiences, and 3 participants were experienced with more than 6 VR sessions. Before the experiment, participants were provided with an explanation of the procedure and safety measures and were informed that they could stop the experiment at any time. The experiment was conducted using Unity version 2022.3.10f1, with the HTC Vive Pro Eye and SenseGlove DK1 to simulate interactions with the shapes, providing both force and vibration feedback during manipulation. 4.2. Experimental Results 4.2.1. Performance measurement In the stacking cubes task, the average task completion time was 76.5 seconds, with a standard deviation of 25.1 seconds, making it the second fastest task with the least variation between participants. This suggests that the stacking cubes task was less complex than the other tasks, and participants did not encounter significant difficulties in grasping and placing the cubes. The average cube displacement was 42.7, with a standard deviation of 9.4. The variation in completion times and displacements occurred due to differences in how participants placed the cubes. The gap between the target cubes was not wide enough, leading some participants to interfere with already placed cubes when releasing the cube they were holding. Participants who did not account for this interference often knocked over or displaced the cubes, causing delays as they had to rearrange them, which in turn increased the displacement values when the cubes were left in a displaced state. In the fitting shapes task, the average task completion time was 163.4 seconds, with a standard deviation of 55.5 seconds, making it the longest of the three tasks, with the greatest variance among participants. Some participants experienced difficulty when inserting the shapes into the box. In real life, when an object is inserted into a small space, the end of the object may touch the inside walls, but due to friction and other physical properties, it naturally slides into place. However, in this task, when participants tried to insert a shape at an angle, the end would often wobble or stop altogether, causing unstable physical behavior. Participants who aligned the shapes perfectly with the opening and inserted them straight were able to complete the task quickly, while those who relied on real-world physics took significantly more time to complete the task. A cone, which could be inserted in any direction, was easily placed by all participants, but irregularly shaped objects like squares or pentagons posed challenges for participants with little or no VR experience, as they struggled to align the shape with the opening, resulting in longer completion times. In the throwing balls task, the average task completion time was 75.2 seconds, with a standard deviation of 41.1 seconds. Although the task completion time was similar to the stacking cubes task, individual variation was considerably higher. The average number of throws was 18.8, with a standard deviation of 10 throws. The number of throws increased proportionally with task completion time, as repeated attempts to adjust the trajectory and hit the targets contributed to the increase in task duration. Participants easily hit the closest target within an average of three throws, but had difficulty hitting the farthest target, resulting in many attempts and an overall increase in task completion time. Variations in physical abilities and VR experience also contributed to the wide range of throw attempts. The two fastest participants completed the task in 25 seconds, with 4 and 5 throws, respectively, while the two slowest participants took more than two minutes and needed 30 and 34 throws to complete the task. 4.2.2. Measurement of immersion and task load To assess the realism and immersion of these tasks, the Presence Questionnaire (PQ) was used as the survey tool [25]. The PQ survey consists of four subscales: Involvement, Sensory Fidelity, Adaptation/Immersion, and Interface Quality, and is rated on a 7-point Likert scale from 1 to 7. The reliability of the PQ survey was confirmed with a Cronbach's alpha value of 0.816, indicating a reliable level. The average and standard deviation for each factor were calculated, as shown in Figure 8. Involvement measures how mentally immersed and focused participants were during the task, as well as their perception of interaction with the environment. The average score for this factor was 6.2, with a 3791 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate standard deviation of 0.7, indicating that participants were able to fully immerse themselves and positively interact with the environment. Questions regarding how much participants were visually immersed in the environment received high scores, which can be attributed to the realistic rendering of hand-object interactions through haptic feedback that prevented object penetration, making the manipulation of objects appear natural. The highest-scoring questions pertained to the naturalness of physical interactions. Participants reported feeling that the manipulation of objects was realistic, as the system allowed for non-penetration and provided a variety of grasping methods similar to real-world interactions, as well as enabling them to perceive object stiffness. Sensory Fidelity evaluates how realistic and accurate the sensory stimuli in the virtual environment felt to the participants. The average score was 5.9, with a standard deviation of 1.0. This factor scored lower than Involvement and Adaptation/Immersion, primarily due to three questions addressing auditory stimuli, which received low scores. Since auditory feedback was not a focus of this study, with only the sound of balls hitting targets and objects falling implemented, participants had limited auditory cues to evaluate. However, questions regarding the impact of haptic feedback on virtual object exploration received high scores. This is because the precise and intuitive delivery of haptic feedback — such as controlling finger movement with physical force before object penetration and signaling how much the fingers had penetrated through vibrations — had a positive influence on the participants' experience. Adaptation/Immersion measures how quickly participants adapted to events in the virtual environment based on their actions and how deeply they became immersed in the environment. The average score for this factor was 6.2, with a standard deviation of 0.8. This indicates that most participants adapted to the virtual environment quickly, felt comfortable, and were able to immerse themselves. High scores were given to questions regarding the ability to predict the outcome of their actions, likely due to the natural object manipulation provided by the technology, which offered a familiar experience similar to handling real-world objects, allowing participants to anticipate how the objects would behave. The high scores related to immersion suggest that participants felt the object manipulation process was realistic and that they could predict the results of their actions, enabling them to focus naturally on the tasks. If participants had experienced unnatural or disconnected interactions, the immersion scores would have been lower. The consistency of the sensory information also received high scores, which can be attributed to the precise haptic feedback provided at the exact moment when the fingers visually made contact with or penetrated the object, maintaining a successful sense of consistency. Interface Quality evaluates how much the interface in the virtual environment hindered the task, with lower scores indicating a more positive outcome. The average score for this factor was 2.5, with a standard deviation of 1.2. The wider variation in this factor compared to others can be attributed to the questions about how much the device itself interfered with the tasks. The SenseGlove DK1, which is worn by securing straps to the fingertips, caused discomfort for some participants, particularly during the throwing balls task, as the dynamic movements of the hand caused the fingertip straps to loosen, making hand movements more difficult. However, participants did not report significant issues with delays between real- world actions and those in the virtual environment, nor with immersion problems related to the visual display. 3792 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate Figure 8. Presence measurement. Additionally, the Simulation Task Load Index (SIM-TLX) was used to measure the task load experienced during the experiment [26]. The SIM-TLX survey consists of nine subscales, rated on a 21-point Likert scale from 0 to 20. The reliability of the SIM-TLX survey was confirmed with a Cronbach's alpha value of 0.749, indicating a reliable level. The average and standard deviation for each subscale were calculated, as shown in Figure 9. The Mental Demands subscale, which evaluates the participants' mental fatigue during the task, had an average score of 3.8, with a standard deviation of 3.4, suggesting that participants did not feel significant mental fatigue while performing the tasks. However, the average score for Physical Demands, which assesses physical fatigue, was 6.2 with a standard deviation of 3.5. This suggests that participants felt physical strain due to the use of the exoskeleton device, SenseGlove. The device reduced the freedom of hand movement compared to using bare hands, and participants could have felt burdened by its weight. Additionally, participants unfamiliar with force and vibration feedback may have found the hand movements awkward, leading to accumulated physical fatigue. The average score for Temporal Demands, which evaluates how hurried participants felt during the tasks, was 6.1 with a standard deviation of 4.0. The Frustration subscale, which measures the level of frustration experienced during the tasks, had an average score of 2.6 with a standard deviation of 3.9. In the fitting shapes task, some participants experienced time pressure when they failed to insert the shapes properly due to issues with the physics engine, leading to prolonged task times. In the throwing balls task, some participants struggled to hit the 2-meter target, and despite repeated throws, they could not successfully hit the target, causing them to rush and feel frustrated. Prolonged tasks are closely tied to increased physical fatigue. The average score for Task Complexity, which evaluates how complex the tasks were perceived to be, was 2.8 with a standard deviation of 2.7. These tasks were relatively simple and similar to tasks participants would encounter in real life, which likely contributed to the low score for complexity. Situational Stress, which measures the stress caused by the task environment itself, had an average score of 2.6 with a standard deviation of 2.7. The Distractions subscale, which assesses how much factors inside or outside the virtual environment disrupted participants' focus, had an average score of 1.0 with a standard deviation of 1.2. The Perceptual Strain subscale, which evaluates discomfort or fatigue from processing sensory information (such as visual or tactile input), had an average score of 0.6 with a standard deviation of 1.3. This indicates that participants did not experience significant discomfort or stress, likely due to the familiar and realistic object manipulation process provided by the natural haptic feedback. The Task Control subscale, which assesses how well participants felt they could control their hands and objects during the tasks, had an average score of 6.0 with a standard deviation of 4.2. While most participants felt they could manipulate their hands and objects effectively, some participants experienced discomfort due to the device’s impact on hand movement, as analyzed in the 3793 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate Interface Quality subscale of the PQ. Additionally, participants who spent an extended amount of time on the throwing balls or fitting shapes tasks found it more difficult to control their hands and objects. Figure 9. Task load measurement 5. Conclusion This study proposed an object manipulation technology that utilizes five-finger control and haptic feedback to enhance the sense of realism and immersion during interactions in VR. A physical model of a virtual hand was designed, allowing the fingertips, sides, phalanges, and palm to be used for object manipulation, and an object manipulation module was developed to deliver haptic feedback to the user's hand during the manipulation process. Through this virtual hand model, various grasping techniques that can be performed with a real hand were supported, and the manipulation module provided force and vibration feedback to prevent penetration between the hand and object, allowing users to feel the object's stiffness. An experiment involving three tasks of object manipulation in VR was conducted using this technology. Analysis of the PQ survey responses revealed that most participants evaluated the physical interactions between the hand and object as highly realistic. The sensory feedback, both visual and tactile, provided during the interactions was consistent and natural, leading to deep immersion in the tasks and allowing participants to easily adapt to the virtual environment. In the SIM-TLX survey analysis, participants reported low levels of mental fatigue, stress, frustration, and distraction. However, physical fatigue, time pressure, and task difficulty were rated slightly higher compared to other factors. This suggests that the technology allowed participants to experience a high level of realism in object manipulation and to immerse themselves well in the tasks. Further research is needed with a larger pool of participants to incorporate object weight conditions and simplify the manipulation process, while maintaining high levels of realism and immersion, reducing physical fatigue, and improving task performance. If this advanced technology is applied to tasks in industrial settings that require precise hand movements or to medical procedures in VR, it would enable intuitive and repetitive training, thereby maximizing training effectiveness and efficiency. 3794 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate Acknowledgements: This paper was supported by the Education and Research Promotion Program of KOREATECH in 2024 and following are results of a study on the "Leaders in INdustry-university Cooperation 3.0" Project, supported by the Ministry of Education and National Research Foundation of Korea. Copyright: © 2024 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). References [1] V. Brugada-Ramentol, I. Clemens, and G. G. de Polavieja, "Active control as evidence in favor of sense of ownership in the moving Virtual Hand Illusion", Consciousness and Cognition, vol. 71, pp. 123–135, 2019 https://doi.org/10.1016/j.concog.2019.04.003 [2] F. Argelaguet, L. Hoyet, M. Trico, and A. Lécuyer, "The role of interaction in virtual embodiment: Effects of the virtual hand representation", in 2016 IEEE Virtual Reality (VR), vol. 1, pp. 3–10, Mar. 2016 DOI: 10.1109/VR.2016.7504682 [3] M. Newman, B. Gatersleben, K. J. Wyles, and E. Ratcliffe, "The use of virtual reality in environment experiences and the importance of realism", Journal of Environmental Psychology, vol. 79, p. 101733, 2022. https://doi.org/10.1016/j.jenvp.2021.101733 [4] M. Höll, M. Oberweger, C. Arth, and V. Lepetit, "Efficient physics-based implementation for realistic hand-object interaction in virtual reality", in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), vol. 1, pp. 175– 182, Mar. 2018. DOI: 10.1109/VR.2018.8448284 [5] C. Våpenstad, E. F. Hofstad, T. Langø, R. Mårvik, and M. K. Chmarra, "Perceiving haptic feedback in virtual reality simulators", Surgical Endoscopy, vol. 27, pp. 2391–2397, 2013. https://doi.org/10.1007/s00464-012-2745-y [6] F. Danion, J. S. Diamond, and J. R. Flanagan, "The role of haptic feedback when manipulating nonrigid objects", Journal of Neurophysiology, vol. 107, no. 1, pp. 433–441, 2012. https://doi.org/10.1152/jn.00738.2011 [7] O. Schober, J. Diephuis, P. Wintersberger, and W. Hochleitner, "Heavy Haptics: Simulating Weight in Virtual Reality Applications", in 2023 International Conference on Intelligent Metaverse Technologies & Applications (iMETA), pp. 1–6, Sept. 2023. DOI: 10.1109/iMETA59369.2023.10294811 [8] M. Chessa, G. Maiello, L. K. Klein, V. C. Paulun, and F. Solari, "Grasping objects in immersive Virtual Reality", in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 1749–1754, Mar. 2019. DOI: 10.1109/VR.2019.8798155 [9] J. Kim, "A comparative study on hand-based interfaces in immersive virtual reality", Journal of the Korea Computer Graphics Society, vol. 25, no. 2, pp. 1–9, 2019. DOI: https://doi.org/10.15701/kcgs.2019.25.2.1 [10] S. Oprea, P. Martinez-Gonzalez, A. Garcia-Garcia, J. A. Castro-Vargas, S. Orts-Escolano, and J. Garcia-Rodriguez, "A visually realistic grasping system for object manipulation and interaction in virtual reality environments", Computers & Graphics, vol. 83, pp. 77–86, 2019. https://doi.org/10.1016/j.cag.2019.07.003 [11] I. Choi, E. Ofek, H. Benko, M. Sinclair, and C. Holz, "Claw: A multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality", in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–13, Apr. 2018. https://doi.org/10.1145/3173574.3174228 [12] J. Lee, M. Sinclair, M. Gonzalez-Franco, E. Ofek, and C. Holz, "TORC: A virtual reality controller for in-hand high- dexterity finger interaction", in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1– 13, May 2019. https://doi.org/10.1145/3290605.330030 [13] C. I. Johnson, N. W. Fraulini, E. K. Peterson, J. Entinger, and D. E. Whitmer, "Exploring hand tracking and controller-based interactions in a VR object manipulation task", in International Conference on Human-Computer Interaction, Cham: Springer Nature Switzerland, pp. 64–81, Jul. 2023. https://doi.org/10.1007/978-3-031-48050-8_5 [14] H. R. Rantamaa, J. Kangas, S. K. Kumar, H. Mehtonen, J. Järnstedt, and R. Raisamo, "Comparison of a VR stylus with a controller, hand tracking, and a mouse for object manipulation and medical marking tasks in virtual reality", Applied Sciences, vol. 13, no. 4, p. 2251, 2023. https://doi.org/10.3390/app13042251 [15] J. N. Voigt-Antons, T. Kojic, D. Ali, and S. Möller, "Influence of hand tracking as a way of interaction in virtual reality on user experience", in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–4, May 2020. DOI: 10.1109/QoMEX48832.2020.9123085 [16] Y. Lei, Y. Deng, L. Dong, X. Li, X. Li, and Z. Su, "A novel sensor fusion approach for precise hand tracking in virtual reality-based human-computer interaction", Biomimetics, vol. 8, no. 3, p. 326, 2023. https://doi.org/10.3390/biomimetics8030326 [17] A. K. Shell, A. E. Pena, J. J. Abbas, and R. Jung, "Novel neurostimulation-based haptic feedback platform for grasp interactions with virtual objects", Frontiers in Virtual Reality, vol. 3, p. 910379, 2022. https://doi.org/10.3389/frvir.2022.910379 [18] H. S. Moon, G. Orr, and M. Jeon, "Hand tracking with vibrotactile feedback enhanced presence, engagement, usability, and performance in a virtual reality rhythm game", International Journal of Human–Computer Interaction, vol. 39, no. 14, pp. 2840–2851, 2023. https://doi.org/10.1080/10447318.2022.2087000 [19] S. Scheggi, L. Meli, C. Pacchierotti, and D. Prattichizzo, "Touch the virtual reality: using the Leap Motion controller https://creativecommons.org/licenses/by/4.0/ 3795 Edelweiss Applied Science and Technology ISSN: 2576-8484 Vol. 8, No. 6: 3782-3795, 2024 DOI: 10.55214/25768484.v8i6.2822 © 2024 by the authors; licensee Learning Gate for hand tracking and wearable tactile devices for immersive haptic rendering", in ACM SIGGRAPH 2015 Posters, pp. 1–1, 2015. https://doi.org/10.1145/2787626.2792651 [20] E. Amirpour, M. Savabi, A. Saboukhi, M. R. Gorji, H. Ghafarirad, R. Fesharakifard, and S. M. Rezaei, "Design and optimization of a multi-DOF hand exoskeleton for haptic applications," in 2019 7th International Conference on Robotics and Mechatronics (ICRoM), pp. 270–275, Nov. 2019. IEEE. https://doi.org/10.1109/ICRoM48714.2019.9071900 [21] A. Topini, W. Sansom, N. Secciani, L. Bartalucci, A. Ridolfi, and B. Allotta, "Variable admittance control of a hand exoskeleton for virtual reality-based rehabilitation tasks", Frontiers in Neurorobotics, vol. 15, Art. no. 789743, 2022. https://doi.org/10.3389/fnbot.2021.789743 [22] T. Xue, J. Li, G. Chen, and P. Cesar, "A social VR clinic for knee arthritis patients with haptics", in Proceedings of the ACM International Conference on Interactive Media Experience, vol. 2, 2020. [23] A. A. I. Aly, A. Abbasimoshaei, and T. A. Kern, "Developing a VR training environment for fingers rehabilitation", in 13th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2022, pp. 331–333, 2022. https://doi.org/10.15480/882.4368 [24] I. A. Kuling, K. Gijsbertse, B. N. Krom, K. J. van Teeffelen, and J. B. van Erp, "Haptic feedback in a teleoperated box & blocks task", in Haptics: Science, Technology, Applications: 12th International Conference, EuroHaptics 2020, Leiden, The Netherlands, September 6–9, 2020, Proceedings 12, Springer International Publishing, pp. 96–104, 2020. https://doi.org/10.1007/978-3-030-58147-3_12 [25] B. G. Witmer, C. J. Jerome, and M. J. Singer, "The factor structure of the presence questionnaire", Presence: Teleoperators & Virtual Environments, vol. 14, no. 3, pp. 298–312, 2005. https://doi.org/10.1162/105474605323384654 [26] D. Harris, M. Wilson, and S. Vine, "Development and validation of a simulation workload measure: the simulation task load index (SIM-TLX) ", Virtual Reality, vol. 24, no. 4, pp. 557–566, 2020. https://doi.org/10.1007/s10055-019- 00422-9