Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources https://doi.org/10.3991/ijim.v11i4.6712 Sandi Ljubic Faculty of Engineering, University of Rijeka, Croatia sandi.ljubic@riteh.hr Damir Arbula Faculty of Engineering, University of Rijeka, Croatia damir.arbula@riteh.hr Abstract—In the area of Human-Computer Interaction (HCI), the concept of Around Device Interaction (ADI) is gaining special attention nowadays. It provides the possibility of interacting with the device by making use of gestures made in the surrounding space, as well as by detecting certain ambient changes, thus eliminating the need of direct physical contact. Contact-free interaction can be useful for wearable and smart devices, especially in cases where operating with small buttons and touchscreens appears to be particularly inconvenient. This paper presents alternative techniques for smartphone interaction based on different sensory inputs. Namely, external sources of magnetic field, light, and infrared (IR) signals are used to manipulate device sensors’ readings which are in turn interpreted as the corresponding interaction commands. Proof-of- concept mobile applications are developed and here presented, demonstrating both the potentials and utility of the contact-free interaction. Specifically, small neodymium magnet was used for smartphone text entry, typical battery flash- light served for sending information to device via Morse code, as well as for alarm triggering, and voice-enabled control device was utilized for invoking smartphone calls from a distance. The presented interaction techniques do not require smartphone hardware alterations. While the usage of built-in magnetic field and ambient light sensors is assumed, peripheral infrared receiver can be used in order to provide IR-based support. Keywords—Around Device Interaction (ADI), Human-Computer Interaction (HCI), mobile devices, sensor-based interaction 1 Introduction Touch gestures still represent the dominant interaction technique for devices en- compassed by a contemporary mobile domain. Apart from capacitive stylus, human finger plays a role of main input device for smartphones and tablets in most cases. However, considering small form-factor of touchscreen devices, and impact of the well-known fat finger syndrome, touch interaction impose a number of inherent limi- 66 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources tations. There are many use cases wherein interaction burden is implicitly enhanced (one-handed usage, display occlusion) or interacting via touch gestures is simply not convenient. The feasibility of touch interaction becomes even more questionable with the prospect of ubiquitous and wearable computing, seeing that input controls of cor- responding devices (e.g. smartwatches) are even more constrained. Limitations of touch interaction have been addressed and tackled by enabling al- ternative mobile interaction modalities which rely on sensory input. Since most of the nowadays mobile devices have a rich set of built-in sensors, a number of solutions are developed so as to utilize cameras, microphones, accelerometers, and gyroscopes for augmenting their interaction support. Accelerometers and gyroscopes can be used for monitoring device movements, such as tilt, shake, rotation or swing. Apart from straightforward implementations that utilize the position of the mobile device for changing screen orientation, scrolling, navigation through lists and menus, and con- trolling mobile game applications, tilt-based interaction has been successfully used within alternative text entry solutions for smartphones and tablets [1, 2]. In addition to augmenting interaction with device positional awareness, many re- search attempts are focused on increasing input space around the device. ADI concept allows interaction commands to be invoked by natural movement-based gestures made in the space around the device, as well as by making use of changes detected in the surrounding environment. One of the most recent Google projects, named Soli [3], presents a new sensing technology that uses miniature radar for motion tracking of the human hand. Fine motions of both thumb and index finger are used within a 3D- gesture interaction language. Projects like Soli exemplify the growing popularity and projected utility of the sensor-based interaction in ubiquitous computing. When it comes to ADI supported by typical smartphone sensors, front facing camera is used in vision-based input detection systems for capturing finger gestures performed above the respective sensor [4]. However, occlusion and sensitivity to lighting conditions still limit the use of such vision-based techniques in mobile environments. Built-in microphone can be of particular utility as well, allowing audio signals to be interpret- ed as input commands. For example, finger-tapping on a conventional surface (e.g. wooden table) produces enough audio information for ‘keystroke’ localization, thus enabling ADI-based text entry [5]. The main motivation of our work was to explore ADI possibilities, by utilizing other sensors which are not yet broadly considered (but still embedded in popular gadgets) for interaction purposes. More specifically, we want to present how contact- free interaction with mobile devices can be achieved using magnetic field sensor, ambient light sensor, and infrared sensor. In this paper we deliver proof-of-concept implementations which have a twofold objective: (i) to demonstrate corresponding smartphone sensors as appropriately accurate and reliable, and (ii) to propose some real-life use cases in which related interaction techniques could provide assistance. In the next section we address related work, mainly associated with the existing ef- forts to enhance interaction possibilities by magnetic field manipulation. Unlike the magnetic field, ambient light and infrared signals are hardly ever considered for utili- zation in the ADI domain. Subsequent sections deal with our proof-of-concept im- plementations: key ideas are introduced, and mobile applications, as well as their iJIM ‒ Vol. 11, No. 4, 2017 67 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources usage, are presented. General impressions from initial testing are discussed in the concluding section. 2 Related work One of the first attempts to utilize external magnetic field as a way of input interac- tion for small wristwatch-like devices was formalized in Abracadabra prototype [6]. This prototype relies on specially developed hardware platform (some sort of smart- watch prequel), as well as on permanent disk-shaped neodymium magnet intended to be attached to user’s index finger. By using such setup, interaction space was aug- mented with an additional 300 cm2 sphere-shaped area around the prototype device, originally equipped with 7 cm2 touch-disabled display. Simple finger gestures are then used for activating abstract buttons displayed on small screen, without the need of touching the target device. It was shown that pointing accuracy of the respective interaction technique heavily depends on the posture of both hands, one wearing the device with magnetic sensor, and the other one performing gestures with small mag- net attached to the forefinger. Ketabdar et al. [7] dealt with mobile phone interaction by using changes in magnet- ic field invoked by re-positioning of external magnetic material around the device. Machine learning algorithms were used for movement-based gestures classification. The temporal pattern of the recognized gesture was thus used as a basis for sending different commands to the mobile device. A demo application called MagiTact was implemented, utilizing magnet-based ADI and providing functionalities such as: turn- ing pages in a photo gallery, zooming a map, rejecting/accepting a call, and control- ling a sound volume in music player. The proposed method provides effective con- tact-free interaction technique, however, 3D gestures (i.e. trajectories of the hand holding the magnet) have to be extensively trained beforehand. Also, if permanent magnet with different size and/or magnetic properties is going to be used, then the process of gesture training should be done from scratch. MagiWrite [8] represents an application extension of MagiTact, wherein previously trained 3D gestures are used for entering digits. In the related solution, the user simply “draws” a digit-sign in the air surrounding the mobile device, using a properly shaped magnet (rod, ring, or pen) in her/his hand. Corresponding gesture is then recognized and mapped to the match- ing input command. Just like with the MagiTact, MagiWrite functionality also largely depends on gesture training and machine learning utilization. Involving a complete character set in such gesture-based input would require more sophisticated techniques of both signal processing and template matching. MagPen [9] represents a solution in which a combination of both contact and con- tact-free modality is enabled for stylus-based interaction with a smartphone device. It uses a magnetically driven pen prototype which supports standard touch functionality of a typical capacitive stylus, and ADI based on the magnetic field manipulation. MagPen prototype comes in three different forms: basic pen uses a single coin-shaped permanent magnet embedded in the center of the pen body, identifiable pen relies on different magnetic properties and position of the attached magnet, and pressure- 68 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources sensitive pen has an additional spring which enables changing the position of embed- ded magnet according to the force applied when touching the device display. A proof- of-concept drawing application was developed, so as to demonstrate the usefulness of the proposed gesture-based interaction. The following functionalities were provided: detecting the orientation and beyond-screen position of the pen, recognizing stylus different spinning gestures, identifying different operational modes according to the used pen, and utilizing the applied pressure to make different paint strokes. Machine learning algorithms were used for detecting both the pen orientation and the spinning gestures. TMotion [10] is another solution that enables spatial interaction around mobile de- vice using a magnetic sensing and a specially designed stylus. As opposed to Mag- Pen, the related stylus was built by making use of both the magnet and the inertial measurement unit (IMU). That way, tracking the 3D position of the permanent mag- net can be achieved without machine learning support and without the need of prior gesture training. Specifically, stylus localization is based on the magnetic field vector (provided by smartphone sensor readings), and the current orientation of the embed- ded magnet (provided by the IMU). Example mobile applications showed how pro- vided stylus-based ADI can be successfully used for navigating a multi-level menu interface, as well as for sketching and manipulating 3D models presented on a smartphone display. We conclude the magnet-related work overview with an interesting mobile applica- tion called MagiGuitar [11]. It uses a permanent magnet to mimic the real gesture of playing a guitar. Namely, user holds the guitar strings displayed on a smartphone screen with one hand, and periodically moves the other hand (holding a magnet) around the device, thus imitating the natural strumming actions. The sound of the currently touched strings is reproduced once the user makes a valid strumming ges- ture in the air. The proposed ADI technique thus provide natural and flexible hand gestures for a given context. As opposed to research efforts for providing ADI based on magnetic field manipu- lation, little work has been done in exploiting possibilities of ambient light and infra- red sensors. Spreitzer [12] showed how detected deviations of the ambient light, in- voked by simple tilts of a smartphone, can be used for inferring patterns of interaction with the mobile device. In his work, machine learning and template matching algo- rithms were used for training and detecting interaction gestures, based on their rela- tion with the ambient light sensor readings. However, these gestures were not used for augmenting ADI, but for demonstrating possible security leak associated with mobile interaction. On the other hand, Zhang et al. [13] proposed a system called Okuli in which they used light sensing in order to enhance interaction possibilities of a touchscreen tablet device. In the related solution, the mobile device uses a LED transmitter to project a light beam towards the finger, and two photodetectors for sensing the reflecting light intensity. Finger position is thus localized using visible light communication (VLC) principle. Provided ADI was demonstrated using virtual keyboard and virtual keypad applications, wherein input was performed on a flat surface in front of the tablet. However, it must be noted that this solution does not use iJIM ‒ Vol. 11, No. 4, 2017 69 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources typical ambient light sensor within a mobile device, but requires sophisticated periph- eral hardware module instead. Finally, the lack of research in the area of mobile infrared-supported interaction can be contributed to the fact that IR sensors are very rare in nowadays smartphones and tablets. We make a clear distinction here between IR transmitter (often called a blaster) – which can be found in new popular models, and IR receiver – which is almost non-existent. While transmitter is typically used for controlling home devices such as TVs and media players, hence enabling the mobile device to emulate outputs of standard remote control, infrared-based input is consequently hardly ever consid- ered in a mobile interaction context. According to the aforementioned, augmenting the mobile interaction with IR-based input generally requires additional hardware. For example, Hoverflow prototype [14] involves six IR distance sensors placed along the long edges of a smartphone, allowing ADI with movement-based hand gestures. In that respect, the previously described Okuli system represents IR-based solution as well, since both the LED transmitter and the photodetectors work with near-infrared wavelengths [13]. 3 ADI based on magnetic field manipulation The main idea of magnetic field manipulation is based on the fact that any external magnetic source will have a certain impact on the readings provided by the magnetic field sensor built in a corresponding mobile device. This basic principle is illustrated in Fig. 1. Fig. 1. The effect of permanent magnet placed in the area surrounding the mobile device 70 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources The embedded magnetic sensor (often called a compass) provides a measure of magnetic field strength along three axes (x, y, z), determining the related magnetic field vector H. While in most cases this vector represents Earth’s magnetic field in the given location, the sensor can be also affected partly by some sources of magnetic noise [8]. Hence, vector H represents a superposition of various magnetic sources, of which the Earth’s magnetic field usually has the most significant effect. However, if strong permanent magnet (e.g. neodymium) is placed near the mobile device, it can make a considerable change in ambient magnetic field. Fig. 1 describes such a case, wherein embedded sensor provides measures related to resulting vector H which actu- ally involves both the ambient magnetic field (HA), and the magnetic effect caused by the external magnetic source (HM). According to the aforementioned, there is an option to formally measure a magnet- ic effect solely associated with the used magnet, using a following three-step proce- dure: (i) measure the ambient magnetic field without the presence of the external magnetic source, (ii) place a permanent magnet on a given location near the mobile device and repeat the measuring, and (iii) compute the permanent magnet effect by subtracting the strength of the ambient magnetic field from the values obtained in the second measuring. The effect of permanent magnet on the sensor outputs depends on its magnetic properties, location, and orientation. Magnetic properties are associated with the in- herent strength of the magnetic field imposed by a magnet. If we assume that magnet retains constant orientation (related to the position of the mobile device), then its location can be matched with a distinct magnetic field detected by the sensor. Conse- quently, localization of the permanent magnet can be achieved using simple magnetic field “fingerprinting”. The related concept is illustrated in Fig. 2. Fig. 2. Magnetic field fingerprinting: calibrating the location of the permanent magnet accord- ing to the related sensor measurements iJIM ‒ Vol. 11, No. 4, 2017 71 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources Suppose we want to augment interaction space by providing a virtual plane in front of the device on which magnet-based interaction would take place. A number of dis- tinct areas can be selected on such a plane in order to represent different input com- mands. Fig. 2 presents a case in which a calibration is performed for three square- shaped areas. In the calibration phase of the magnetic field fingerprinting, magnet should be moved within the boundaries of the respective area. These movements will invoke different sensor measurements, forming a vector range that matches the corre- sponding locations (e.g. vector range HM(1) as a magnetic fingerprint of the area 1). All vector ranges thus obtained can be saved and henceforth used for magnet localiza- tion. In the localization phase, current measurements provided by the magnetic sensor are constantly checked against the previously saved magnetic fingerprints. If match is found, magnet location is inferred, and a given command can be issued. Different interaction designs can be implemented according to the described meth- od. No matter how many input commands are required, the related locations should be selected in such a way to minimize the possibility of magnetic fingerprint overlaps. In our proof-of-concept application, we present two possibilities of magnet-based text entry. While the first one is based on full QWERTY mapping, the second one relies on cursor navigation within a keyboard layout. 3.1 Proof-of-concept: text entry using magnet A nickel-plated neodymium magnet cube (12!12!12 mm), with N45 grade and 0,53T (magnetic flux density on the surface of the cube) was used in our proof-of- concept implementation. We introduce two text entry methods which rely on the magnet-based ADI. The first method uses a standard QWERTY character layout printed on the regular paper. The related layout consists of 33 zones representing virtual keys, as shown on Fig. 3. Fig. 3. Character layout used in the first text entry method enabled by magnet-based ADI An Android application was developed that supports calibration of different zones around the mobile device by obtaining and storing their magnetic field fingerprints. Magnetic field vector ranges are, together with the associated characters, saved in Android Shared Preferences in a key-value set. The highest sampling rate was used, 72 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources providing ~110 sensor measurements per second. Calibrating character layout from Fig. 3 required no more than 10 minutes. Using the provided text entry method is rather straightforward and requires mini- mal preparation in a given setup. Smartphone and virtual keyboard (paper) have to be placed according to the relative positions used in the calibration procedure, and ambi- ent magnetic field has to be registered in order to cancel its effect. Contact-free text entry in smartphone application is then simply achieved by moving permanent magnet within the virtual keyboard layout (Fig. 4). Since the fingerprint matching is conduct- ed using the same high sampling rate as in the calibration process (110 Hz), additional timing mechanisms are applied in order to prevent unwanted character entries. Specif- ically, we used a dwell time – minimum time needed to retain the magnet in a valid zone in order to input matching character, and a delay time – a time required to keep the magnet in a valid zone if the same character should be entered consecutively. While both time values can be configured through the application settings, initial testing showed that 150 ms and 700 ms (respectively) provide a good basis for elimi- nating possible timing issues. Fig. 4. ADI-based text entry using permanent magnet and full QWERTY paper mockup Given the strength of the used permanent magnet, no particular problems were de- tected neither in calibration process nor in the fingerprint matching. Magnetic field manipulations showed to be distinctively sensed in different zones of the A4 paper mockup. Interaction space was thus augmented by an additional 210!297 mm area in front of the device. The character zones used in the presented method were not strictly rectangle-shaped, although they can be inscribed in 22!22 mm area. Space character zone is the only exception, occupying larger area of 134!22 mm. In horizontal rows, the space between two adjacent virtual keys was 2 mm only. Regarding the smartphone model, we used Samsung Galaxy S5 running Android Marshmallow OS. In the second magnet-based text entry method we propose a solution which as- sumes lower entry rate, but works with only a few fingerprinted zones. As opposed to full QWERTY mockup with direct character selection, a cursor-based input scheme is implemented instead. Specifically, we developed an Android keyboard in which text entry is performed by navigating the cursor within the character layout. The user is required to use five commands in order to perform text entry: up, down, left, right, and select. Consequently, input design rely on five distinct zones that have to be cali- iJIM ‒ Vol. 11, No. 4, 2017 73 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources brated for magnet-based interaction. Fig. 5 presents the proposed keyboard layout, as well as the paper mockup of the “command panel”. Fig. 5. Cursor-based character layout used in the second text entry method enabled by magnet- based ADI (left); command panel paper mockup (right) As expected, calibration procedure is even less time-consuming in this case, since magnetic field fingerprints have to be obtained for five square-shaped zones only. Although 2 mm showed to be a viable distance between two neighboring zones in the respective setup, related free space can be furthermore increased in order to avoid possible issues related to magnetic fingerprints overlapping. For a proof-of-concept demonstration, we used 20!20 mm zones and a 20 mm distance from the center zone (selection command). In this case, calibration was carried out with the smartphone being portrait-oriented in relation to the paper mockup. The usage example of cursor- based text entry method, augmented by magnet-based interaction, is shown in Fig. 6. Fig. 6. Cursor-based text entry method enabled by magnetic field manipulation 4 ADI based on ambient light manipulation The ambient light sensor employed in many of today’s smartphones provides in- formation about the intensity of the surrounding light. The readings reported from the ambient light sensor are given in SI lux units, measuring the illumination intensity on a front-face surface of the mobile device. This sensory input is typically used to au- 74 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources tomatically adapt the screen brightness appropriately. For example, brightness is usu- ally increased when device display is in direct sunlight (in order to remain content visibility), whereas darker surroundings imply automatic screen dimming (in order to reduce eye fatigue). Light sensor can be furthermore utilized in a way that specific changes in sur- rounding illumination, being intentional (user-driven) or unplanned, trigger input actions within some mobile application. In such a way, any light source (e.g. light bulb, battery flashlight, LED lamp) can became an input control medium. The concept of light-based interaction with the mobile device can thus be applied by supporting following actions: (i) measuring and registering ambient illumination, excluding all external factors, (ii) setting light thresholds, i.e. lighting conditions (in lux) which correspond to certain action triggers, and (iii) defining application commands which will be fired once the light sensor detects changes hitting the threshold values. Fig. 7 demonstrates the mentioned concept, showing how typical battery flashlight can be used to trigger different measurements of the ambient light sensor. Fig. 7. The effect of external light source illuminating the mobile device In the presented scenario sensor detects illumination intensities LSOFF (ambient lighting) and LSON when flashlight, facing mobile device from a certain distance and a certain angle, is turned off and on respectively. While the effect of external light source can be straightforwardly determined as the (LSON - LSOFF) difference, the tar- get light threshold, indicating the external lighting, could be set anywhere between LSOFF and LSON (according to the precision required in the application context). 4.1 Proof-of-concept: alarm activation and Morse code communication using lighting source We firstly developed a mobile application which serves as a simple security alarm. It displays current measurements of the ambient light sensor and provides the possi- iJIM ‒ Vol. 11, No. 4, 2017 75 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources bility to configure three illumination thresholds. These values correspond to applica- tion actions as follows: (i) when detected ambient light hits the lowest threshold, phone call will be automatically made to the predefined contact number; (ii) when illumination intensity reaches the second threshold, the mobile device will start to constantly vibrate; (iii) finally, if the highest illumination is perceived, smartphone will reproduce high-volume sound signal repeatedly. The mentioned application was tested in two different contexts. In the first one the Android smartphone was located in a room at the place with moderate illumination (~60 lux). Changes in lighting (up to ~250 lux) were invoked using battery flashlight, by gradually pointing its light beam towards the smartphone display from a 2m dis- tance. All alarms were activated according to the set light thresholds in the right or- der. In the second context smartphone was positioned near the closed door in the dark room, in a way that outside lighting can reach its sensor if someone would open that door. The more opened door in this setup implicated the larger area of the device display being illuminated. We were able to simulate “peeking” through the slightly opened door, which eventually triggered the silent call alarm, as well as abrupt enter- ing which successfully activated all three alarms. The snapshot of the application, showing current status with two active alarms, can be seen in Fig. 8 (left). Fig. 8. Lighting-based proof-of-concept application snapshots: automatic alarm invocation (left), sending Morse codes (right) The second proof-of-concept solution involve light-based Morse code communica- tion. We developed a mobile application able to detect lighting signals that corre- spond to Morse code letters. The same threshold-based principle was used, additional- ly supported with timing constraints. Namely, if certain illumination level was detect- ed by the sensor, a timer was triggered to measure the duration of such light signal, thus enabling differentiation between dot and dash. Both the light threshold and the dot duration can be set through the application user interface, as shown in Fig. 8 76 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources (right). According to the Morse code rules, for the duration of a dash we used three dot durations. We were able to successfully send light-based Morse codes from a 2m distance us- ing battery flashlight. Threshold value indicating the required flashing intensity was set according to the actual ambient illumination. Among different timing values test- ed, we found 1 s and 3 s intervals to be a good choice for dot and dash durations, respectively. Application additionally tolerates the offset of 0,5 s, relaxing the need to be absolutely precise with lighting pulse durations. An example of the application usage is shown in Fig. 9. The developed application can be an asset for learning Morse code communication in an interesting and entertaining way. Since most of the contemporary smartphones come with camera and a corresponding flashlight, this built-in light source can be utilized to send light-based Morse codes from one smartphone to another. Ambient light sensor embedded in the used Samsung Galaxy S5 model was provid- ing illumination values with the sampling rate of no more than 6 Hz. Many smartphone models have light sensors able to provide higher sampling rates (~100 Hz); however 5 Hz rate was acceptable for aforementioned proof-of-concept applica- tions. Fig. 9. External lighting source used in order to send Morse codes. In the real setup, flashing was successfully detected on a 2m distance. 5 Interaction based on infrared source While nowadays smartphone devices are equipped with magnetic and visible light sensors, IR sensors are not so common. As already noted, large number of smartphone models are equipped with IR blasters. The only device equipped with IR sensor we were able to test was HTC One M8. The range of its' sensor was too low to be of any practical use other than recording and learning IR commands that can later be used to control other devices such as TV, DVD players etc. Fortunately, there are small and cheap off-the-shelf devices that can be connected to smartphone via OTG USB port. One of these devices is FLIRC [15] that we used in iJIM ‒ Vol. 11, No. 4, 2017 77 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources order to equip the smartphone device with IR receiving support. The concept of this device is very simple: on the USB side it can emulate regular keyboard device and on the receiving side it can detect and decode regular IR commands using consumer IR (CIR) protocols (Fig. 10). In the configuration, user can map detected signals with keyboard keys thus providing simple way of controlling device i.e. OS and applica- tions running on it. Base carrier frequency of IR signal is standard 38 kHz used to extend range and provide resistance to other sources of IR light, mainly daylight but also some artificial light sources [16]. After carrier signal bandpass filtering, receiver detects stream of low frequency On-off keying (OOK) modulated IR signal that can be either decoded or used as-is in the raw format. Fig. 10. Standard consumer IR signal detection and processing 5.1 Proof-of-concept: assistive technology voice activated phone call In the assistive technology domain standard use case includes voice activated de- vice that can control multiple appliances in the users' home. Apart from voice recog- nition, this device usually includes integration with existing home automation and other legacy appliances. Most legacy appliances such as TV, Stereo or Air Condition already have IR re- ceivers and can be controlled via IR signals. Thus many control devices such as Pilot Pro or Pilot Mini already come with IR blaster included. One of the many functions that this scenario includes is making and receiving phone calls. Usually this is done using IR commands being sent to specialized and expensive IR enabled phones such as GewaTel 200 [17]. On the other hand, we demonstrate how IR commands can be alternatively received by smartphones (Fig. 11), thus allowing implementation of many assistive mobile applications. In our proof-of-concept solution we used off the shelf PilotMini device. This is voice enabled home automation control device targeted to assistive technology sce- narios. It comes with WiFi IRTrans infrared transmitter, Bluetooth speaker and mi- crophone included (Fig. 12). IR codes are recorded beforehand and mapped in PilotMini configuration with re- spective voice commands. After that FLIRC was configured to receive and recognize recorded IR codes and map them to keyboard keys. On the receiving smartphone side, we have developed a proof-of-concept Android application that takes key presses emulated by FLIRC and can make phone calls ei- ther to preprogrammed phone numbers (i.e. from phonebook) or custom numbers entered using IR signals. The potential to expand function set of this setup is limited only by smartphone features exposable to mobile applications. 78 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources Fig. 11. Smartphone as an IR receiver providing support in assistive technology scenario. Any type of IR-enabled remote control can be used (manually activated, voice activated). Fig. 12. PilotMini voice enabled control device, FLIRC IR receiver and smartphone with proof-of-concept Android app. In the real setup the distance between PilotMini and smartphone was approx. 6m. 6 Conclusion In this paper we presented alternative interaction techniques which do not require direct contact with the target mobile device. Instead, they rely on changes being made in the surrounding space that can be detected by the device’s sensors and subsequent- ly used as corresponding input commands. Specifically, possibilities of utilizing mag- netic, lighting, and infrared sources were investigated. It was successfully demon- strated how interaction space around the target device can be augmented by making use of intentional ambient manipulations. The range of increased interaction space iJIM ‒ Vol. 11, No. 4, 2017 79 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources depends on the used source. Changes in magnetic field were successfully sensed by the smartphone device within a 30 cm distance, flashes of light were effectively de- tected within a 2 m range, and IR signals were received from any spot in a 6 m ! 6 m room. Proof-of-concept applications were developed for Android-based mobile devices, demonstrating real-life scenarios in which sensor-based interaction could become convenient. The related solutions do not require alterations in device hardware or sophisticated peripherals. Conversely, magnetic field and light sensors were used, which are the integral part of contemporary mobile devices. Regarding IR support, we used inexpensive and easy to use product in order to enable IR receiving on the smartphone device. In the presented applications, external sources of magnetic field, light, and IR sig- nals are becoming main “tools” in interaction with the device. As demonstrated, input commands can thus be issued by permanent magnet repositioning, light flashing, and IR code transmitting. While any standard IR remote control can demonstrate useful- ness in the respective context, we opted for assistive technology scenario in which device able to receive voice instructions and transmit corresponding IR codes was used. Our proof-of-concept solutions do not require computationally demanding and time-consuming gesture learning/training. Magnet-based interaction is based on sim- ple magnetic field fingerprinting, requiring only once-per-setup calibration procedure. User is required to move permanent magnet among different zones within a surface representing virtual command panel. In this respect, the possibility of involving zone fingerprinting in 3D could be furthermore investigated. Light-based interaction relies solely on detecting previously configured illumination thresholds. User can deliber- ately trigger such detection by making use of external light source (such as flashlight). On the other hand, light-based input commands can be invoked automatically, as the result of both slow and sudden changes in ambient light (e.g. monitoring daylight illumination trough time, or detecting unexpected power outages, respectively). Final- ly, IR-based interaction is the most straightforward, since it only involves code trans- mitting via some IR-enabled device. Proper configuration, i.e. mapping IR codes to corresponding input commands, is additionally required. A family of interaction techniques presented in this paper extend the input lan- guage by making use of aforementioned sensors only. They all showed to be reliable and appropriately accurate for given contexts. Since high sampling rates can be used for magnetic field and ambient light sensors, issues related to power consumption could be furthermore explored. Specifically, the effect of sensor sampling rate on device battery drain could be examined for each proof-of-concept application. 7 References [1] D. Fitton et al., “Exploring tilt-based text input for mobile devices with teenagers,” In: Proc. Int. BCS Human Computer Interaction Conf. (BCS-HCI’13), 2013, pp. 1-6. [2] S. Ljubic, “Three Text Entry Methods Based on Smartphone Discrete Tilting: An Empiri- cal Evaluation,” In: Proc. Universal Access in Human-Computer Interaction 80 http://www.i-jim.org Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources (UAHCI/HCII’16), LNCS 9738, 2016, pp. 141-152. https://doi.org/10.1007/978-3-319- 40244-4_14 [3] Google ATAP. Project Soli [Online]. Available: https://atap.google.com/soli/ [4] P. Schmieder et al., “Thumbs Up: 3D Gesture Input on Mobile Phones Using the Front Facing Camera,” In: Proc. INTERACT’13, LNCS 8118, 2013, pp. 318-336. https://doi.org/10.1007/978-3-642-40480-1_20 [5] J. Wang et al., “Ubiquitous Keyboard for Small Mobile Devices: Harnessing Multipath Fading for Fine-Grained Keystroke Localization,” In: Proc. Int. Conf. Mobile systems, ap- plications, and services (MobiSys’14), 2014, pp. 14-27. https://doi.org/10.1145/ 2594368.2594384 [6] C. Harrison and S. E. Hudson, “Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices,” In: Proc. Ann. ACM Symp. User Interface Software and Technology (UIST’09), 2009, pp. 121-124. https://doi.org/10.1145/ 1622176.1622199 [7] H. Ketabdar, M. Roshandel, and K. A. Yuksel, “Towards Using Embedded Magnetic Field Sensor for Around Mobile Device 3D Interaction,” In: Proc. Int. Conf. Human Computer Interaction with Mobile Devices and Services (MobileHCI’10), 2010, pp. 153-156. https://doi.org/10.1145/1851600.1851626 [8] H. Ketabdar, M. Roshandel, and K. A. Yuksel, “MagiWrite: Towards Touchless Digit En- try Using 3D Space Around Mobile Devices,” In: Proc. Int. Conf. Human Computer Inter- action with Mobile Devices and Services (MobileHCI’10), 2010, pp. 443-446. https://doi.org/10.1145/1851600.1851701 [9] S. Hwang et al., “MagPen: Magnetically Driven Pen Interactions on and Around Conven- tional Smartphones,” In: Proc. Int. Conf. Human Computer Interaction with Mobile Devic- es and Services (MobileHCI’13), 2013, pp. 412-415. https://doi.org/10.1145/2493 190.2493194 [10] S. H. Yoon, K. Huo, and K. Ramani, “TMotion: Embedded 3D Mobile Input Using Mag- netic Sensing Technique,” In: Adjunct Proc. Ann. ACM Symp. User Interface Software and Technology (UIST’15 Adjunct), 2015, pp. 71-72. https://doi.org/10.1145/ 2815585.2815723 [11] H. Ketabdar et al., “MagiGuitar: A Guitar That is Played in Air!,” In: Proc. Int. Conf. Hu- man Computer Interaction with Mobile Devices and Services (MobileHCI’12), 2012, pp. 181-184. https://doi.org/10.1145/2371664.2371704 [12] R. Spreitzer, “PIN Skimming: Exploiting the Ambient-Light Sensor in Mobile Devices,” In: Proc. ACM Workshop on Security and Privacy in Smartphones and Mobile Devices (SPSM’14), 2014, pp. 51-62. https://doi.org/10.1145/2666620.2666622 [13] C. Zhang et al., “Extending Mobile Interaction Through Near-Field Visible Light Sens- ing,” In: Proc. Ann. Int. Conf. Mobile Computing and Networking (MobCom’15), 2015, pp. 345-357. [14] S. Kratz and M. Rohs, “HoverFlow: Exploring Around-Device Interaction with IR Dis- tance Sensors,” In: Proc. Int. Conf. Human Computer Interaction with Mobile Devices and Services (MobileHCI’09), 2009, pp. 1-4. https://doi.org/10.1145/1613858.1613912 [15] FLIRC [Online]. Available: https://flirc.tv/ [16] A. J. C. Moreira, R. T. Valadas, and A. M. de Oliveira Duarte, “Performance of infrared transmission systems under ambient light interference,” IEE Proceedings - Optoelectron- ics, vol. 143(6), pp. 339-346, December 1996. https://doi.org/10.1049/ip-opt:19960696 [17] GewaTel 200 [Online]. Available: http://www.abilia.com/en/product/gewatel-200-uk iJIM ‒ Vol. 11, No. 4, 2017 81 Paper—Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources 8 Authors Sandi Ljubic currently holds a postdoctoral position at the Department of Com- puter Engineering (University of Rijeka, Faculty of Engineering, Vukovarska 58, 51000 Rijeka, Croatia). His research interests span a range of topics in the HCI field, including mobile interaction, predictive modeling and evaluation, universal access and text entry methods. Damir Arbula currently holds a postdoctoral position at the Department of Com- puter Engineering (University of Rijeka, Faculty of Engineering, Vukovarska 58, 51000 Rijeka, Croatia). His main research interests are in the field of distributed algo- rithms and their application to ad-hoc and wireless sensor networks. Article submitted 01 February 2017. Published as resubmitted by the authors 13 April 2017. 82 http://www.i-jim.org iJIM – Vol. 11, No. 4, 2017 Contact-free Interaction with Mobile Devices using Magnetic, Lighting and Infrared Sources