ArticlePDF Available

# UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces

Authors:

## Abstract and Figures

We present UbiquiTouch, an ultra low power wireless touch interface. With an average power consumption of 30.91μW, UbiquiTouch can run on energy harvested from ambient light. It achieves this performance through low power touch sensing and passive communication to a nearby smartphone using ambient FM backscatter. This approach allows UbiquiTouch to be deployed in mobile situations both in indoor and outdoor locations, without the need for any additional infrastructure for operation. To demonstrate the potential of this technology, we evaluate it in several different and realistic scenarios. Finally, we address the future application space for this technology.
Content may be subject to copyright.
27
UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces
ANANDGHAN WAGHMARE, Georgia Institute of Technology, USA
QIUYUE XUE, Georgia Institute of Technology, USA
DINGTIAN ZHANG, Georgia Institute of Technology, USA
YUHUI ZHAO, Georgia Institute of Technology, USA
SHIVAN MITTAL, Georgia Institute of Technology, USA
NIVEDITA ARORA, Georgia Institute of Technology, USA
CEARA BYRNE, Georgia Institute of Technology, USA
THAD STARNER, Georgia Institute of Technology, USA
GREGORY D ABOWD, Georgia Institute of Technology, USA
We present UbiquiTouch, an ultra low power wireless touch interface. With an average power consumption of 30.91
µ
W,
UbiquiTouch can run on energy harvested from ambient light. It achieves this performance through low power touch sensing
and passive communication to a nearby smartphone using ambient FM backscatter. This approach allows UbiquiTouch to be
deployed in mobile situations both in indoor and outdoor locations, without the need for any additional infrastructure for
operation. To demonstrate the potential of this technology, we evaluate it in several dierent and realistic scenarios. Finally,
we address the future application space for this technology.
CCS Concepts:
Human computer interaction (HCI) Interaction devices
;
Human-centered computing
Ubiquitous and mobile computing.
Additional Key Words and Phrases: Low power computing, Backscatter communication, Touch interaction, Power harvesting
ACM Reference Format:
Anandghan Waghmare, Qiuyue Xue, Dingtian Zhang, Yuhui Zhao, Shivan Mittal, Nivedita Arora, Ceara Byrne, Thad Starner,
and Gregory D Abowd. 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces. Proc. ACM Interact. Mob. Wearable
Ubiquitous Technol. 4, 1, Article 27 (March 2020), 22 pages. https://doi.org/10.1145/3380989
1 INTRODUCTION
Touch interaction is a fundamental interaction technique for computing interfaces. However, touch interaction is
currently limited to devices such as phones, laptops, smartwatches, etc. Extending touch sensing from devices to
objects and surfaces in everyday life can enhance them with interactivity and improve day-to-day interactions.
The research community has explored a number of ways to extend the interaction space using acoustics [
12
,
48
],
RF sensing [
7
,
19
], and capacitive sensing [
11
,
45
]. Most of these solutions either require dedicated infrastructure
Authors’ addresses: Anandghan Waghmare, Georgia Institute of Technology, Atlanta, USA, anandghan@gatech.edu; Qiuyue Xue, Georgia
Institute of Technology, Atlanta, USA; Dingtian Zhang, Georgia Institute of Technology, Atlanta, USA; Yuhui Zhao, Georgia Institute of
Technology, Atlanta, USA; Shivan Mittal, Georgia Institute of Technology, Atlanta, USA; Nivedita Arora, Georgia Institute of Technology,
Atlanta, USA; Ceara Byrne, Georgia Institute of Technology, Atlanta, USA; Thad Starner, Georgia Institute of Technology, Atlanta, USA;
Gregory D Abowd, Georgia Institute of Technology, Atlanta, USA, abowd@gatech.edu.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst
page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy
otherwise, or republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a fee. Request permissions from
permissions@acm.org.
2474-9567/2020/3-ART27 $15.00 https://doi.org/10.1145/3380989 Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:2 Waghmare et al. or battery-powered hardware to sense and communicate back touch input to a computing entity that can respond appropriately. The logistics of providing power and a communication path is what limits the surfaces that can support touch interaction. One way to address these challenges is to develop self-sustaining approaches that power the sensing and communication needed to support touch input. Previous battery-free approaches have used RFID [ 20 , 34 , 49 ] or other wireless powering methods [ 35 , 43 ] but require additional infrastructure to be present in the environment (e.g., an RFID reader). Power harvesting solutions, such as solar cells, do not provide enough continuous power in many indoor articial light settings to support the conventional sensing and communication channels (e.g. Bluetooth Low Energy (BLE) [ 22 ]). To address this, we developed UbiquiTouch (Figure 1), a low power sensing solution that can detect touch input and then communicate that input wirelessly to a nearby FM enabled smartphone by backscattering ambient FM radio waves. This approach works without the need for any additional custom infrastructure in the environment. It consumes 31 µ W of power on average, enabling it to work both indoors and outdoors with most commodity solar cells. In this paper, we discuss the relevant related research that inspires and informs UbiquiTouch. We provide an overview of our technical approach, with detailed descriptions for the implementation of the touch sensing, encoding of the touch event in a form that can be wirelessly communicated, and the use of ambient FM backscatter to communicate passively to a nearby smartphone. We describe the results of evaluations to explain how the prototype works in practice. We then explore the potential application space for UbiquiTouch. Fig. 1. UbiquiTouch System Contributions System design for ultra-low power wireless touch interface : UbiquiTouch demonstrates an end-to- end pipeline for building self-sustainable interactive touch surfaces. It consists of a touchpad whose layout is dictated by the target application, a custom ultra-low-power circuit to sense and transmit the touch location leveraging ambient FM radio waves, and a software program that decodes the transmitted data in real-time. Feasibility study in dierent settings : We conduct an evaluation of our system with 20 participants in both indoor and outdoor locations to demonstrate that UbiquiTouch can be used in practical application scenarios. We also discuss in detail how dierent parameters aect the system’s performance and their trade-os. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:3 Exploration of interesting usage contexts : UbiquiTouch opens new possibilities for interaction on everyday objects and surfaces. We demonstrate the potential of UbiquiTouch in setting where a self- sustainable touch input would be relevant, e.g. input on clothing while using AR/VR, touch input on paper for interactive public poster, etc. 2 RELATED WORK The choice of touch sensing interface is driven by its target application with trade-os between performance (e.g. reliability, latency, precision), scalability (e.g. single-point, multi-touch and multiple device detection), ease of deployment, and power consumption. Our related work section intersects with three key research areas. First, we discuss touch sensing methods for everyday objects and surfaces with focus on functionality, power consumption and scalability. Next, we look at system design trends in the domain of self-sustainable sensing and communication. Lastly, we combine both the themes together and critically look at the current eld of the self-sustainable touch sensing systems for everyday objects, analysing it carefully for functionality, instrumentation overhead and performance trade-os. 2.1 Touch Sensing over Everyday Objects and Surfaces A spectrum of techniques has been explored in the last decade to achieve reliable touch detection on surfaces of dierent sizes and geometry. Light sensing techniques using optical ber [ 47 ], laser range nder [ 3 ] and depth-sensing camera [ 24 ] have been employed to determine grasp, touch, and movement. Large surfaces like tables or walls augmented with glass can be made touch-sensitive using FTIR sensing technique [ 1 ]. Even though these light-based methods allow for granular detection of touch interactions, they are extremely power-intensive and require expensive instrumentation of the infrastructure. Another way to detect touch events on everyday surfaces is based on sound, by carefully placing microphones on or near the surface of the object [ 12 , 17 , 18 , 32 ]. Wimmer et al. [ 48 ] applied time domain reectometry to curved surfaces in order to enable touch interactions on them. These auditory methods are cheaper than light-based techniques but are still quite power intensive. They require a microcontroller which consumes mW-W of power for data communication, processing or storage. The same limitation is true for touch detection solutions that have resistive [ 13 ], capacitative [ 28 , 33 , 37 ] or piezoelectric [ 29 ] sensors embedded in the object itself. Recently, inexpensive do-it-yourself techniques [ 51 53 ] have been demonstrated in the literature for augmenting any surface with touch sensing capability. Even though these solutions have a low power requirement and are thus versatile for touch sensing, they are still far from a practical solution where everyday surfaces can be interactive. There is a need to consider system architecture level along with an application mindset to build a practical, interactive touch detection solution that is optimised for power and functionality. 2.2 Self-sustainable Sensing and Communication Systems Improvements in the eciency of power harvesting (e.g. solar [ 10 ], wireless power transfer) with a simultaneous decrease in the power of operation [ 16 , 23 , 31 , 41 ] has resulted in making self-sustainable systems a reality. One of the earliest systems, Wireless Identication and Sensing Platform (WISP), augmented RFID tags with sensors so that the tag itself can send sensed data(e.g. camera, audio, accelerometer) to nearby reader [ 36 , 49 ]. WISP works by rst digitizing the sensed data and then using a state machine to perform data computation or communication using low-power micro-controller like MSP 430. WISP consumes < 10mW of power, thus lasting for a considerable amount of time when duty-cycling is applied. A recent wave of battery-free sensing systems optimizes power even further by shifting the digital tasks to the receiver and keeping the tag as a purely analogue system with cleverly designed encoding schemes to communicate the information reliably to a receiver. Such systems consume power on the order of 100 mW [ 26 , 35 , 43 ]. To save the infrastructure cost for the RF Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:4 Waghmare et al. transmitters and receiver, another trend in the self-sustainable communication domain is to nd clever ways of using existing infrastructure to transmit data. Cohn et al. [ 4 ] utilized powerline coupling to communicate sensed data in a home setting. One recent trend has been to utilize readily available radio frequency waves like FM [ 44 ], WiFi [ 2 , 50 ] or TV signal [ 21 , 30 ] for ambient backscatter communication. This makes the system more practical to use and deploy. We leverage these two technology trends to build UbiquiTouch, an ultra-low-power touch sensing system. 2.3 Self-sustainable Touch Sensing Power is a major bottleneck in scaling touch interactions to everyday objects. There has been other previous work that enables self-sustainable touch and gestures detection based on inexpensive passive RFID tags to facilitate new-style interaction interfaces [ 20 ]. PaperID [ 19 ] and RapID [ 40 ] demonstrated RFID-based touch interaction possibilities for objects by detecting change of the backscattered signal. RIO [ 34 ] recently improved its performance detecting subtle changes in antenna impedance when a human nger touches the surface of an RFID tag. Live Tag [ 7 ] used thin, exible chipless RFID with commodity WiFi to eliminate the need for using expensive transceivers. ZeroPowerTouch [ 25 , 42 ] demonstrates a receiver design that operates only from the energy harvested from the received message to work as a touch sensor. All of these systems rely on dedicated RF equipment to be present for their operation, restricting availability to a certain location and contributing to deployment costs. UbiquiTouch overcomes this requirement by using existing resources present in the ambient environment for operation. Inspired by the technology trends and lack of a practical self-sustainable interactive touch detection system in the existing literature, we propose UbiquiTouch as a low-power system for real-time touch detection without the need for signicant infrastructure to be deployed. 3 TECHNOLOGY 3.1 Overview The UbiquiTouch system consists of a touchpad, a custom ultra-low-power circuit to sense and transmit the touched location, and a software program to decode the transmitted data. The touchpad consists of a 2D grid of single touchpoints that causes a voltage potential change when touched. The detected location on the touchpad is encoded as a binary data packet and transmitted by backscattering ambient FM radio waves. This transmission is received as an audio signal by a commodity smartphone with a built-in FM receiver. The audio is then processed to identify and decode the data packets. 3.2 Touch Sensing Principle The sensing technique is based on the mild conductivity of human skin which allows for electrical signals to propagate through it. The resistance of human skin is generally between 1-100 k and depends on factors like skin moisture and body temperature [ 6 ]. When skin simultaneously touches two electrodes at dierent voltage potentials, it creates an electrical path between them, causing a ow of current between the electrodes. The presence or absence of this current can be used to detect human touch. In our setup, we use two electrodes, one as a positive potential and another one as a voltage sensing electrode. When skin simultaneously contacts these two electrodes, it induces a current ow between them, resulting in voltage change at the sensing electrode, which is measured to sense touch. One such electrode pair makes a single touchpoint. We create a 2D arrangement of such electrode pairs or touchpoints for our system. Figure 2 shows the sensing principle. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:5 3.3 Touch Pad Design Our touchpad design consists of rows and columns which act as sensing electrodes. Both rows and columns are exposed as square-shaped conductive pads and are laid out, as shown in 3. The rest of the medium on the touchpad is an electrode maintained at a positive voltage. This layout tightly packs the touchpoints and evenly distributes both rows and columns. Fig. 2. Sensing principle Fig. 3. Touchpad layout In this design, the intersection of a row and column is one touchpoint on the touchpad. For a user’s touch to be correctly detected, the nger must touch both the column and row at the contact point and induce a positive voltage in it. For this to happen, the dimensions of the row and column should be such that an average human ngertip should be able to touch at least one row and column pad. Furthermore, the voltage on the positive potential electrode should be suciently high to produce a positive voltage for dierent user’s skin resistances on the row and column electrodes. To determine the best physical dimensions of the row and columns pads, we performed a pilot study among the researchers in which we tested several combinations of electrode pairs with dierent surface areas and distances between them. All combinations were repeated for dierent values of the positive voltage. 4(a) shows the evaluation board used in this investigation made with tin-coated copper electrodes. From our analysis, we found that increasing the surface area of electrodes while decreasing the distance between electrodes increases the probability of touch detection. Based on these results, we designed our touchpad with a pad size of 2.5mm x 2.5 mm and maintained the positive voltage at 4 V. Figure 4(b) displays the touchpad with four rows and four columns. 3.4 Touch Location Encoding Each touchpoint on the touchpad is represented by a unique binary address combining the address of the touched row and column. The rows and columns are addressed independently by two separate priority binary encoders where the rows and column lines are inputs to the two encoders. The choice of binary encoders for addressing helps to save power as binary encoders can read all inputs simultaneously and avoids the need to poll individual inputs. Polling inputs one-by-by would require more circuitry which will consume more power. A binary encoder takes 2 N bits of input and encodes it into an N-bit output. Since our touchpad prototype has four rows and columns, we use two 2-bit encoders, the output combination of which is a 4-bit location that has been touched. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:6 Waghmare et al. (a) Touchpoint evaluation board (b) 4x4 Touchpad Fig. 4. Touchpad PCBs This approach has the limitation that binary encoders can only read one input at a time—implying that if multiple rows/columns are touched simultaneously, the encoders will output a wrong result. This problem is solved by using a priority encoder which assigns priority to inputs and ensures that if multiple rows/columns are touched simultaneously, only the one with the highest priority is recorded. This, in terms of the design, means that the rows and columns pads can be made smaller and tightly packed to provide a higher touch resolution. 3.5 Communication 3.6 Overview Communication is one of the most power-intensive operations in wireless devices, especially for low-power devices. This higher power requirement is due to the need to generate high-frequency RF signals that consume several watts of power. To save energy on communication, we use the technique of backscatter communication. Backscatter communication eliminates the need to produce high-frequency RF signals by modulating and reecting existing RF radiation. A typical bistatic backscatter system consists of a backscatter node, a continuous wave emitter, and a receiver. The backscatter node receives a carrier signal from the emitter and reects a part of it to the receiver. By modulating the reection of the incoming RF signals, the backscatter node embeds its information on the existing signal without generating the carrier wave itself. For UbiquiTouch, we use ambient FM signal as the signal source instead of a dedicated transmitter, as most locations have FM radio stations which are widely accessible and transmit continuously. Wang et al. have shown that this ambient FM signal is strong enough for supporting backscatter communication [ 44 ]. On the receiving side, the backscattered signal is captured by a commercial mobile phone with a built-in FM receiver. By using the receiver and transmitter as existing resources in the environment, UbiquiTouch runs without requiring any dedicated infrastructure. This is critical for supporting the goal of ubiquitous availability of the passive communication channel. 3.7 Transmission The sensed location on the touchpad is transmitted wirelessly as a binary data packet to a nearby FM receiver by backscattering information on ambient FM radio waves. Since the backscatter communication link is weak due to the low signal to noise ratio of the backscattered signals, we take two measures for improving the communication quality. First, until a point is touched on the touchpad, the touchpad hardware continuously re-transmits the packet corresponding to the touched point. If a packet is corrupted in transmission, the next packet could be used Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:7 for decoding location. Second, in each transmission, we create redundancy by including the detected location twice in the packet for error checking at the receiver. By simply including a second copy of the data in the packet, we avoid generating error-correcting codes and save power. Each data packet starts with a 7-bit header (1010111) which helps the receiver to nd the start of the data packet. The header is followed by two copies of the 4-bit binary address of the location. Fig 5shows a data packet. Fig. 5. Data packet structure Fig. 6. Headphone Jack We encode our binary data on the FM wave using an On-O Keying (OOK) modulation scheme. In this scheme, binary one is represented by the presence of a backscatter signal while its absence represents a binary zero. A backscatter signal is generated by modulating the radar cross-section of an antenna in the presence of a carrier wave. We perform binary modulation in our prototype and hence switch our antenna between two discrete impedance states. The rate of this switching also determines the frequency of the produced backscatter signal. If the backscatter antenna is switched at frequency fba ck to backscatter an ambient FM signal centered at frequency fc , then the backscattered signal generated will be at a frequency of fc+fba ck [ 44 ]. This implies that the FM receiver will need to tune at the center frequency of fc+fbac k to receive the backscattered signal. 3.8 Receiver The transmitted data is received by a nearby smartphone with a built-in FM receiver. Most smartphones have an FM radio integrated into their hardware [ 8 ], which makes them a commonly available receiver in the environment. The phone’s FM radio does not provide access to the raw RF signal but instead decodes it and provides it as an audio stream. We process this audio stream in software to detect and decode data packets. FM receivers in smartphones generally do not have an internal FM antenna, requiring the users to plug in their headphones to receive the FM signal. We do not want to require the use of headphones for UbiquiTouch to work, so we designed our system to work with only a small headphone jack inserted into the phone without any long headphone wire connected to it. This small jack, as shown in gure 6, can be used continuously without hindering regular use of the phone. Figure 7shows the system overview. 3.9 Power Management UbiquiTouch touchpad hardware is self-sustaining and works only using the energy harvested from ambient light through a photovoltaic cell. It achieves this through its low power circuitry and passive communication technique. To optimize usage of the harvested energy, the hardware has a sleep mode. When the touchpad is not Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:8 Waghmare et al. Fig. 7. System Overview being used, the electronics components for data encoding and transmission are switched o. Human touch is then used as a trigger to turn the complete system back on. 4 SYSTEM IMPLEMENTATION 4.1 Hardware The raw input to the touchpad hardware is the four rows and four columns which develop a positive voltage when touched by the user. This developed voltage diers across users and based on our test analysis when the positive electrode is at 4 V; this voltage is above 200 mV for all users. Because the rows/columns are inputs to a priority encoder which operates at voltage levels consistent with digital logic, we use voltage comparators [ 14 ] to convert the changing analogue input into a sharply dened binary output. We analyze the voltage on the rows/columns for touch or no touch scenario by comparing it to a reference voltage of 150mV. This reference voltage value also helps to lter out false positives which may occur because of the voltage induced in the rows/columns due to ambient AC line noise or other RF disturbances. The four rows and columns are each connected to the inputs of a 2-bit priority binary encoder dened by the following Boolean equations and represented in Table 1. O1=I1+I2(1) O0=¬(I2+¬I1)+I3(2) Table 1. Truth table for 2-bit priority binary encoder (X = Don’t care) Input 3 I3 Input 2 I2 Input 1 I1 Input 0 I0 Output 1 O1 Output 0 O0 Valid V 0 0 0 0 X X 0 0 0 0 1 0 0 1 0 0 1 X 0 1 1 0 1 X X 1 0 1 1 X X X 1 1 1 The boolean OR operations are implemented by combining the OR inputs through diodes (1N14148) to a common cathode as output, and the boolean AND operation was performed using the voltage comparator [ 14 ] Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:9 in an inverting conguration. The two priority encoders output 2 bits each, representing the row and column touched by the user. To serialize these four bits for wireless transmission, we use a parallel-load serial-out shift register [ 27 ]. We daisy chain two 8-bit shift registers where the rst 7 bits have a hardcoded header (1010111) and the next 8 bits are the 4-bit encoder output repeated twice. The shift register serially outputs one bit per clock cycle and is fed by an external clock provided by a CSS555 timer running at 100 Hz [ 39 ]. The inputs of the shift registers are loaded about ve times per seconds as determined by another CSS555 timer, which sets the refresh rate of the touchpad as 5 Hz. The serial output of the shift register is the input to the backscatter communication part of the circuitry. The backscatter portion consists of a square wave oscillator (TS3002) and an RF switch [ 5 ]. The RF IC is used to switch the impedance of a dipole antenna used in our system in two dierence impedance states. The RF switch is fed by the output of the oscillator. Hence, the output frequency of the oscillator determines the switching frequency of the RF switch; in other words, the frequency fba ck for our backscatter communication. The oscillator runs at 300kHz so fbac k is 300kHz. The oscillator output is controlled by the serial output of the shift register and is enabled for a binary one or disabled for a binary zero. The rate at which the shift registers outputs bits denes the rate at which the bits are transmitted, hence the shift register clock sets the transmission rate, and for our system, it is 100 bits per second. All of the circuitry is powered by a photovoltaic cell (Panasonic - BSG AM-5412CAR) of dimensions 50mm x 33mm and the harvested energy is managed by Texas Instrument’s BQ25570 chip. 4.2 Soware We built a real-time software pipeline to decode the data received from the touchpad hardware by the FM receiver. In our setup, we use a Motorola One smartphone with a built-in FM receiver chip as our receiver and use the default FM radio app on the phone to tune to the transmit frequency fc+fba ck . The phone outputs the received signal as audio data which we stream over Bluetooth to a Macbook Pro laptop for processing. Processing data on the laptop is done for ease of debugging, but this processing can also be done locally on the phone itself. The raw audio received from the phone is stored in a buer and processed using a sliding window approach in a Python program. Each sliding window is of length 7 bit and slides with a 50% overlap. For every window, the data is rst passed through a Random Forest machine learning classier to detect any active transmission from the touchpad. The classier is trained using audio data recorded on the phone when there was an active transmission of random bits and while there is no transmission. The signal recorded during active transmission has a distinct frequency response compared to static noise recorded during no transmission. Therefore, we use features in the frequency domain (spectral spread and spectral centroid) while training the classier. The trained classier demonstrated 99.7% 5-fold cross-validation accuracy. On detecting an active transmission in a sliding window, the window is searched for the start of the packet. This is done by passing the window through a bandpass lter to have frequencies between 15000-18000 Hz and then cross-correlating the known pattern of the header (1010111) with the sliding window. The position of the max value in this cross-correlation result indicates the start of the packet. After knowing the beginning of the packet, the location of all the bits can be calculated using the known length of each bit. The next step in this process is to decode the payload. To do this, the individual energies of all the bits in the payload are calculated. For a single bit, energy is calculated by summing the magnitude of all the values representing that bit in the raw signal. Then the bit energies are compared to the bit energy of the known ones and zeros in the header to identify them. The decoded packet is then checked for bit errors by comparing the redundant bits in the packet. Finally, a packet without any bit error is accepted. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:10 Waghmare et al. 5 EVALUATION 5.1 Overview Our evaluation measures the overall system performance and assesses the factors which aect performance. We quantify system performance in terms of three metrics: (1) Accuracy of touch detection as the percentage of time a touch interaction occurred and was correctly registered by the system. (2) Response time as the time between a touch interaction at the touchpad and the system correctly registering it the rst time. (3) Power consumed by the UbiquiTouch hardware. During this evaluation, we also collect metrics on factors aecting the system performance. 5.2 Factors Aecting System Performance The overall touch detection chain consists of three parts—touch detection by the touchpad hardware, transmission of that information over the wireless link, and decoding it at the receiver/phone. Each of these can aect overall system performance. 1) Touchpad hardware: The power consumed by the sensing circuitry can dier across users as the sensing works by passing a small current through the skin of the user, and the skin resistance can vary across users which aect the current. 2) Communication channel: Our system communicates by backscattering ambient FM radio waves, which is a weak communication link due to the low signal strength of the backscattered signal. It is also aected by the varying signal strength of the ambient FM signal at dierent locations and hence can introduce inaccuracies in the system. 3) Receiver: The receiver relies on a machine learning classier to detect an active transmission and a de- coding algorithm to locate and decipher the packet. The individual performance of these can aect the overall performance. 5.3 Evaluation Procedure To evaluate the system in common real-world situations, we evaluated the system in two scenarios—when the user is holding the phone (receiver) in their hand and while the phone is in the user’s pocket. We replicate these two scenarios in both outdoor and indoor settings. The outdoor sessions were conducted on a patio during daylight hours with the light intensity ranging from 200 lux to 5000 lux. The indoor sessions were held in a uorescent lighting oce environment with a light intensity of about 300 lux. In all sessions, the touchpad was kept in front of the user on a horizontal at surface while the user was interacting. The user’s initial task was to touchpoints on the touchpad, visually indicated to the user, one by one for a total of 16 points in every setting. The point to be touched was generated randomly from a uniform distribution and was projected by a portable projector onto the touchpad. The projector was operated by a laptop computer, and the projected target was a circle of radius 5 mm (Figure 8(b)). The 16 targets were each displayed one by one for 3 seconds, followed by a 2-second break in between. The projection was also accompanied by an audio cue played on the laptop that informed the user to start and stop touching the target. The user study setup is shown in Figure 8(a) During the interactions, the phone logged the detected locations, processing times and other events for post- analysis on the performance of transmission (Packet Error Rate(PER)) and receiver software (Processing time). We also recorded raw audio from the interactions as received by phone. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:11 (a) User Study Setup (b) Projection on Ubiqui- Touch (c) Bot for simulating touch Fig. 8. User Study Setup For measuring response time precisely, we built a bot shown in gure 8(c) to simulate human touch on the touchpad. The bot helped avoid the non-constant reaction time of the user in performing a touch action. We use a slow-motion camera to nd the latency of the bot and included it in our calculations. Although the bot did the touch, we asked the users to hold the phone in their hands while the bot was touching. This enabled the human body’s eect on RF elds to be maintained while receiving a transmission on the phone as in our other experiments. In both indoor and outdoor settings, we ran the bot for 16 points per user. Finally, to measure the current consumption for sensing for each user, we asked them to touch every point on the touchpad one by one and recorded the average current. Other user-independent current consumption such as for encoding, communication, etc. was recorded independently. 6 EVALUATION RESULTS 6.1 Accuracy The average accuracy achieved by the system is over 95% across all scenarios. Table 2shows the highest, lowest and average accuracies in dierent situations. Figure 9indicates the individual accuracies for all participants. Table 2. System accuracy (in percentage) Max accuracy Min accuracy Average accuracy In hand Indoor 99.31 93.81 97.43 Outdoor 99.34 82.75 95.36 In pocket Indoor 99.63 91.67 96.98 Outdoor 99.61 87.86 95.04 On average, for both indoor and outdoor situations, the accuracy when the phone was in hand was slightly higher than when the phone was in the pocket. The potential explanation for this observation is the possible dampening of the backscattered signals while propagating through the user’s clothes, which reduces the signal quality of the received signal on the phone. Despite the high accuracy of the system, even a few errors can lead to unexpected output in end-user applica- tions. To mitigate this, one strategy is to utilize the packets re-transmissions done by the UbiquiTouch system. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:12 Waghmare et al. From our analysis of the user study data, two successive receptions of the same location ensure that the location is touched. Checking for this pattern at the receiver can enable UbiquiTouch system to support accurate touch interactions. (a) Outdoor System Accuracy (b) Indoor System Accuracy Fig. 9. User Study Results 6.2 Response Time The average response time of the system over 20 users is 773.6 ms. The response time of the system can be broadly broken down into multiple sub-parts. Starting at the hardware level, since the refresh rate of the touchpad is 5 Hz there can be a 0-200 ms latency in detecting touch. A subsequent transmission of the sensed location, 15 bits at 100 bits per second, takes 150 ms to send the location. This might take more time if the packet received is incorrect or identiable. User study data analysis shows that it takes up to a maximum of two packets to correctly detect a position. This, therefore, sets the maximum transmission time to 300 ms. On the receiving end of the phone, as we forward the raw data to a laptop, it adds a Bluetooth transmission latency of about 300 ms. Finally, the laptop processes one packet in about 99.8 ms. Table 3lists breakdown of response time and 10 shows the average system response time per user. The response time of the system is higher than the standard touch interfaces used in daily life. This higher than usual response time is a result of a tradeo between speed and energy consumption of the hardware. The speed is sacriced for low energy consumption to open up new possibilities of interaction. The slower response time can be masked by making the touch interactions longer. For example, a single touch to a point can be changed to stretching the point as a line and replacing the single touch with a swipe on the line, thus increasing the time of interaction. In the following application section, we provide design examples based on a similar approach for building applications using UbiquiTouch that can account for the higher response time while providing good user experience. 6.3 Communication ality To measure how wireless communication aects the system performance, we recorded the number of received packets for each user, for both outdoor and indoor, for both the in hand and in pocket scenarios. Table 4shows the Packet Error Rate (PER) in dierent scenarios and Fig. 11 shows the distribution of received packets for each user. From table 4, it can be seen that the PER is higher outdoors compared to the indoor environment, which means more incorrect packets are received in the outdoors. Another nding is that while holding the phone in hand, it Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:13 Table 3. Breakdown of response time. System task Time taken (ms) Detection of touch 0-200 Transmission of touch location 150-300 Transmission of audio data from phone to laptop 300 Processing of a data packet 99.8 Average response time observed in user study = 773.6 Fig. 10. System Response Time. Table 4. Packet Error Ratio (PER) in dierent scenarios. Packet Error Rate (PER) In hand Indoor 15.67 Outdoor 41.61 In pocket Indoor 12.76 Outdoor 48.50 usually receives more packets than when the phone is in the pocket, irrespective of whether outdoors or indoors, as the backscattered signal is attenuated when propagating through clothing. The communication quality was also observed to vary according to environmental conditions. From g. 11 (a) it can be seen that the number of correct packets for user7 to user10 (average number is 116 for in hand and 98 for in pocket) is much lower than the number of correct packets for User 1 to User 6 (average number is 225 for in hand and 192 for in pocket). This is because that we conducted an outdoor user study of User 1 to User 6 on one day, and conducted a study of User 7 to User 10 on another day. For the same reason, Fig. 11 (b) demonstrates that the number of correct packets received for User 11 to User 13 is lower than the number of correct packets received for User 14 to User 20. Although the communication quality diered on dierent days, from Figure 9it can be seen the accuracy of the system remained consistent. 6.4 Power We measured the power consumption of the dierent parts of our circuit while the system was being actively used. Table 5lists the average power measurements. From these measurements, it can be seen that most of the power is consumed by the communication module. Specically, the oscillator used to shift the frequency of the backscattered signal consumes the most power. The power consumption for touch sensing is slightly dierent Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:14 Waghmare et al. (a) Outdoor (b) Indoor Fig. 11. Number of Packets received. for dierent users because of the variance in skin resistance across users. Figure 12 shows the user-dependent sensing power consumption for each participant. Table 5. Power consumption by dierent circuit components Circuit component Average power consumed in active mode (µW) Encoding 5.4 Sensing - user independent 3 Sensing - user dependent 5.76 Communication 12.75 Power Management 4 Total = 30.91µW We tested the system under dierent light conditions to see the minimum amount of light required for our system to work in realtime. We found that the system required a minimum of 200 Lux of light to work reliably with the current photocell. With this lighting requirement, according to the IES Lighting Handbook [ 38 ], our system would work reliably in indoor environments like homes, oces, restrooms, libraries and also outside in daylight. Fig. 12. Touch sensing current consumption for each participant. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:15 7 EXAMPLE APPLICATIONS We envision UbiquiTouch to be used in everyday life, adding self-sustainable interactivity to everyday surfaces. The following applications concepts highlight how UbiquiTouch can be used in the future. 7.1 Input on Clothing Input on clothing provides a medium to interact with wearable devices such as AR/VR headsets. Current solutions, such as Google’s Jacquard [ 9 ], provide such interaction capabilities but come with the logistical challenge of maintaining batteries in the clothing to support the functionality. This creates the inconvenience of charging batteries and carrying the battery’s extra weight and volume. To solve this problem, a touch interface can be built into clothing using UbiquiTouch, as shown in 13. UbiquiTouch provides a touch interface without the need for a battery and no need for external operating infrastructure allowing it to function in the clothing-related mobile scenarios. Fig. 13. UbiquiTouch touch interface on clothing The touchpad in 13 is made using conductive fabric sewed into the sleeve of a jacket and a conductive thread stitched in the arm as an antenna. The PCB and solar cell could be packaged and put on the clothing in a small waterproof package in the future. This interface could be used for micro-interactions such as swipes (Up, Down, Left, Right) for navigation and long press for selection with AR/VR headsets or other wearables like Bluetooth headsets. 7.2 Touch Input on Paper Touch input on paper opens up new possibilities for interaction. We demonstrate a few scenarios where adding interactivity to everyday paper material enhances the user experience. Posters communicate visually and provide quick access to information. Posters sometimes include a QR code to get related information, but it is often up to the user to nd it. Here, we present a scenario in which users are presented with digital information by interacting with a physical poster. Scenario Figure 14(a), a display of multiple events happening in the university over the week attracts a student’s attention. The student, while going through the list of events, nds an event of interest and would like to be notied about it. To do this, the student swipes the event on the poster, which sends a calendar invitation for the event to the student’s phone. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:16 Waghmare et al. Posters may also be used to receive information from users using UbiquiTouch, in addition to providing information. Scenario 1 A user walking on a street notices an advertisement poster Figure 14(b) about a TV show of interest, asking the users to vote for their favourite characters on the show. The user swipes directly onto a character in the form of a checkmark to vote for the role. The poster communicates the voting information to the TV show servers via the user’s phone. (a) Events poster for receiving event infor- mation on user’s phone (b) Poster to gather user votes (c) Poster for collecting user feedback Fig. 14. Application examples of UbiquiTouch on paper Scenario 2 People on the way out of the clothing section in a mall are presented with a poster 14(c) asking them about their shopping experience. By swiping on the appropriate emoji on the poster, the users can register their feedback. Similar posters can be used to gather other types of feedback, such as on cleanliness in public toilets, quality of food in restaurants, etc., which can help improve the quality of public services. These paper-based interfaces can be put up both indoor/outdoor or at public/private spaces without requiring any additional technical infrastructure or maintenance. Previous work like [ 44 ] used FM backscatter to make posters broadcast information. UbiquiTouch adds onto that functionality by adding real-time input on the posters such that users can selectively receive or provide information. In all the application examples, the interactions were designed as swipes or long-presses which makes them longer to perform. Also, the interaction area for swipe was just one touchpoint in an elongated shape. For example, Figure 14(b) the point was in the shape of a tickmark, in Figure 14(c) the point was presented as an emoji, etc. These design decisions support the interactions to be longer which helps mask the system latency. The longer interaction time also means that the user is in contact with a single touchpoint longer, enabling the system to perform multiple retransmissions per point which add robustness to the communication through redundancy. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:17 8 DISCUSSION 8.1 Dierent Touchpad Design and Fabrication Techniques The layout of rows and columns presented in Figure 15 is one of the several possible designs of the touchpad. The touchpad can be laid out in other patterns or shapes to suit dierent applications. Also, the touchpad can be fabricated in dierent materials with dierent form factors. To illustrate dierent fabrication aspects, we present two touchpad prototypes build with dierent fabrication techniques and materials. (1) First, we present a paper-based interface with the rows and columns printed on with conductive inkjet printing. Figure 15 shows this prototype. To create insulation between rows and columns, we print them on separate sheets of paper and laser-cut holes on the top sheet to expose the conductive lines on the bottom sheet. (a) Fabrication steps in inkjet printed touch interface (b) Flexible form Fig. 15. Inkjet printed 2 layer design for UbiquiTouch touchpad (2) The next prototype is developed with a more DIY approach requiring only basic equipment and materials. This interface is made on a regular sheet of paper with rows/columns created on with copper tape. Insulation is conceived between the rows and columns by separating them with scotch tape. Additionally, as an optional step, the contacts between dierent pieces of copper tape were reinforced by soldering them together. This prototype is shown in Figure 16. Fig. 16. Touchpad using copper tape and non-conductive Scotch tape Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:18 Waghmare et al. Throughout the usage of these two interfaces, the copper tape-based design was found to be more stable with frequent bending and unbending. While the printed touchpad initially worked ne, but with repeatedly touching by the nger, the ink began to come o. So the printed touchpad looks more suitable for limited-time applications. 8.2 Scaling the Touchpad The touchpad can be scaled in terms of touch resolution, surface area or both. The main parameters which aect the touchpad are the number of touchpoints and the size of each touchpoint. The size of a touchpoint is independent of the UbiquiTouch circuitry and is application-driven, and the encoders dene the number of touchpoints, the system can support. For an R-bit row encoder with a C-bit column encoder, the max number of points the system can support is 2 (R+C) . In this section, to shed light on scalability, we contrast between dierent parameters such as the number of touchpoints, touch resolution, etc. when the size of the touchpoints and the number of bits in the encoders vary. The relationship is presented in the table 6. Table 6. Relationship while scaling touchpad Number of touchpoints Touchpad resolution Surface area of touchpad Response time Number of encoder bits increase while size of touchpoint remains same Increases exponentially Remains constant as the resolution is dene by the size of touch points Increases exponentially as number of touch points increase exponentially Increases linearly due to the increased transmission time of extra bits. Size of touchpoints increase while number of touchpoints remain same Remains same Decreases Increases Remains same as the time of transmission remains constant 8.3 Multi Touchpad Environment Some scenarios may require the deployment of multiple touchpads in the same location. If more than one touchpad transmits at the same frequency, it can cause interference in communication between touchpads. To avoid this interference, the transmission frequency fc+fba ck must be selected to be unique for each touchpad, allowing multiple touchpads to co-exist. 8.4 UbiquiTouch as a Communication Platform In this paper, we sense touch input, package it in a binary format and transmit it to a nearby receiver. Nevertheless, digital information can also be fed into the system from other sensors and used as a low-power communication platform. UbiquiTouch can thus be used as a ubiquitous and infrastructure-less communication platform. 9 FUTURE WORK 9.1 Decoding Packets on the Phone Currently, the audio data from the phone is streamed to a laptop computer over Bluetooth for processing. This processing can be done by the phone itself, which will make the system more mobile and reduce the response time by avoiding Bluetooth transmission latency. One of the challenges in achieving this, on Android OS, is that the Android OS does not give third-party applications access to live FM audio streams. A possible solution might be to root the phone to gain system-level permissions and then access the FM radio stream. Another problem may be that the lower computing power on the phone compared to a laptop computer may mean slower data processing speed. Still, the time saved in avoiding Bluetooth latency may compensate for it. Running processing on the phone also means doing computation on a battery constraint environment which would require more ecient implementation of algorithms and platform-based optimizations to save power. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:19 9.2 Alternate Modulation Schemes Dierent modulation schemes can be used to achieve higher bit rates in data transmission to reduce transmission time. With the current binary modulation scheme, the system can represent one bit per symbol - either 1 or 0. However, using schemes such as Quadrature amplitude modulation (QAM) or 4 Frequency shift keying (4-FSK), it is possible to represent two bits per symbol, doubling the transmission rate. 9.3 Antenna Size Antenna size for the FM Range is on the order of feet, which limits the UbiquiTouch touch interfaces to larger objects and surfaces to accommodate for the long antenna. There are two possible approaches to solve this problem - (1) We can exploit the fact that the human body can partially act as an FM antenna [ 15 ]. Thus if we electrically couple our system with the user’s body, it can help with the RF transmissions which would mean requiring a smaller FM antenna. Based on this idea, we propose a design for an interactive keypad using UbiquiTouch. The keypad presented in Figure 17 (a) and (b) contains numbers 0-9 which, when touched, can communicate this information to an FM enable smartphone nearby. The handle on the keypad in Figure 17(b) is designed as an aordance to help couple the user’s body with the system while providing a way to hold the keypad. (2) For FM backscatter, a smaller antenna would mean less eective transmission, which would eectively decrease the range for wireless communication. But in cases where long-range is not required, such as when the phone(FM receiver) is placed next to the UbiquiTouch interfaces, a small antenna can also provide functionality. An initial exploration into this suggests that UbiquiTouch system can work with a monopole antenna as small as a 7cm when the phone is kept in close contact with it. Based on this aspect, we present an interactive storybook concept, as shown in Figure 17(c). Touching on an image in the storybook can present the associated content with it on the phone. In this example, touching on the guitar can play a guitar melody on the phone. (a) Keypad front (b) Keypad back (c) Interactive storybook Fig. 17. Future applications for UbiquiTouch 10 CONCLUSION UbiquiTouch is a touch interface which can sense and wirelessly transmit touch events in a very low power budget. With an average power consumption of 30.91 µ W, UbiquiTouch can harvest the energy required for realtime operation from ambient light even in indoor lighting conditions. We did a feasibility study to evaluate Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:20 Waghmare et al. the UbiquiTouch system with multiple users in indoor and outdoor settings with more than 95% accuracy for touch detection in both scenarios. We later explored the potential applications for this technology - input on clothing or touch input on paper. UbiquiTouch promises to open new interaction opportunities for everyday objects, allowing computation to be truly woven in the fabric of everyday life[46]. ACKNOWLEDGMENTS We thank the reviewers for their helpful feedback on the paper. We would also like to thank Dr. Shwetak Patel for fruitful discussions on the project, Yang Zhang for brainstorming the application space and Spiros Daskalakis for helping with FM backscatter communication. We want to express our gratitude to Libby Lavitt for assisting with voiceover for the project video. Finally, we are grateful to the participants who participated in the user study. REFERENCES [1] Thomas Augsten, Konstantin Kaefer, René Meusel, Caroline Fetzer, Dorian Kanitz, Thomas Sto, Torsten Becker, Christian Holz, and Patrick Baudisch. 2010. Multitoe: high-precision interaction with back-projected oors based on high-resolution multi-touch input. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 209–218. [2] Dinesh Bharadia, Kiran Raj Joshi, Manikanta Kotaru, and Sachin Katti. 2015. Back: High throughput wi backscatter. ACM SIGCOMM Computer Communication Review 45, 4 (2015), 283–296. [3] Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: multi-touch interaction around small devices. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, 201–204. [4] Gabe Cohn, Erich Stuntebeck, Jagdish Pandey, Brian Otis, Gregory D Abowd, and Shwetak N Patel. 2010. SNUPI: sensor nodes utilizing powerline infrastructure. In Proceedings of the 12th ACM international conference on Ubiquitous computing. ACM, 159–168. [5] Analog Devices. 2017. ADG901/ADG902. Retrieved April 1, 2019 from https://www.analog.com/media/en/technical-documentation/data- sheets/ADG901_902.pdf [6] Geddes LA. Fish RM. 2009. Conduction of electrical current to and through the human body: a review. Eplasty (2009). [7] Chuhan Gao, Yilong Li, and Xinyu Zhang. 2018. LiveTag: Sensing Human-Object Interaction through Passive Chipless WiFi Tags. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). USENIX Association, Renton, WA, 533–546. https://www.usenix.org/conference/nsdi18/presentation/gao [8] April Gaser. 2016. YOUR PHONE HAS AN FM CHIP. SO WHY CAN’T YOU LISTEN TO THE RADIO? Retrieved April 1, 2019 from https://www.wired.com/2016/07/phones-fm-chips-radio-smartphone/ [9] Google. 2016. JACQUARD by Google. Retrieved April 1, 2019 from https://atap.google.com/jacquard/ [10] Martin A Green, Yoshihiro Hishikawa, Ewan D Dunlop, Dean H Levi, Jochen Hohl-Ebinger, and Anita WY Ho-Baillie. 2018. Solar cell eciency tables (version 52). Progress in Photovoltaics 26, 7 (2018). [11] Tobias Grosse-Puppendahl, Sebastian Herber, Raphael Wimmer, Frank Englert, Sebastian Beck, Julian von Wilmsdor, Reiner Wichert, and Arjan Kuijper. 2014. Capacitive near-eld communication for ubiquitous interaction and perception. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 231–242. [12] Chris Harrison and Scott E Hudson. 2008. Scratch input: creating large, inexpensive, unpowered and mobile nger input surfaces. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, 205–208. [13] David Holman, Nicholas Fellion, and Roel Vertegaal. 2014. Sensing touch using resistive graphs. In Proceedings of the 2014 conference on Designing interactive systems. ACM, 195–198. [14] Texas Instruments. 2015. TLV3691. Retrieved April 1, 2019 from http://www.ti.com/lit/ds/symlink/tlv3691.pdf [15] Behailu Kibret, Assefa Teshome, and Daniel Lai. 2014. Human body as antenna and its eect on human body communications. Progress In Electromagnetics Research 148 (08 2014), 193–207. https://doi.org/10.2528/PIER14061207 [16] Sunkook Kim, Hyuk-Jun Kwon, Sunghun Lee, Hongshik Shim, Youngtea Chun, Woong Choi, Jinho Kwack, Dongwon Han, MyoungSeop Song, Sungchul Kim, et al . 2011. Low Power Flexible Organic Light-Emitting Diode Display Device. Advanced Materials 23, 31 (2011), 3511–3516. [17] Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In The 31st Annual ACM Symposium on User Interface Software and Technology. ACM, 213–224. [18] Gierad Laput, Eric Brockmeyer, Scott E Hudson, and Chris Harrison. 2015. Acoustruments: Passive, acoustically-driven, interactive controls for handheld devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2161–2170. [19] Hanchuan Li, Eric Brockmeyer, Elizabeth J Carter, Josh Fromm, Scott E Hudson, Shwetak N Patel, and Alanson Sample. 2016. PaperID: A technique for drawing functional battery-free wireless interfaces on paper. In Proceedings of the 2016 CHI Conference on Human Factors Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. UbiquiTouch: Self Sustaining Ubiquitous Touch Interfaces 27:21 in Computing Systems. ACM, 5885–5896. [20] Hanchuan Li, Can Ye, and Alanson P Sample. 2015. IDSense: A human object interaction detection system based on passive UHF RFID. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2555–2564. [21] Vincent Liu, Aaron Parks, Vamsi Talla, Shyamnath Gollakota, David Wetherall, and Joshua R Smith. 2013. Ambient backscatter: wireless communication out of thin air. In ACM SIGCOMM Computer Communication Review, Vol. 43. ACM, 39–50. [22] Elke Mackensen, Matthias Lai, and Thomas M Wendt. 2012. Bluetooth Low Energy (BLE) based wireless sensors. In SENSORS, 2012 IEEE. IEEE, 1–4. [23] Gaurav Mathur, Peter Desnoyers, Paul Chukiu, Deepak Ganesan, and Prashant Shenoy. 2009. Ultra-low power data storage for sensor networks. ACM Transactions on Sensor Networks (TOSN) 5, 4 (2009), 33. [24] Nobuyuki Matsushita and Jun Rekimoto. 1997. HoloWall: designing a nger, hand, body, and object sensitive wall. In Proceedings of the 10th annual ACM symposium on User interface software and technology. ACM, 209–210. [25] Philipp Mayer, Raphael Strebel, and Michele Magno. 2019. ZeroPowerTouch: Zero-Power Smart Receiver for Touch Communication and Sensing in Wearable Applications. In 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 944–947. [26] Saman Naderiparizi, Mehrdad Hessar, Vamsi Talla, Shyamnath Gollakota, and Joshua R Smith. 2018. Towards battery-free { HD } video streaming. In 15th {USENIX}Symposium on Networked Systems Design and Implementation ( {NSDI}18). 233–247. [27] Nexperia. 2016. 74HC597 8-bit shift register. Retrieved April 1, 2019 from https://assets.nexperia.com/documents/data-sheet/74HC_ HCT597.pdf [28] Alex Olwal, Jon Moeller, Greg Priest-Dorman, Thad Starner, and Ben Carroll. 2018. I/O Braid: Scalable Touch-Sensitive Lighted Cords Using Spiraling, Repeating Sensing Textiles and Fiber Optics. In The 31st Annual ACM Symposium on User Interface Software and Technology. ACM, 485–497. [29] Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch & activate: adding interactivity to existing objects using active acoustic sensing. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 31–40. [30] Aaron N Parks, Angli Liu, Shyamnath Gollakota, and Joshua R Smith. 2014. Turbocharging ambient backscatter communication. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 619–630. [31] Massoud Pedram and Jan M Rabaey. 2002. Power aware design methodologies. Springer Science & Business Media. [32] DT Pham, Z Ji, O Peyroutet, M Yang, Z Wang, and M Al-Kutubi. 2006. Localisation of impacts on solid objects using the Wavelet Transform and Maximum Likelihood Estimation. In Intelligent Production Machines and Systems. Elsevier, 541–547. [33] Ivan Poupyrev, Chris Harrison, and Munehiko Sato. 2012. Touché: touch and gesture sensing for the real world. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. ACM, 536–536. [34] Swadhin Pradhan, Eugene Chai, Karthikeyan Sundaresan, Lili Qiu, Mohammad A Khojastepour, and Sampath Rangarajan. 2017. Rio: A pervasive rd-based touch gesture interface. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking. ACM, 261–274. [35] Vaishnavi Ranganathan, Sidhant Gupta, Jonathan Lester, Joshua R Smith, and Desney Tan. 2018. RF Bandaid: A Fully-Analog and Passive Wireless Interface for Wearable Sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 79. [36] Alanson P Sample, Daniel J Yeager, Pauline S Powledge, Alexander V Mamishev, Joshua R Smith, et al. 2008. Design of an RFID-based battery-free programmable sensing platform. IEEE transactions on instrumentation and measurement 57, 11 (2008), 2608. [37] Valkyrie Savage, Xiaohan Zhang, and Björn Hartmann. 2012. Midas: fabricating custom capacitive touch sensors to prototype interactive objects. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 579–588. [38] Illuminating Engineering Society. 2011. IES Handbook. Retrieved Janurary 25, 2020 from https://www.ies.org/product/the-lighting- handbook-10th- edition [39] Custom Silicon Solutions. 2009. CSS555 Timer. Retrieved April 1, 2019 from http://www.customsiliconsolutions.com/downloads/ Revised%20Standard%20products/CSS555_Spec.pdf [40] Andrew Spielberg, Alanson Sample, Scott E Hudson, Jennifer Manko, and James McCann. 2016. RapID: A framework for fabricating low-latency interactive objects with RFID tags. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5897–5908. [41] Thad Starner. 2001. The challenges of wearable computing: Part 1. Ieee Micro 21, 4 (2001), 44–52. [42] Raphael Strebel and Michele Magno. 2018. Zero-power receiver for touch communication and touch sensing. In 2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 150–151. [43] Vamsi Talla, Bryce Kellogg, Shyamnath Gollakota, and Joshua R Smith. 2017. Battery-Free Cellphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 25. [44] Anran Wang, Vikram Iyer, Vamsi Talla, Joshua R Smith, and Shyamnath Gollakota. 2017. FM Backscatter: Enabling Connected Cities and Smart Fabrics.. In 14th USENIX Symposium on Networked Systems Design and Implementation NSDI 17. 243–258. [45] Yuntao Wang, Jianyu Zhou, Hanchuan Li, Tengxiang Zhang, Minxuan Gao, Zhuolin Cheng, Chun Yu, Shwetak Patel, and Yuanchun Shi. 2019. FlexTouch: Enabling Large-Scale Interaction Sensing Beyond Touchscreens Using Flexible and Conductive Materials. Proc. ACM Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. 27:22 Waghmare et al. Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article 109 (Sept. 2019), 20 pages. https://doi.org/10.1145/3351267 [46] Mark Weiser. 1991. The Computer for the 21st Century. Scientic American 265, 3 (1991), 94–105. [47] Raphael Wimmer. 2010. FlyEye: grasp-sensitive surfaces using optical ber. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction. ACM, 245–248. [48] Raphael Wimmer and Patrick Baudisch. 2011. Modular and deformable touch-sensitive surfaces based on time domain reectometry. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 517–526. [49] Daniel J Yeager, Alanson P Sample, Joshua R Smith, and Joshua R Smith. 2008. Wisp: A passively powered uhf rd tag with sensing and computation. RFID handbook: Applications, technology, security, and privacy (2008), 261–278. [50] Pengyu Zhang, Dinesh Bharadia, Kiran Joshi, and Sachin Katti. 2016. Hitchhike: Practical backscatter using commodity wi. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM. ACM, 259–271. [51] Yang Zhang and Chris Harrison. 2018. Pulp Nonction: Low-Cost Touch Tracking for Paper. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 117. [52] Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick: Low-cost touch sensing using electric eld tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1–14. [53] Yang Zhang, Chouchang Jack Yang, Scott E Hudson, Chris Harrison, and Alanson Sample. 2018. Wall++: Room-Scale Interactive and Context-Aware Sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 273. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 1, Article 27. Publication date: March 2020. ... Self-sustainable active communication powered by high-energy actions (e.g., a door closing) have been proposed for direction sensing, but such techniques do not support interactions such as speech or simple touches as they are too low in power [76]. Recently, Ubiquitouch has shown FM backscatter-based touch and swipe sensing but requires a power budget of about 35 and is powered by a solar cell of 50 2 [65]. Such systems cannot be easily converted to a post-it like form factor or have the simplicity of MARS (<10 components), whose simplicity may lead to it being printable (subsection 6.2). ... ... It does not require a constant like the commonly used ADG-902 RF switch IC in similar works [54,65]. Figure 7 shows the relationship between , and of the JFET measured using an Agilent E5272A. ... Conference Paper Full-text available Augmenting everyday surfaces with interaction sensing capability that is maintenance-free, low-cost (∼$1), and in an appropriate form factor is a challenge with current technologies. MARS (Multi-channel Ambiently-powered Realtime Sensing) enables battery-free sensing and wireless communication of touch, swipe, and speech interactions by combining a nanowatt programmable oscillator with frequency-shifted analog backscatter communication. A zero-threshold voltage field-effect transistor (FET) is used to create an oscillator with a low startup voltage (∼500 mV) and current (< 2uA), whose frequency can be affected through changes in inductance or capacitance from the user interactions. Multiple MARS systems can operate in the same environment by tuning each oscillator circuit to a different frequency range. The nanowatt power budget allows the system to be powered directly through ambient energy sources like photodiodes or thermoelectric generators. We differentiate MARS from previous systems based on power requirements, cost, and part count and explore different interaction and activity sensing scenarios suitable for indoor environments.
... Wang et al. [35] have demonstrated that smart posters are able to convey information (e.g., audio and notifications) to nearby users by backscattering FM radio signals. Waghmare et al. [78] have shown that posters can be used to collect user feedback (e.g., shopping experience) and transmit the collected information to a nearby smartphone using FM backscatter. ...
Article
Owing to the nature of ultra-low-power and extremely low deployment as well as maintenance costs, ambient backscatter technology is promising to become a prominent choice of future low-power communications systems, especially Internet of Things (IoT). Considerable studies have been reported in the emerging area of ambient backscatter communications. However, there is a lack of this kind of survey that is specific to a certain area of ambient backscatter communications and discusses some of the latest systems and the developments in the emerging area. Therefore, in this paper, we provide a comprehensive survey of the existing literature related to ambient backscatter communications. We first present the basic principles of ambient backscatter communications covering architecture, basic techniques, and primer knowledge of ambient signals. After that, we provide a new taxonomy for ambient backscatter communications systems, based on the type of ambient signals. We also review different ambient backscatter communications systems proposed in the literature for each category. Then, we describe potential applications driven by ambient backscatter communications. Finally, some open issues and challenges for future research are identified and discussed.
... The RF switch changes its state according to the oscillator frequency, resulting in a frequency shift in the backscattered signal. The concept of a touchpad using FM backscatter was presented in UbiquiTouch [92], which modulates a touch point on a surface to its corresponding time-series pattern of the frequency shift. OFDMA backscatter localization with ultra-low power was also proposed in Ref. [93] using an extended MUSIC algorithm. ...
Article
Full-text available
Context recognition is a topic that has garnered considerable interest in the ubiquitous and pervasive computing research community. A wide variety of Internet-of-things devices with micro-electromechanical system (MEMS) sensors are used to obtain sensor data (e.g., acceleration, vibration, and sound) related to target contexts. However, devices for context recognition also have limitations such as deployment cost, battery maintenance cost, and the requirement for wearing/carrying the devices. To solve this problem, wireless sensing has attracted the attention of many researchers because it enables device-free and/or maintenance-free context recognition. In this study, we will comprehensively review studies on context recognition by wireless sensing, focusing on WiFi channel state information (CSI), radio-frequency identification (RFID), and backscatter. We will also discuss the design choices of wireless sensing with their pros and cons through a review of the state-of-the-art.
Article
Ubiquitous touch sensing surfaces are largely influenced by touchscreens' look and feel and fail to express the physical richness of existing surrounding materials. We introduce Sensurfaces, a plug-and-play electronic module that allows to rapidly experiment with touch-sensitive surfaces while preserving the original appearance of materials. Sensurfaces is composed of plug-and-play modules that can be connected together to expand the size and number of materials composing a sensitive surface. The combination of Sensurfaces modules allows the creation of small or large multi-material sensitive surfaces that can detect multi-touch but also body proximity, pose, pass, or even human steps. In this paper, we present the design and implementation of Sensurfaces. We propose a design space describing the factors of Sensurfaces interfaces. Then, through a series of technical evaluations, we demonstrate the capabilities of our system. Finally, we report on two workshops validating the usability of our system.
Article
Energy-efficient and unconstrained wearable sensing platforms are essential for ubiquitous healthcare and activity monitoring applications. This paper presents Twin Meander Coil for wirelessly connecting battery-free on-body sensors to a textile-based reader knitted into clothing. This connection is based on passive inductive telemetry (PIT), wherein an external reader coil collects data from passive sensor coils via the magnetic field. In contrast to standard active sensing techniques, PIT does not require the reader to power up the sensors. Thus, the reader can be fabricated using a lossy conductive thread and industrial knitting machines. Furthermore, the sensors can superimpose information such as ID, touch, rotation, and pressure on its frequency response. However, conventional PIT technology needs a strong coupling between the reader and the sensor, requiring the reader to be small to the same extent as the sensors' size. Thus, applying this technology to body-scale sensing systems is challenging. To enable body-scale readout, Twin Meander Coil enhances the sensitivity of PIT technology by dividing the body-scale meander-shaped reader coils into two parts and integrating them so that they support the readout of each other. To demonstrate its feasibility, we built a prototype with a knitting machine, evaluated its sensing ability, and demonstrated several applications.
Chapter
In this paper, we introduce a new problem setting for mobile robots based on backscatter-based communication and sensing. Ambient backscatter communication is a technology that transmits/receives data only by switching the impedance of the antenna at high speed without creating a carrier wave on the transmitting side (target backscatter tag). It modulates and transmits data by turning on/off radio waves and reflecting/absorbing radio waves such as Wi-Fi existing in the environment. Data transmission and backscatter tags’ sensing can be done with several tens of $$\mu$$W of power consumption while general Wi-Fi communication requires several tens of mW of power consumption. We have developed a software defined radio (SDR) system for backscatter-based communication and advanced sensing of humans and objects. By equipping each SDR system with multiple antennas and implementing a mechanism to estimate the direction of backscatter communication with high accuracy, and by using multiple SDR systems, our SDR systems can not only transmit/receive data of ambient backscatter communication but also analyze signals obtained from backscatter tags and estimate their positions concurrently with an error of a few centimeters. Computation by a swarm of autonomous mobile robots is one of the most active fields in the distributed computing community. By arranging the multiple SDR systems in a target area, it may be possible to give new environmental conditions to the time-series position estimation of mobile robots. Such environmental conditions can be used for finding a new problem setting for the mobile robots and/or context recognition of humans and objects using mobile robots and backscatter-based communication.
Article
We present ENGAGE, the first battery-free, personal mobile gaming device powered by energy harvested from the gamer actions and sunlight. Our design implements a power failure resilient Nintendo Game Boy emulator that can run off-the-shelf classic Game Boy games like Tetris or Super Mario Land. This emulator is capable of intermittent operation by tracking memory usage, avoiding the need for always checkpointing all volatile memory, and decouples the game loop from user interface mechanics allowing for restoration after power failure. We build custom hardware that harvests energy from gamer button presses and sunlight, and leverages a mixed volatility memory architecture for efficient intermittent emulation of game binaries. Beyond a fun toy, our design represents the first battery-free system design for continuous user attention despite frequent power failures caused by intermittent energy harvesting. We tackle key challenges in intermittent computing for interaction including seamless displays and dynamic incentive-based gameplay for energy harvesting. This work provides a reference implementation and framework for a future of battery-free mobile gaming in a more sustainable Internet of Things.
Article
In this paper, we present FlexTouch, a technique that enables large-scale interaction sensing beyond the spatial constraints of capacitive touchscreens using passive low-cost conductive materials. This is achieved by customizing 2D circuit-like patterns with an array of conductive strips that can be easily attached to the sensing nodes on the edge of the touchscreen. FlexTouch requires no hardware modification, and is compatible with various conductive materials (copper foil tape, silver nanoparticle ink, ITO frames, and carbon paint), as well as fabrication methods (cutting, coating, and ink-jet printing). Through a series of studies and illustrative examples, we demonstrate that FlexTouch can support long-range touch sensing for up to 4 meters and everyday object presence detection for up to 2 meters. Finally, we show the versatility and feasibility of FlexTouch through applications such as body posture recognition, human-object interaction as well as enhanced fitness training experiences.
Conference Paper
We introduce I/O Braid, an interactive textile cord with embedded sensing and visual feedback. I/O Braid senses proximity, touch, and twist through a spiraling, repeating braiding topology of touch matrices. This sensing topology is uniquely scalable, requiring only a few sensing lines to cover the whole length of a cord. The same topology allows us to embed fiber optic strands to integrate co-located visual feedback. We provide an overview of the enabling braiding techniques, design considerations, and approaches to gesture detection. These allow us to derive a set of interaction techniques, which we demonstrate with different form factors and capabilities. Our applications illustrate how I/O Braid can invisibly augment everyday objects, such as touch-sensitive headphones and interactive drawstrings on garments, while enabling discoverability and feedback through embedded light sources.
Conference Paper
Despite sound being a rich source of information, computing devices with microphones do not leverage audio to glean useful insights about their physical and social context. For example, a smart speaker sitting on a kitchen countertop cannot figure out if it is in a kitchen, let alone know what a user is doing in a kitchen - a missed opportunity. In this work, we describe a novel, real-time, sound-based activity recognition system. We start by taking an existing, state-of-the-art sound labeling model, which we then tune to classes of interest by drawing data from professional sound effect libraries traditionally used in the entertainment industry. These well-labeled and high-quality sounds are the perfect atomic unit for data augmentation, including amplitude, reverb, and mixing, allowing us to exponentially grow our tuning data in realistic ways. We quantify the performance of our approach across a range of environments and device categories and show that microphone-equipped computing devices already have the requisite capability to unlock real-time activity recognition comparable to human accuracy.
Article
This paper presents a passive wireless RF sensor platform (RFSP), with only analog components, that harvests energy from an RF source and reflects data as a direct subcarrier modulation, thus making it battery free. A fully-analog architecture results in an ultra-low power device (under 200 μW) with a low component count, reducing the physical footprint. We envision such a platform to enable medical sensing systems that fit on a small bandaid like flexible structure, require no-battery, or charging and are able to provide continuous physiological monitoring. To realize this vision, we have developed and optimized a novel RF architecture that 1) directly maps sensor output to frequency modulation and transmits it to a remote receiver processing unit (RPU). This direct frequency mapping allows all further digitization and computation to be moved to the RPU -- reducing power and size requirements on the RFSP; 2) harvests energy from the carrier signal transmitted by a simple continuous wave transmitter, thereby requiring no batteries or supercap; and 3) uses backscatter to communicate with the RPU enabling ultra-low power requirements. The total power consumption of our prototype device leveraging this architecture was measured to be between 35 μW and 160 μW. We demonstrate that the RFSP can harvest sufficient power, sense, and communicate continuously without necessity for energy storage at a distance of 4 m from a transmitter emitting a 915 MHz continuous wave at 26 dBm (0.39 W). Prior backscatter systems typically have power budgets of 1 mW and require energy storage (battery or supercap), RFSP's sub 200 μW power consumption provides a significant improvement and longer range for a given TX power. To demonstrate applicability to real-world health sensing and the flexibility to adapt to different sensors, this paper presents results from breathing, heart rate, temperature, and sound sensing applications.
Article
Consolidated tables showing an extensive listing of the highest independently confirmed efficiencies for solar cells and modules are presented. Guidelines for inclusion of results into these tables are outlined and new entries since January 2018 are reviewed.
Conference Paper
Paper continues to be a versatile and indispensable material in the 21st century. Of course, paper is a passive medium with no inherent interactivity, precluding us from computationally-enhancing a wide variety of paper-based activities. In this work, we present a new technical approach for bringing the digital and paper worlds closer together, by enabling paper to track finger input and also drawn input with writing implements. Importantly, for paper to still be considered paper, our method had to be very low cost. This necessitated research into materials, fabrication methods and sensing techniques. We describe the outcome of our investigations and show that our method can be sufficiently low-cost and accurate to enable new interactive opportunities with this pervasive and venerable material.
Conference Paper
Human environments are typified by walls, homes, offices, schools, museums, hospitals and pretty much every indoor context one can imagine has walls. In many cases, they make up a majority of readily accessible indoor surface area, and yet they are static their primary function is to be a wall, separating spaces and hiding infrastructure. We present Wall++, a low-cost sensing approach that allows walls to become a smart infrastructure. Instead of merely separating spaces, walls can now enhance rooms with sensing and interactivity. Our wall treatment and sensing hardware can track users' touch and gestures, as well as estimate body pose if they are close. By capturing airborne electromagnetic noise, we can also detect what appliances are active and where they are located. Through a series of evaluations, we demonstrate Wall++ can enable robust room-scale interactive and context-aware applications.
Conference Paper
In this paper, we design and develop RIO, a novel battery-free touch sensing user interface (UI) primitive for future IoT and smart spaces. RIO enables UIs to be constructed using off-the-shelf RFID readers and tags, and provides a unique approach to designing smart IoT spaces. With RIO, any surface can be turned into a touch-aware surface by simply attaching RFID tags to them. RIO also supports custom-designed RFID tags, and thus allows specially customized UIs to be easily deployed into a real-world environment. RIO is built using the technique of impedance tracking: when a human finger touches the surface of an RFID tag, the impedance of the antenna changes. This change manifests as a change in the phase of the RFID backscattered signal, and is used by RIO to track fine-grained touch movement over both off-the shelf and custom built tags. We study this impedance behavior in-depth and show how RIO is a reliable UI primitive that is robust even within a multi-tag environment. We leverage this primitive to build a prototype of RIO that can continuously locate a finger during a swipe movement to within 3 mm of its actual position. We also show how custom-design RFID tags can be built and used with RIO, and provide two example applications that demonstrate its real-world use.