Tengxiang Zhang
Xin Yi
Yuntao Wang
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Temporal synchronous target selection is an association-free selection technique: users select a target by generating signals (e.g., finger taps and hand claps) in sync with its unique temporal pattern. However, classical pattern set design and input recognition algorithm of such techniques did not leverage users' behavioral information, which limits their robustness to imprecise inputs. In this paper, we improve these two key components by modeling users' interaction behavior. In the first user study, we asked users to tap a finger in sync with blinking patterns with various period and delay, and modeled their finger tapping ability using Gaussian distribution. Based on the results, we generated pattern sets for up to 22 targets that minimized the possibility of confusion due to imprecise inputs. In the second user study, we validated that the optimized pattern sets could reduce error rate from 23% to 7% for the classical Correlation recognizer. We also tested a novel Bayesian, which achieved higher selection accuracy than the Correlation recognizer when the input sequence is short. The informal evaluation results show that the selection technique can be effectively scaled to different modalities and sensing techniques.
Supplemental Material
Available for Download
zip
zhang.zip (31 MB)
Supplemental movie, appendix, image and software files for, Facilitating Temporal Synchronous Target Selection through User Behavior Modeling
- Tomoko Aoki, Peter R. Francis, and Hiroshi Kinoshita. 2003. Differences in the abilities of individual fingers during the performance of fast, repetitive tapping movements. Experimental Brain Research 152, 2 (July 2003), 270--280. https://doi.org/10.1007/s00221-003-1552-zGoogle Scholar
Cross Ref
- Nivedita Arora, Steven L. Zhang, Fereshteh Shahmiri, Diego Osorio, Yi-Cheng Wang, Mohit Gupta, Zhengjun Wang, Thad Starner, Zhong Lin Wang, and Gregory D. Abowd. 2018. SATURN: A Thin and Flexible Self-powered Microphone Leveraging Triboelectric Nanogenerator. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 2 (July 2018), 60:1--60:28. https://doi.org/10.1145/3214263Google ScholarDigital Library
- Daniel Ashbrook, Carlos Tejada, Dhwanit Mehta, Anthony Jiminez, Goudam Muralitharam, Sangeeta Gajendra, and Ross Tallents. 2016. Bitey: An Exploration of Tooth Click Gestures for Hands-free User Interface Control. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '16). ACM, New York, NY, USA, 158--169. https://doi.org/10.1145/2935334.2935389Google ScholarDigital Library
- Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, Eric Larson, and Shwetak Patel. 2013. DopLink: Using the Doppler Effect for Multi-device Interaction. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 583--586. https://doi.org/10.1145/2493432.2493515Google ScholarDigital Library
- N. Barbara and T. A. Camilleri. 2016. Interfacing with a speller using EOG glasses. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 001069--001074. https://doi.org/10.1109/SMC.2016.7844384Google ScholarCross Ref
- Alessio Bellino. 2018. SEQUENCE: a remote control technique to select objects by matching their rhythm. Personal and Ubiquitous Computing (March 2018). https://doi.org/10.1007/s00779-018-1129-2Google Scholar
- Peter Bennett, Stuart Nolan, Ved Uttamchandani, Michael Pages, Kirsten Cater, and Mike Fraser. 2015. Resonant Bits: Harmonic Interaction with Virtual Pendulums. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 49--52. https://doi.org/10.1145/2677199.2680569Google ScholarDigital Library
- Marcus Carter, Eduardo Velloso, John Downs, Abigail Sellen, Kenton O'Hara, and Frank Vetere. 2016. PathSync: Multi-User Gestural Interaction with Touchless Rhythmic Path Mimicry. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3415--3427. https://doi.org/10.1145/2858036.2858284Google ScholarDigital Library
- Kaifei Chen, Jonathan Fürst, John Kolb, Hyung-Sin Kim, Xin Jin, David E. Culler, and Randy H. Katz. 2018. SnapLink: Fast and Accurate Vision-Based Appliance Control in Large Commercial Buildings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4 (Jan. 2018), 129:1--129:27. https://doi.org/10.1145/3161173Google ScholarDigital Library
- Christopher Clarke, Alessio Bellino, Augusto Esteves, and Hans Gellersen. 2017. Remote Control by Body Movement in Synchrony with Orbiting Widgets: An Evaluation of TraceMatch. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3 (Sept. 2017), 45:1--45:22. https://doi.org/10.1145/3130910Google ScholarDigital Library
- Christopher Clarke, Alessio Bellino, Augusto Esteves, Eduardo Velloso, and Hans Gellersen. 2016. TraceMatch: A Computer Vision Technique for User Input by Tracing of Animated Controls. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 298--303. https://doi.org/10.1145/2971648.2971714Google ScholarDigital Library
- Christopher Clarke and Hans Gellersen. 2017. MatchPoint: Spontaneous Spatial Coupling of Body Movement for Touchless Pointing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 179--192. https://doi.org/10.1145/3126594.3126626Google ScholarDigital Library
- Adrian A. de Freitas, Michael Nebeling, Xiang 'Anthony' Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K. Dey. 2016. Snap-To-It: A User-Inspired Platform for Opportunistic Device Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5909--5920. https://doi.org/10.1145/2858036.2858177Google Scholar
- Augusto Esteves, Eduardo Velloso, Andreas Bulling, and Hans Gellersen. 2015. Orbits: Gaze Interaction for Smart Watches Using Smooth Pursuit Eye Movements. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 457--466. https://doi.org/10.1145/2807442.2807499Google ScholarDigital Library
- Augusto Esteves, David Verweij, Liza Suraiya, Rasel Islam, Youryang Lee, and Ian Oakley. 2017. SmoothMoves: Smooth Pursuits Head Movements for Augmented Reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 167--178. https://doi.org/10.1145/3126594.3126616 event-place: Québec City, QC, Canada.Google ScholarDigital Library
- Jackson Feijó Filho, Wilson Prata, and Thiago Valle. 2012. Breath Mobile: A Low-cost Software-based Breathing Controlled Mobile Phone Interface. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services Companion (MobileHCI '12). ACM, New York, NY, USA, 157--160. https://doi.org/10.1145/2371664.2371697Google ScholarDigital Library
- Jackson Feijó Filho, Thiago Valle, and Wilson Prata. 2012. Breath Mobile: A Software-based Hands-free and Voice-free Breathing Controlled Mobile Phone Interface. In Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '12). ACM, New York, NY, USA, 217--218. https://doi.org/10.1145/2384916.2384961Google ScholarDigital Library
- Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2016. Do That, There: An Interaction Technique for Addressing In-Air Gesture Systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2319--2331. https://doi.org/10.1145/2858036.2858308Google ScholarDigital Library
- Chuhan Gao, Yilong Li, and Xinyu Zhang. 2018. LiveTag: Sensing Human-Object Interaction through Passive Chipless WiFi Tags. In Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI '18). Renton, WA, USA.Google Scholar
- Emilien Ghomi, Guillaume Faure, Stéphane Huot, Olivier Chapuis, and Michel Beaudouin-Lafon. 2012. Using Rhythmic Patterns As an Input Method. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1253--1262. https://doi.org/10.1145/2207676.2208579Google ScholarDigital Library
- Jessica A. Grahn and Matthew Brett. 2007. Rhythm and Beat Perception in Motor Areas of the Brain. Journal of Cognitive Neuroscience 19, 5 (May 2007), 893--906. https://doi.org/10.1162/jocn.2007.19.5.893Google ScholarDigital Library
- Antti Jylhä and Cumhur Erkut. 2009. A Hand Clap Interface for Sonic Interaction with the Computer. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, New York, NY, USA, 3175--3180. https://doi.org/10.1145/1520340.1520452Google ScholarDigital Library
- Tiiu Koskela and Kaisa Väänänen-Vainio-Mattila. 2004. Evolution Towards Smart Home Environments: Empirical Evaluation of Three User Interfaces. Personal Ubiquitous Comput. 8, 3-4 (July 2004), 234--240. https://doi.org/10.1007/s00779-004-0283-xGoogle ScholarDigital Library
- Andreas Krause and Carlos Guestrin. 2005. Near-optimal Nonmyopic Value of Information in Graphical Models. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI'05). AUAI Press, Arlington, Virginia, United States, 324--331. http://dl.acm.org/citation.cfm?id=3020336.3020377 event-place: Edinburgh, Scotland.Google ScholarDigital Library
- Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. ACM, 213--224. https://doi.org/10.1145/3242587.3242609Google Scholar
- Luis Leiva, Matthias Böhmer, Sven Gehring, and Antonio Krüger. 2012. Back to the App: The Costs of Mobile Application Interruptions. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '12). ACM, New York, NY, USA, 291--294. https://doi.org/10.1145/2371574.2371617Google ScholarDigital Library
- Felix Xiaozhu Lin, Daniel Ashbrook, and Sean White. 2011. RhythmLink: Securely Pairing I/O-constrained Devices by Tapping. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 263--272. https://doi.org/10.1145/2047196.2047231Google ScholarDigital Library
- Carlos H. Morimoto and Marcio R. M. Mimica. 2005. Eye Gaze Tracking Techniques for Interactive Applications. Comput. Vis. Image Underst. 98, 1 (April 2005), 4--24. https://doi.org/10.1016/j.cviu.2004.07.010Google ScholarCross Ref
- G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. 1978. An analysis of approximations for maximizing submodular set functions---I. Mathematical Programming 14, 1 (Dec. 1978), 265--294. https://doi.org/10.1007/BF01588971Google ScholarDigital Library
- Shwetak N. Patel and Gregory D. Abowd. 2003. A 2-Way Laser-Assisted Selection Scheme for Handhelds in a Physical Environment. In UbiComp 2003: Ubiquitous Computing (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg, 200--207. https://doi.org/10.1007/978-3-540-39653-6_16Google Scholar
- Filipe Quintal, Augusto Esteves, Fábio Caires, Vítor Baptista, and Pedro Mendes. 2019. Wattom: A Consumption and Grid Aware Smart Plug with Mid-air Controls. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '19). ACM, New York, NY, USA, 307--313. https://doi.org/10.1145/3294109.3295642 event-place: Tempe, Arizona, USA.Google ScholarDigital Library
- Bruno H. Repp. 2003. Rate Limits in Sensorimotor Synchronization With Auditory and Visual Sequences: The Synchronization Threshold and the Benefits and Costs of Interval Subdivision. Journal of Motor Behavior 35, 4 (Dec. 2003), 355--370. https://doi.org/10.1080/00222890309603156Google ScholarCross Ref
- Bruno H. Repp. 2005. Sensorimotor synchronization: A review of the tapping literature. Psychonomic Bulletin & Review 12, 6 (Dec. 2005), 969--992. https://doi.org/10.3758/BF03206433Google ScholarCross Ref
- Gabriel Reyes, Jason Wu, Nikita Juneja, Maxim Goldshtein, W. Keith Edwards, Gregory D. Abowd, and Thad Starner. 2018. SynchroWatch: One-Handed Synchronous Smartwatch Gestures Using Correlation and Magnetic Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4 (Jan. 2018), 158:1--158:26. https://doi.org/10.1145/3161162Google ScholarDigital Library
- Duk Shin, Atsushi Katayama, Kyoungsik Kim, Hiroyuki Kambara, Makoto Sato, and Yasuharu Koike. 2006. Using a Myokinetic Synthesizer to Control of Virtual Instruments. In Proceedings of the 16th International Conference on Advances in Artificial Reality and Tele-Existence (ICAT'06). Springer-Verlag, Berlin, Heidelberg, 1233--1242. https://doi.org/10.1007/11941354_128Google ScholarDigital Library
- Linda E. Sibert and Robert J. K. Jacob. 2000. Evaluation of Eye Gaze Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '00). ACM, New York, NY, USA, 281--288. https://doi.org/10.1145/332040.332445 event-place: The Hague, The Netherlands.Google Scholar
- Zheng Sun, Aveek Purohit, Raja Bose, and Pei Zhang. 2013. Spartacus: Spatially-aware Interaction for Mobile Devices Through Energy-efficient Audio Sensing. In Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '13). ACM, New York, NY, USA, 263--276. https://doi.org/10.1145/2462456.2464437Google ScholarDigital Library
- Eduardo Velloso, Marcus Carter, Joshua Newn, Augusto Esteves, Christopher Clarke, and Hans Gellersen. 2017. Motion Correlation: Selecting Objects by Matching Their Movement. ACM Trans. Comput.-Hum. Interact. 24, 3 (2017), 22:1--22:35. https://doi.org/10.1145/3064937Google ScholarDigital Library
- David Verweij, Augusto Esteves, Saskia Bakker, and Vassilis-Javed Khan. 2019. Designing Motion Matching for Real-World Applications: Lessons from Realistic Deployments. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '19). ACM, New York, NY, USA, 645--656. https://doi.org/10.1145/3294109.3295628 event-place: Tempe, Arizona, USA.Google ScholarDigital Library
- Mélodie Vidal, Andreas Bulling, and Hans Gellersen. 2013. Pursuits: Spontaneous Interaction with Displays Based on Smooth Pursuit Eye Movement and Moving Targets. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 439--448. https://doi.org/10.1145/2493432.2493477Google ScholarDigital Library
- Ju Wang, Liqiong Chang, Omid Abari, and Srinivasan Keshav. 2019. Are RFID Sensing Systems Ready for the Real World?. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services - MobiSys '19. ACM Press, Seoul, Republic of Korea, 366--377. https://doi.org/10.1145/3307334.3326084Google ScholarDigital Library
- Andrew D. Wilson and Hrvoje Benko. 2014. CrossMotion: Fusing Device and Image Motion for User Identification, Tracking and Device Association. ACM Press, 216--223. https://doi.org/10.1145/2663204.2663270Google Scholar
- Jacob Otto Wobbrock. 2009. TapSongs: Tapping Rhythm-based Passwords on a Single Binary Sensor. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). ACM, New York, NY, USA, 93--96. https://doi.org/10.1145/1622176.1622194Google ScholarDigital Library
- Jason Wu, Cooper Colglazier, Adhithya Ravishankar, Yuyan Duan, Yuanbo Wang, Thomas Ploetz, and Thad Starner. 2018. Seesaw: rapid one-handed synchronous gesture interface for smartwatches. ACM, 17--20. https://doi.org/10.1145/3267242.3267251Google Scholar
- Xuhai Xu, Chun Yu, Anind K. Dey, and Jennifer Mankoff. 2019. Clench Interface: Novel Biting Input Techniques. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 275:1--275:12. https://doi.org/10.1145/3290605.3300505 event-place: Glasgow, Scotland Uk.Google ScholarDigital Library
- Shumin Zhai, Carlos Morimoto, and Steven Ihde. 1999. Manual and Gaze Input Cascaded (MAGIC) Pointing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 246--253. https://doi.org/10.1145/302979.303053 event-place: Pittsburgh, Pennsylvania, USA.Google ScholarDigital Library
- Ben Zhang, Yu-Hsiang Chen, Claire Tuna, Achal Dave, Yang Li, Edward Lee, and Björn Hartmann. 2014. HOBS: Head Orientation-based Selection in Physical Spaces. In Proceedings of the 2Nd ACM Symposium on Spatial User Interaction (SUI '14). ACM, New York, NY, USA, 17--25. https://doi.org/10.1145/2659766.2659773Google ScholarDigital Library
- Cheng Zhang, Xiaoxuan Wang, Anandghan Waghmare, Sumeet Jain, Thomas Ploetz, Omer T. Inan, Thad E. Starner, and Gregory D. Abowd. 2017. FingOrbits: Interaction with Wearables Using Synchronized Thumb Movements. In Proceedings of the 2017 ACM International Symposium on Wearable Computers (ISWC '17). ACM, New York, NY, USA, 62--65. https://doi.org/10.1145/3123021.3123041Google Scholar
- T. Zhang, N. Becker, Y. Wang, Y. Zhou, and Y. Shi. 2017. BitID: Easily Add Battery-Free Wireless Sensors to Everyday Objects. In 2017 IEEE International Conference on Smart Computing (SMARTCOMP). 1--8. https://doi.org/10.1109/SMARTCOMP.2017.7946990Google ScholarCross Ref
- Tengxiang Zhang, Xin Yi, Ruolin Wang, Yuntao Wang, Chun Yu, Yiqin Lu, and Yuanchun Shi. 2018. Tap-to-Pair: Associating Wireless Devices with Synchronous Tapping. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4 (Dec. 2018), 201:1--201:21. https://doi.org/10.1145/3287079Google ScholarDigital Library
- Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick: Low-Cost Touch Sensing Using Electric Field Tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1--14. https://doi.org/10.1145/3025453.3025842Google ScholarDigital Library
- Yang Zhang, Gierad Laput, and Chris Harrison. 2018. Vibrosight: Long-Range Vibrometry for Smart Environment Sensing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 225--236. https://doi.org/10.1145/3242587.3242608Google ScholarDigital Library
Index Terms
- Facilitating Temporal Synchronous Target Selection through User Behavior Modeling
Recommendations
Evaluation of gaze-added target selection methods suitable for general GUIs
This paper examines three gaze-added methods, the Auto, Manual, and SemiAuto, that have a potential to increase the efficiency of target selection operations in general GUI environments such as MS-Windows and Mac-OS. These three methods employ the human'...
Crossing-based selection with direct touch input
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsFundamental performance results for crossing-based selec-tion tasks with direct touch input are presented. A close adaptation of Accot and Zhai's indirect stylus crossing ex-periment reveals similar trends for direct touch input: touch crossing task ...
Predicting Gaze-based Target Selection in Augmented Reality Headsets based on Eye and Head Endpoint Distributions
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsTarget selection is a fundamental task in interactive Augmented Reality (AR) systems. Predicting the intended target of selection in such systems can provide users with a smooth, low-friction interaction experience. Our work aims to predict gaze-based ...
Comments