Abstract
We propose HeadGesture, a hands-free input approach to interact with Head Mounted Display (HMD) devices. Using HeadGesture, users do not need to raise their arms to perform gestures or operate remote controllers in the air. Instead, they perform simple gestures with head movement to interact with the devices. In this way, users' hands are free to perform other tasks, e.g., taking notes or manipulating tools. This approach also reduces the hand occlusion of the field of view [11] and alleviates arm fatigue [7]. However, one main challenge for HeadGesture is to distinguish the defined gestures from unintentional movements. To generate intuitive gestures and address the issue of gesture recognition, we proceed through a process of Exploration - Design - Implementation - Evaluation. We first design the gesture set through experiments on gesture space exploration and gesture elicitation with users. Then, we implement algorithms to recognize the gestures, including gesture segmentation, data reformation and unification, feature extraction, and machine learning based classification. Finally, we evaluate user performance of HeadGesture in the target selection experiment and application tests. The results demonstrate that the performance of HeadGesture is comparable to mid-air hand gestures, measured by completion time. Additionally, users feel significantly less fatigue than when using hand gestures and can learn and remember the gestures easily. Based on these findings, we expect HeadGesture to be an efficient supplementary input approach for HMD devices.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, HeadGesture: HandsFree Input Approach Leveraging Head Movements for HMD Devices
- 2018. The Head Slide. Website. (2018). Retrieved July 22th, 2018 from https://www.youtube.com/watch?v=BH7fIPEPvoY.Google Scholar
- 2018. HTC Vive. Website. (2018). Retrieved March 7, 2018 from https://www.vive.com/us/.Google Scholar
- 2018. iPhone X. Website. (2018). Retrieved March 7, 2018 from https://www.apple.com/iphone-x/.Google Scholar
- 2018. Microsoft Hololens. Website. (2018). Retrieved March 7, 2018 from https://www.microsoft.com/en-us/hololens.Google Scholar
- GR Arce. 2005. Nonlinear Signal Processing: A Statistical Approach Wiley: New Jersey. (2005).Google Scholar
- Ronald T Azuma. 1997. A survey of augmented reality. Presence: Teleoperators & Virtual Environments 6, 4 (1997), 355--385. Google ScholarDigital Library
- Yuki Ban, Takuji Narumi, Tatsuya Fujii, Sho Sakurai, Jun Imura, Tomohiro Tanikawa, and Michitaka Hirose. 2013. Augmented Endurance: Controlling Fatigue While Handling Objects by Affecting Weight Perception Using Augmented Reality. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 69--78. Google ScholarDigital Library
- Steffi Beckhaus, Kristopher J Blom, and Matthias Haringer. 2007. ChairIO--the chair-based Interface. Concepts and technologies for pervasive games: a reader for pervasive gaming research 1 (2007), 231--264.Google Scholar
- Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series.. In KDD workshop, Vol. 10. Seattle, WA, 359--370. Google ScholarDigital Library
- B Biguer, M Jeannerod, and C Prablanc. 1982. The coordination of eye, head, and arm movements during reaching at a single visual target. Experimental brain research 46, 2 (1982), 301--304.Google Scholar
- Volkert Buchmann, Stephen Violich, Mark Billinghurst, and Andy Cockburn. 2004. FingARtips: Gesture Based Direct Manipulation in Augmented Reality. In Proceedings of the 2Nd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (GRAPHITE '04). ACM, New York, NY, USA, 212--221. Google ScholarDigital Library
- Marcio C. Cabral, Carlos H. Morimoto, and Marcelo K. Zuffo. 2005. On the Usability of Gesture Interfaces in Virtual Reality Environments. In Proceedings of the 2005 Latin American Conference on Human-computer Interaction (CLIHC '05). ACM, New York, NY, USA, 100--108. Google ScholarDigital Library
- Xiang 'Anthony' Chen and Yang Li. 2016. Bootstrapping User-Defined Body Tapping Recognition with Offline-Learned Probabilistic Representation. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 359--364. Google ScholarDigital Library
- Douglas A Craig and Hung T Nguyen. 2006. Wireless real-time head movement system using a personal digital assistant (PDA) for control of a power wheelchair. In Engineering in Medicine and Biology Society, 2005. IEEE-EMBS 2005. 27th Annual International Conference of the. IEEE, 772--775.Google Scholar
- Andrew Crossan, Mark McGill, Stephen Brewster, and Roderick Murray-Smith. 2009. Head Tilting for Interaction in Mobile Contexts. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '09). ACM, New York, NY, USA, Article 6, 10 pages. Google ScholarDigital Library
- Andrew Crossan, John Williamson, Stephen Brewster, and Rod Murray-Smith. 2008. Wrist Rotation for Interaction in Mobile Contexts. In Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '08). ACM, New York, NY, USA, 435--438. Google ScholarDigital Library
- Gerwin de Haan, Eric J. Griffith, and Frits H. Post. 2008. Using the Wii Balance Board&Trade; As a Low-cost VR Interaction Device. In Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology (VRST '08). ACM, New York, NY, USA, 289--290. Google ScholarDigital Library
- James Diebel. 2006. Representing attitude: Euler angles, unit quaternions, and rotation vectors. Matrix 58, 15-16 (2006), 1--35.Google Scholar
- Augusto Esteves, David Verweij, Liza Suraiya, Rasel Islam, Youryang Lee, and Ian Oakley. 2017. SmoothMoves: Smooth Pursuits Head Movements for Augmented Reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 167--178. Google ScholarDigital Library
- Marcela Fejtová, Luis Figueiredo, Petr Novák, Olga Štěpánková, and Ana Gomes. 2009. Hands-free interaction with a computer and other technologies. Universal Access in the Information Society 8, 4 (2009), 277. Google ScholarDigital Library
- Jinjuan Feng and Andrew Sears. 2004. Using Confidence Scores to Improve Hands-free Speech Based Navigation in Continuous Dictation Systems. ACM Trans. Comput.-Hum. Interact. 11, 4 (Dec. 2004), 329--356. Google ScholarDigital Library
- Dmitry O Gorodnichy and Gerhard Roth. 2004. Nouse 'use your nose as a mouse' perceptual vision technology for hands-free games and interfaces. Image and Vision Computing 22, 12 (2004), 931--942.Google ScholarCross Ref
- Tovi Grossman and Ravin Balakrishnan. 2005. The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 281--290. Google ScholarDigital Library
- Chris Harrison, Robert Xiao, Julia Schwarz, and Scott E. Hudson. 2014. TouchTools: Leveraging Familiarity and Skill with Physical Tools to Augment Touch Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2913--2916. Google ScholarDigital Library
- Jibo He, Alex Chaparro, Bobby Nguyen, Rondell Burge, Joseph Crandall, Barbara Chaparro, Rui Ni, and Shi Cao. 2013. Texting While Driving: Is Speech-based Texting Less Risky Than Handheld Texting?. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '13). ACM, New York, NY, USA, 124--130. Google ScholarDigital Library
- Lawrence J Hettinger and Gary E Riccio. 1992. Visually induced motion sickness in virtual environments. Presence: Teleoperators & Virtual Environments 1, 3 (1992), 306--310.Google ScholarDigital Library
- Hans-Günter Hirsch and David Pearce. 2000. The Aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions. In ASR2000-Automatic Speech Recognition: Challenges for the new Millenium ISCA Tutorial and Research Workshop (ITRW).Google Scholar
- Gabriel M Hughes. {n. d.}. Moving Forward with Virtual Reality. Spring and Summer 2015 ({n. d.}), 43.Google Scholar
- Anja Jackowski, Marion Gebhard, and Axel Gräser. 2016. A novel head gesture based interface for hands-free control of a robot. In Medical Measurements and Applications (MeMeA), 2016 IEEE International Symposium on. IEEE, 1--6.Google ScholarDigital Library
- Rob Jacob and Sophie Stellmach. 2016. What You Look at is What You Get: Gaze-based User Interfaces. interactions 23, 5 (Aug. 2016), 62--65. Google ScholarDigital Library
- Shahram Jalaliniya, Diako Mardanbegi, and Thomas Pederson. 2015. MAGIC Pointing for Eyewear Computers. In Proceedings of the 2015 ACM International Symposium on Wearable Computers (ISWC '15). ACM, New York, NY, USA, 155--158. Google ScholarDigital Library
- Shahram Jalaliniya, Diako Mardanbeigi, Thomas Pederson, and Dan Witzner Hansen. 2014. Head and Eye Movement As Pointing Modalities for Eyewear Computers. In Proceedings of the 2014 11th International Conference on Wearable and Implantable Body Sensor Networks Workshops (BSNWORKSHOPS '14). IEEE Computer Society, Washington, DC, USA, 50--53. Google ScholarDigital Library
- Pei Jia, Huosheng H Hu, Tao Lu, and Kui Yuan. 2007. Head gesture recognition for hands-free control of an intelligent wheelchair. Industrial Robot: An International Journal 34, 1 (2007), 60--68.Google ScholarCross Ref
- Viktoria A Kettner and Jeremy IM Carpendale. 2013. Developing gestures for no and yes: Head shaking and nodding in infancy. Gesture 13, 2 (2013), 193--209.Google ScholarCross Ref
- Mikko Kytö, Barrett Ens, Thammathip Piumsomboon, Gun A. Lee, and Mark Billinghurst. 2018. Pinpointing: Precise Head- and Eye-Based Target Selection for Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 81, 14 pages. Google ScholarDigital Library
- Edmund LoPresti, David M. Brienza, Jennifer Angelo, Lars Gilbertson, and Jonathan Sakai. 2000. Neck Range of Motion and Use of Computer Head Controls. In Proceedings of the Fourth International ACM Conference on Assistive Technologies (Assets '00). ACM, New York, NY, USA, 121--128. Google ScholarDigital Library
- Shahzad Malik, Abhishek Ranjan, and Ravin Balakrishnan. 2005. Interacting with Large Displays from a Distance with Vision-tracked Multi-finger Gestural Input. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). ACM, New York, NY, USA, 43--52. Google ScholarDigital Library
- Diako Mardanbegi, Dan Witzner Hansen, and Thomas Pederson. 2012. Eye-based Head Gestures. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, NY, USA, 139--146. Google ScholarDigital Library
- Richard A Monty and John W Senders. 2017. Eye movements and psychological processes. Routledge.Google Scholar
- Louis-Philippe Morency and Trevor Darrell. 2006. Head Gesture Recognition in Intelligent Interfaces: The Role of Context in Improving Recognition. In Proceedings of the 11th International Conference on Intelligent User Interfaces (IUI '06). ACM, New York, NY, USA, 32--38. Google ScholarDigital Library
- Meredith Ringel Morris, Anqi Huang, Andreas Paepcke, and Terry Winograd. 2006. Cooperative Gestures: Multi-user Gestural Interactions for Co-located Groupware. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 1201--1210. Google ScholarDigital Library
- Meredith Ringel Morris, Jacob O Wobbrock, and Andrew D Wilson. 2010. Understanding users' preferences for surface gestures. In Proceedings of graphics interface 2010. Canadian Information Processing Society, 261--268. Google ScholarDigital Library
- Miguel A. Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of Pre-designed and User-defined Gesture Sets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1099--1108. Google ScholarDigital Library
- Niels Christian Nilsson, Stefania Serafin, and Rolf Nordahl. 2014. The Influence of Step Frequency on the Range of Perceptually Natural Visual Walking Speeds During Walking-in-place and Treadmill Locomotion. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology (VRST '14). ACM, New York, NY, USA, 187--190. Google ScholarDigital Library
- Tomi Nukarinen, Jari Kangas, Oleg Špakov, Poika Isokoski, Deepak Akkil, Jussi Rantala, and Roope Raisamo. 2016. Evaluation of HeadTurn: An Interaction Technique Using the Gaze and Head Turns. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (NordiCHI '16). ACM, New York, NY, USA, Article 43, 8 pages. Google ScholarDigital Library
- Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst, and Andy Cockburn. 2013. User-defined gestures for augmented reality. In IFIP Conference on Human-Computer Interaction. Springer, 282--299. Google ScholarDigital Library
- I. POUPYREV. 1998. Egocentric object manipulation in virtual environments: Empirical evaluation of interaction techniques. Computer Graphics Forum, EUROGRAPHICS'98 17, 3 (1998), 41--52.Google Scholar
- Kathrin Probst, David Lindlbauer, Michael Haller, Bernhard Schwartz, and Andreas Schrempf. 2014. A Chair As Ubiquitous Input Device: Exploring Semaphoric Chair Gestures for Focused and Peripheral Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 4097--4106. Google ScholarDigital Library
- Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-defined Motion Gestures for Mobile Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 197--206. Google ScholarDigital Library
- Mel Slater, Martin Usoh, and Anthony Steed. 1994. Steps and ladders in virtual reality. In Virtual Reality Software And Technology. World Scientific, 45--54. Google ScholarDigital Library
- Oleg Špakov, Poika Isokoski, and Päivi Majaranta. 2014. Look and lean: accurate head-assisted eye pointing. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 35--42. Google ScholarDigital Library
- Oleg Špakov and Päivi Majaranta. 2012. Enhanced gaze interaction using simple head gestures. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. ACM, 705--710. Google ScholarDigital Library
- Richard Stoakley, Matthew J. Conway, and Randy Pausch. 1995. Virtual Reality on a WIM: Interactive Worlds in Miniature. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 265--272. Google ScholarDigital Library
- Zhenyu Tang, Chenyu Yan, Sijie Ren, and Huagen Wan. 2016. HeadPager: Page Turning with Computer Vision Based Head Interaction. In Asian Conference on Computer Vision. Springer, 249--257.Google Scholar
- Sam Tregillus. 2016. VR-Drop: Exploring the Use of Walking-in-Place to Create Immersive VR Games. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 176--179. Google ScholarDigital Library
- Sam Tregillus, Majed Al Zayer, and Eelke Folmer. 2017. Hands-free Omnidirectional VR Navigation Using Head Tilt. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4063--4068. Google ScholarDigital Library
- Sam Tregillus and Eelke Folmer. 2016. VR-STEP: Walking-in-Place Using Inertial Sensing for Hands Free Navigation in Mobile VR Environments. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1250--1255. Google ScholarDigital Library
- Javier Varona, Cristina Manresa-Yee, and Francisco J Perales. 2008. Hands-free vision-based interface for computer accessibility. Journal of Network and Computer Applications 31, 4 (2008), 357--374. Google ScholarDigital Library
- Radu-Daniel Vatavu. 2012. User-defined Gestures for Freehand TV Control. In Proceedings of the 10th European Conference on Interactive TV and Video (EuroITV '12). ACM, New York, NY, USA, 45--48. Google ScholarDigital Library
- Julie Wagner, Eric Lecolinet, and Ted Selker. 2014. Multi-finger Chords for Handheld Tablets: Recognizable and Memorable. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2883--2892. Google ScholarDigital Library
- Jia Wang and Robert W. Lindeman. 2011. Silver Surfer: A System to Compare Isometric and Elastic Board Interfaces for Locomotion in VR. In Proceedings of the 2011 IEEE Symposium on 3D User Interfaces (3DUI '11). IEEE Computer Society, Washington, DC, USA, 121--122. http://dl.acm.org/citation.cfm?id=2013881.2014229 Google ScholarDigital Library
- Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined Gestures for Surface Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 1083--1092. Google ScholarDigital Library
- Yukang Yan, Chun Yu, Xiaojuan Ma, Xin Yi, Ke Sun, and Yuanchun Shi. 2018. VirtualGrasp: Leveraging Experience of Interacting with Physical Objects to Facilitate Digital Object Retrieval. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 78, 13 pages. Google ScholarDigital Library
- Shanhe Yi, Zhengrui Qin, Ed Novak, Yafeng Yin, and Qun Li. 2016. Glassgesture: Exploring head gesture interface of smart glasses. In Computer Communications, IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on. IEEE, 1--9.Google ScholarDigital Library
Index Terms
- HeadGesture: Hands-Free Input Approach Leveraging Head Movements for HMD Devices
Recommendations
Eye&Head: Synergetic Eye and Head Movement for Gaze Pointing and Selection
UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and TechnologyEye gaze involves the coordination of eye and head movement to acquire gaze targets, but existing approaches to gaze pointing are based on eye-tracking in abstraction from head motion. We propose to leverage the synergetic movement of eye and head, and ...
The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments
MuC '22: Proceedings of Mensch und Computer 2022Natural user interfaces are on the rise. Manufacturers for Augmented, Virtual, and Mixed Reality head mounted displays are increasingly integrating new sensors into their consumer grade products, allowing gesture recognition without additional hardware. ...
Pinch, Click, or Dwell: Comparing Different Selection Techniques for Eye-Gaze-Based Pointing in Virtual Reality
ETRA '21 Short Papers: ACM Symposium on Eye Tracking Research and ApplicationsWhile a pinch action is gaining popularity for selection of virtual objects in eye-gaze-based systems, it is still unknown how well this method performs compared to other popular alternatives, e.g., a button click or a dwell action. To determine pinch’...
Comments