Xiao Zhang
Skip Abstract Section
Abstract
Cognitive load assessment is crucial for user studies and human--computer interaction designs. As a noninvasive and easy-to-use category of measures, current photoplethysmogram- (PPG) based assessment methods rely on single or small-scale predefined features to recognize responses induced by people’s cognitive load, which are not stable in assessment accuracy. In this study, we propose a machine-learning method by using 46 kinds of PPG features together to improve the measurement accuracy for cognitive load. We test the method on 16 participants through the classical n-back tasks (0-back, 1-back, and 2-back). The accuracy of the machine-learning method in differentiating different levels of cognitive loads induced by task difficulties can reach 100% in 0-back vs. 2-back tasks, which outperformed the traditional HRV-based and single-PPG-feature-based methods by 12--55%. When using “leave-one-participant-out” subject-independent cross validation, 87.5% binary classification accuracy was reached, which is at the state-of-the-art level. The proposed method can also support real-time cognitive load assessment by beat-to-beat classifications with better performance than the traditional single-feature-based real-time evaluation method.
- Yomna Abdelrahman, Eduardo Velloso, Tilman Dingler, Albrecht Schmidt, and Frank Vetere. 2017. Cognitive heat: Exploring the usage of thermal imaging to unobtrusively estimate cognitive load. Proc. ACM Interact. Mobile Wear. Ubiq. Technol. 1, 3 (2017), 33.Google Scholar
Digital Library
- S. R. Alty, N. Angaritajaimes, S. C. Millasseau, and P. J. Chowienczyk. 2007. Predicting arterial stiffness from the digital volume pulse waveform. IEEE Trans. Bio-med. Eng. 54, 12 (2007), 2268--75.Google ScholarCross Ref
- Pavlo Antonenko, Fred Paas, Roland Grabner, and Tamara Van Gog. 2010. Using electroencephalography to measure cognitive load. Educ. Psychol. Rev. 22, 4 (2010), 425--438.Google ScholarCross Ref
- Hasan Ayaz, Meltem Izzetoglu, Scott Bunce, Terry Heiman-Patterson, and Banu Onaral. 2007. Detecting cognitive activity related hemodynamic signal for brain computer interface using functional near infrared spectroscopy. In Proceedings of the 3rd International IEEE/EMBS Conference on Neural Engineering 2007 (CNE’07). IEEE, 342--345.Google ScholarCross Ref
- Annemarie Brouwer, Maarten A. Hogervorst, Jan B. F. Van Erp, Tobias Heffelaar, Patrick H Zimmerman, and Robert Oostenveld. 2012. Estimating workload using EEG spectral power and ERPs in the n-back task. J. Neur. Eng. 9, 4 (2012), 045008.Google ScholarCross Ref
- Anne-Marie Brouwer, Maarten A. Hogervorst, Michael Holewijn, and Jan B. F. van Erp. 2014. Evidence for effects of task difficulty but not learning on neurophysiological variables associated with effort. Int. J. Psychophysiol. 93, 2 (2014), 242--252.Google ScholarCross Ref
- Roland Brunken, Jan L. Plass, and Detlev Leutner. 2003. Direct measurement of cognitive load in multimedia learning. Educ. Psychol. 38, 1 (2003), 53--61.Google ScholarCross Ref
- Fang Chen, Natalie Ruiz, Eric Choi, Julien Epps, M. Asif Khawaja, Ronnie Taib, Bo Yin, and Yang Wang. 2012. Multimodal behavior and interaction as indicators of cognitive load. ACM Trans. Interact. Intell. Syst. 2, 4, Article 22 (Jan. 2012), 36 pages. DOI:https://doi.org/10.1145/2395123.2395127Google ScholarDigital Library
- Y. Cho, N. Bianchi-Berthouze, and S. J. Julier. 2017. DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In Proceedings of the 2017 7th International Conference on Affective Computing and Intelligent Interaction (ACII’17). IEEE, 456--463. DOI:https://doi.org/10.1109/ACII.2017.8273639Google Scholar
- Krista E. DeLeeuw and Richard E. Mayer. 2008. A comparison of three measures of cognitive load: Evidence for separable measures of intrinsic, extraneous, and germane load.J. Educ. Psychol. 100, 1 (2008), 223.Google ScholarCross Ref
- Jack Dennerlein, Theodore Becker, Peter Johnson, Carson Reynolds, and Rosalind W. Picard. 2003. Frustrating computer users increases exposure to physical factors. In Proceedings of the International Ergonomics Association.Google Scholar
- Gautier Durantin, J.-F. Gagnon, Sébastien Tremblay, and Frédéric Dehais. 2014. Using near infrared spectroscopy and heart rate variability to detect mental overload. Behav. Brain Res. 259, 1 (2014), 16--23.Google ScholarCross Ref
- M. Elgendi. 2012. On the analysis of fingertip photoplethysmogram signals. Curr. Cardiol. Rev. 8, 1 (2012).Google ScholarCross Ref
- Yuan Gao, Nadia Bianchiberthouze, and Hongying Meng. 2012. What does touch tell us about emotions in touchscreen-based gameplay? ACM Trans. Comput.-Hum. Interact. 19, 4 (2012), 1--30.Google ScholarDigital Library
- Annebet D. Goedhart, Sophie Van Der Sluis, Jan H. Houtveen, Gonneke Willemsen, and Eco J. C. De Geus. 2007. Comparison of time and frequency domain measures of RSA in ambulatory recordings. Psychophysiology 44, 2 (2007), 203--215.Google ScholarCross Ref
- David Grimes, Desney S. Tan, Scott E. Hudson, Pradeep Shenoy, and Rajesh P. N. Rao. 2008. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, New York, NY, 835--844. DOI:https://doi.org/10.1145/1357054.1357187Google Scholar
- Eija Haapalainen, SeungJun Kim, Jodi F. Forlizzi, and Anind K. Dey. 2010. Psycho-physiological measures for assessing cognitive load. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing (UbiComp’10). ACM, New York, NY, 301--310. DOI:https://doi.org/10.1145/1864349.1864395Google Scholar
- C. W. Harris, J. L. Edwards, A Baruch, W. A. Riley, B. E. Pusser, W. J. Rejeski, and D. M. Herrington. 2000. Effects of mental stress on brachial artery flow-mediated vasodilation in healthy normal individuals. Am. Heart J. 139, 3 (2000), 405--411.Google Scholar
- Joshua Harrison, Kurtuluş İzzetoğlu, Hasan Ayaz, Ben Willems, Sehchang Hah, Ulf Ahlstrom, Hyun Woo, Patricia A. Shewokis, Scott C. Bunce, and Banu Onaral. 2014. Cognitive workload and learning assessment during the implementation of a next-generation air traffic control technology using functional near-infrared spectroscopy. IEEE Trans. Hum.-Mach. Syst. 44, 4 (2014), 429--440.Google ScholarCross Ref
- Nis Hjortskov, Dag Rissén, Anne Katrine Blangsted, Nils Fallentin, Ulf Lundberg, and Karen Søgaard. 2004. The effect of mental stress on heart rate variability and blood pressure during computer work. Eur. J. Appl. Physiol. 92, 1--2 (2004), 84--89.Google ScholarCross Ref
- Maarten A. Hogervorst, Anne-Marie Brouwer, and Jan B. F. van Erp. 2014. Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Front. Neurosci. 8, 1 (2014), 322. DOI:https://doi.org/10.3389/fnins.2014.00322Google ScholarCross Ref
- C. Iani, D. Gopher, and P. Lavie. 2004. Effects of task difficulty and invested mental effort on peripheral vasoconstriction. Psychophysiology 41, 5 (2004), 789--798.Google ScholarCross Ref
- Meltem Izzetoglu, Scott C. Bunce, Kurtulus Izzetoglu, Banu Onaral, and Kambiz Pourrezaei. 2007. Functional brain imaging using near-infrared technology. IEEE Eng. Med. Biol. Mag. 26, 4 (2007), 38.Google ScholarCross Ref
- Thorsten Joachims. 2006. Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’06). ACM, New York, NY, 217--226. DOI:https://doi.org/10.1145/1150402.1150429Google ScholarDigital Library
- Mishel Johns, Srinath Sibi, and Wendy Ju. 2014. Effect of cognitive load in autonomous vehicles on driver performance during transfer of control. In Adjunct Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI’14). ACM, New York, NY, 1--4. DOI:https://doi.org/10.1145/2667239.2667296Google ScholarDigital Library
- W. M. Jubadi and S. F. A. Mohd Sahak. 2009. Heartbeat monitoring alert via SMS. In Proceedings of the IEEE Symposium onIndustrial Electronics 8 Applications, 2009. ISIEA 2009. IEEE, 1--5.Google ScholarCross Ref
- J. W. H. Kalsbeek and J. H. Ettema. 1963. Continuous recording of heart rate and the measurement of perceptual load. Ergonomics 6 (1963), 306--307.Google Scholar
- A. Reşit Kavsaoğlu, Kemal Polat, and M. Recep Bozkurt. 2014. A novel feature ranking algorithm for biometric recognition with PPG signals. Comput. Biol. Med. 49, 1 (2014), 1--14. DOI:https://doi.org/10.1016/j.compbiomed.2014.03.005Google ScholarCross Ref
- Wayne K. Kirchner. 1958. Age differences in short-term retention of rapidly changing information. J. Exp. Psychol. 55, 4 (1958), 352.Google ScholarCross Ref
- Naveen Kumar and Jyoti Kumar. 2016. Measurement of cognitive load in HCI systems using EEG power spectrum: An experimental study. Proc. Comput. Sci. 84, 1 (2016), 70--78.Google ScholarCross Ref
- L. Luo, L. Xiao, D. Miao, and X. Luo. 2012. The relationship between mental stress induced changes in cortisol levels and vascular responses quantified by waveform analysis: Investigating stress-dependent indices of vascular changes. In Proceedings of the 2012 International Conference on Biomedical Engineering and Biotechnology. IEEE, 929--933. DOI:https://doi.org/10.1109/iCBEB.2012.437Google ScholarDigital Library
- Yongqiang Lyu, Xiaomin Luo, Jun Zhou, Chun Yu, Congcong Miao, Tong Wang, Yuanchun Shi, and Ken-ichi Kameyama. 2015. Measuring photoplethysmogram-based stress-induced vascular response index to assess cognitive load and stress. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 857--866. DOI:https://doi.org/10.1145/2702123.2702399Google ScholarDigital Library
- Richard E. Mayer and Roxana Moreno. 2003. Nine ways to reduce cognitive load in multimedia learning. Educ. Psychol. 38, 1 (2003), 43--52.Google ScholarCross Ref
- D. McDuff, S. Gontarek, and R. Picard. 2014. Remote measurement of cognitive stress via heart rate variability. In Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2957--2960. DOI:https://doi.org/10.1109/EMBC.2014.6944243Google Scholar
- D. Mcduff, S. Gontarek, and R. W. Picard. 2014. Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Trans. Bio-med. Eng. 61, 10 (2014), 2593--601.Google ScholarCross Ref
- Daniel J. McDuff, Javier Hernandez, Sarah Gontarek, and Rosalind W. Picard. 2016. COGCAM: Contact-free measurement of cognitive stress during computer tasks with a digital camera. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 4000--4004. DOI:https://doi.org/10.1145/2858036.2858247Google Scholar
- Calvin K. L. Or and Vincent G. Duffy. 2007. Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occupat. Ergon. 7, 2 (2007), 83--94.Google ScholarCross Ref
- Sharon Oviatt. 2006. Human-centered design meets cognitive load theory: Designing interfaces that help people think. In Proceedings of the 14th ACM International Conference on Multimedia (MM’06). ACM, New York, NY, 871--880. DOI:https://doi.org/10.1145/1180639.1180831Google ScholarDigital Library
- Fred Paas, Juhani E. Tuovinen, Huib Tabbers, and Pascal W. M. Van Gerven. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38, 1 (2003), 63--71.Google ScholarCross Ref
- Fred G. W. C. Paas and Jeroen J. G. Van Merriënboer. 1994. Instructional control of cognitive load in the training of complex cognitive tasks. Educ. Psychol. Rev. 6, 4 (1994), 351--371.Google ScholarCross Ref
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12, 1 (2011), 2825--2830.Google ScholarDigital Library
- Ming Zher Poh, Daniel J. Mcduff, and Rosalind W. Picard. 2011. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Bio-med. Eng. 58, 1 (2011), 7.Google ScholarCross Ref
- Yu Shi, Natalie Ruiz, Ronnie Taib, Eric Choi, and Fang Chen. 2007. Galvanic skin response (GSR) as an index of cognitive load. In Extended Abstracts on Human Factors in Computing Systems (CHI EA’07). ACM, New York, NY, 2651--2656. DOI:https://doi.org/10.1145/1240866.1241057Google Scholar
- K. G. Shin and P. Ramanathan. 1994. Real-time computing: A new discipline of computer science and engineering. Proc. IEEE 82, 1 (Jan. 1994), 6--24. DOI:https://doi.org/10.1109/5.259423Google ScholarCross Ref
- Erin T. Solovey, Marin Zec, Enrique Abdon Garcia Perez, Bryan Reimer, and Bruce Mehler. 2014. Classifying driver workload using physiological and driving performance data: Two field studies. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI’14). ACM, New York, NY, 4057--4066. DOI:https://doi.org/10.1145/2556288.2557068Google ScholarDigital Library
- Willem B. Verwey and Hans A. Veltman. 1996. Detecting short periods of elevated workload: A comparison of nine workload assessment techniques.J. Exp. Psychol. Appl. 2, 3 (1996), 270--285.Google ScholarCross Ref
- L. Wang, Emma Pickwell-Macpherson, Y. P. Liang, and Y. T. Zhang. 2009. Noninvasive cardiac output estimation using a novel photoplethysmogram index. In Proceedings of the International Conference of the IEEE Engineering in Medicine 8 Biology Society. IEEE, 1746.Google Scholar
- Shouyi Wang, Jacek Gwizdka, and W. Art Chaovalitwongse. 2016. Using wireless EEG signals to assess memory workload in the -back task. IEEE Trans. Hum.-Mach. Syst. 46, 3 (2016), 424--435.Google ScholarCross Ref
- Jianhua Zhang, Zhong Yin, and Rubin Wang. 2015. Recognition of mental workload levels under complex human--machine collaboration by using physiological features and adaptive support vector machines. IEEE Trans. Hum.-Mach. Syst. 45, 2 (2015), 200--214.Google ScholarCross Ref
- G. Zhao, Y. J. Liu, and Y. Shi. 2018. Real-time assessment of the cross-task mental workload using physiological measures during anomaly detection. IEEE Trans. Hum.-Mach. Syst. 48, 2 (Apr. 2018), 149--160. DOI:https://doi.org/10.1109/THMS.2018.2803025Google ScholarCross Ref
- Ferdinand Rudolf Hendrikus Zijlstra. 1993. Efficiency in Work Behaviour: A Design Approach for Modern Tools. Ph.D. Dissertation. Delft University, Delft.Google Scholar
Index Terms
- Photoplethysmogram-based Cognitive Load Assessment Using Multi-Feature Fusion Model
Recommendations
Measuring Photoplethysmogram-Based Stress-Induced Vascular Response Index to Assess Cognitive Load and Stress
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsQuantitative assessment for cognitive load and mental stress is very important in optimizing human-computer system designs to improve performance and efficiency. Traditional physiological measures, such as heart rate variation (HRV), blood pressure and ...
Non-Invasive Measurement of Cognitive Load and Stress Based on the Reflected Stress-Induced Vascular Response Index
Measuring cognitive load and stress is crucial for ubiquitous human--computer interaction applications to dynamically understand and respond to the mental status of users, such as in smart healthcare, smart driving, and robotics. Various quantitative ...
The Cognitive Load Affects the Interaction Pattern of Emotion and Working Memory
Emotion might selectively affect spatial and verbal cognitive activities and the selective interaction patter could be modulated by cognitive load. To test the hypotheses, the authors used event-related potentials ERPs technique to investigate the ...
Comments