Xiaoran Fan
ABSTRACT
This paper presents Audioplethysmography (APG), a novel cardiac monitoring modality for active noise cancellation (ANC) headphones. APG sends a low intensity ultrasound probing signal using an ANC headphone's speakers and receives the echoes via the on-board feedback microphones. We observed that, as the volume of ear canals slightly changes with blood vessel deformations, the heartbeats will modulate these ultrasound echoes. We built mathematical models to analyze the underlying physics and propose a multi-tone APG signal processing pipeline to derive the heart rate and heart rate variability in both constrained and unconstrained settings. APG enables robust monitoring of cardiac activities using mass-market ANC headphones in the presence of music playback and body motion such as running.
We conducted an eight-month field study with 153 participants to evaluate APG in various conditions. Our studies conform to the (Institutional Review Board) IRB policies from our company. The presented technology, experimental design, and results have been reviewed and further improved by feedback garnered from our internal Health Team, Product Team, User Experience (UX) Team and Legal team. Our results demonstrate that APG achieves consistently high HR (3.21% median error across 153 participants in all scenarios) and HRV (2.70% median error in interbeat interval, IBI) measurement accuracy. Our UX study further shows that APG is resilient to variation in: skin tone, sub-optimal seal conditions, and ear canal size.
- Improving skin tone representation across google. Website, 2012.Google Scholar
- Onyx ii model 9560 finger pulse oximeter. Website, 2012.Google Scholar
- BOSE QC-35 Wireless Headphones. Website, 2020.Google Scholar
- Gras 45bb kemar head & torso. Webpage, 2020.Google Scholar
- Gras ra0404 prepolarized hi-res ear simulator. Webpage, 2020.Google Scholar
- Polar h10 - the gold standard. Webpage, 2020.Google Scholar
- A research space for earable computing. Webpage, 2020.Google Scholar
- Samgsung Galaxy Buds. Website, 2020.Google Scholar
- Wireless ecg, spo2, ptt and heart rate monitor reference design for medical and consumer wearables. Website, 2020.Google Scholar
- Apple airpods. Website, 2021.Google Scholar
- Earable: Helping the world sleep better. Webpage, 2022.Google Scholar
- Find local maxima. Website, 2022.Google Scholar
- Sawtooth wave. Website, 2022.Google Scholar
- Zephyr™ performance systems. Website, 2022.Google Scholar
- Apx58x b series audio analyzers. Website, 2023.Google Scholar
- Developing the monk skin tone scale. Website, 2023.Google Scholar
- Gras 12ba, 12be and 12bb ccp power modules with teds support. Website, 2023.Google Scholar
- How many decibels does a human speak normally. Website, 2023.Google Scholar
- Rme adi-2 pro fs r black edition. Website, 2023.Google Scholar
- Rme fireface ufx+. Website, 2023.Google Scholar
- Tascam series 8p dyna. Website, 2023.Google Scholar
- Alberto P Avolio, Mark Butlin, and Andrew Walsh. Arterial blood pressure measurement and pulse wave analysis---their role in enhancing cardiovascular assessment. Physiological measurement, 2009.Google Scholar
- Ananta Narayanan Balaji, Andrea Ferlini, Fahim Kawsar, and Alessandro Montanari. Stereo-bp: Non-invasive blood pressure sensing with earables. In Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications, 2023.Google ScholarDigital Library
- Nam Bui, Nhat Pham, Jessica Jacqueline Barnitz, Zhanan Zou, Phuc Nguyen, Hoang Truong, Taeho Kim, Nicholas Farrow, Anh Nguyen, Jianliang Xiao, et al. ebp: A wearable system for frequent and comfortable blood pressure monitoring from user's ear. In The 25th annual international conference on mobile computing and networking, 2019.Google Scholar
- Kayla-Jade Butkow, Ting Dang, Andrea Ferlini, Dong Ma, and Cecilia Mascolo. heart: Motion-resilient heart rate monitoring with in-ear microphones. arXiv preprint arXiv:2108.09393, 2021.Google Scholar
- Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemelmacher, Shwetak Patel, and Steven M Seitz. Clearbuds: wireless binaural earbuds for learning-based speech enhancement. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022.Google ScholarDigital Library
- Romit Roy Choudhury. Earable computing: A new area to think about. In Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications, 2021.Google ScholarDigital Library
- Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing, 2009.Google ScholarCross Ref
- Francis A Duck. Medical and non-medical protection standards for ultrasound and infrasound. Progress in biophysics and molecular biology, 2007.Google Scholar
- Xiaoran Fan, Longfei Shangguan, Siddharth Rupavatharam, Yanyong Zhang, Jie Xiong, Yunfei Ma, and Richard Howard. Headfi: bringing intelligence to all headphones. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021.Google ScholarDigital Library
- Xiaoran Fan and Trausti Thormundsson. Design earable sensing systems: Perspectives and lessons learned from industry. In EarComp, 2023.Google Scholar
- Andrea Ferlini, Dong Ma, Robert Harle, and Cecilia Mascolo. Eargate: gait-based user identification with in-ear microphones. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021.Google ScholarDigital Library
- Hannes Gamper, Sakari Tervo, and Tapio Lokki. Head orientation tracking using binaural headset microphones. In Audio Engineering Society Convention 131. Audio Engineering Society, 2011.Google Scholar
- Francis Roosevelt Gilliam III, Robert Ciesielski, Karlen Shahinyan, Pratistha Shakya, John Cunsolo, Jal Mahendra Panchal, Bartlomiej Król-Józaga, Monika Król, Olivia Kierul, Charles Bridges, et al. In-ear infrasonic hemodynography with a digital health device for cardiovascular monitoring using the human audiome. npj Digital Medicine, 2022.Google Scholar
- Jon D Hendrix, Jeffrey D Alderson, Chin Huang Yong, and Ryan A Hellman. Feedback howl management in adaptive noise cancellation system, 2019. US Patent 10,290,296.Google Scholar
- Yincheng Jin, Yang Gao, Xiaotao Guo, Jun Wen, Zhengxiong Li, and Zhanpeng Jin. Earhealth: an earphone-based acoustic otoscope for detection of multiple ear diseases in daily life. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022.Google ScholarDigital Library
- Fahim Kawsar, Chulhong Min, Akhil Mathur, Alessandro Montanari, Utku Günay Acer, and Marc Van den Broeck. esense: Open earable platform for human sensing. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, 2018.Google ScholarDigital Library
- Wolfgang Klippel. Nonlinear large-signal behavior of electrodynamic loudspeakers at low frequencies. Journal of the Audio Engineering Society, 1992.Google Scholar
- Dong Ma, Andrea Ferlini, and Cecilia Mascolo. Oesense: employing occlusion effect for in-ear human sensing. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, 2021.Google ScholarDigital Library
- Monique Moderiano, Maureen McEvoy, Jessie Childs, and Adrian Esterman. Safety of ultrasound exposure: knowledge, attitudes, and practices of australasian sonographers. Journal of Diagnostic Medical Sonography, 2018.Google ScholarCross Ref
- Anh Nguyen, Raghda Alqurashi, Zohreh Raghebi, Farnoush Banaei-Kashani, Ann C Halbower, and Tam Vu. A lightweight and inexpensive in-ear sensing system for automatic whole-night sleep stage monitoring. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM, 2016.Google ScholarDigital Library
- Shahriar Nirjon, Robert F Dickerson, Qiang Li, Philip Asare, John A Stankovic, Dezhi Hong, Ben Zhang, Xiaofan Jiang, Guobin Shen, and Feng Zhao. Musicalheart: A hearty way of listening to music. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems, 2012.Google ScholarDigital Library
- Jun Nishimura and Tadahiro Kuroda. Eating habits monitoring using wireless wearable in-ear microphone. In 2008 3rd International Symposium on Wireless Pervasive Computing. IEEE, 2008.Google ScholarCross Ref
- International Non-Ionizing Radiation Committee of the International Radiation Protection Association et al. Interim guidelines on limits of human exposure to airborne ultrasound. Health Physics, 1984.Google Scholar
- Sebastian Passler and Wolf-Joachim Fischer. Food intake activity detection using a wearable microphone system. In 2011 Seventh International Conference on Intelligent Environments. IEEE, 2011.Google ScholarDigital Library
- Nhat Pham, Tuan Dinh, Zohreh Raghebi, Taeho Kim, Nam Bui, Phuc Nguyen, Hoang Truong, Farnoush Banaei-Kashani, Ann Halbower, Thang Dinh, et al. Wake: a behind-the-ear wearable system for microsleep detection. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, 2020.Google ScholarDigital Library
- Tobias Röddiger, Christopher Clarke, Paula Breitling, Tim Schneegans, Haibin Zhao, Hans Gellersen, and Michael Beigl. Sensing with earables: A systematic literature review and taxonomy of phenomena. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.Google ScholarDigital Library
- Tobias Röddiger, Daniel Wolffram, David Laubenstein, Matthias Budde, and Michael Beigl. Towards respiration rate monitoring using an in-ear headphone inertial measurement unit. In EarComp, 2019.Google ScholarDigital Library
- Daniel Rouseff, John A Flynn, James A Ritcey, and Warren LJ Fox. Acoustic communication using time-reversal signal processing: Spatial and frequency diversity. In AIP Conference Proceedings. American Institute of Physics, 2004.Google ScholarCross Ref
- Fred Shaffer and Jay P Ginsberg. An overview of heart rate variability metrics and norms. Frontiers in public health, 2017.Google Scholar
- SV Shilko, SP Salivonchik, et al. Calculation of pulse wave parameters with account of blood vessel deformation. Russian Journal of Biomechanics, 2001.Google Scholar
- Robert R Sokal and Neal L Oden. Spatial autocorrelation in biology: 1. methodology. Biological journal of the Linnean Society, 1978.Google Scholar
- Phyllis K Stein, Matthew S Bosner, Robert E Kleiger, and Brooke M Conger. Heart rate variability: a measure of cardiac autonomic tone. American heart journal, 1994.Google Scholar
- Wei Sun, Franklin Mingzhe Li, Benjamin Steeper, Songlin Xu, Feng Tian, and Cheng Zhang. Teethtap: Recognizing discrete teeth gestures using motion and acoustic sensing on an earpiece. In 26th International Conference on Intelligent User Interfaces, 2021.Google ScholarDigital Library
- Xue Sun, Jie Xiong, Chao Feng, Wenwen Deng, Xudong Wei, Dingyi Fang, and Xiaojiang Chen. Earmonitor: In-ear motion-resilient acoustic sensing using commodity earphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2023.Google Scholar
- Conny MA van Ravenswaaij-Arts, Louis AA Kollee, Jeroen CW Hopman, Gerard BA Stoelinga, and Herman P van Geijn. Heart rate variability. Annals of internal medicine, 1993.Google Scholar
- Joseph C Volpe Jr. Heart rate monitor for controlling entertainment devices, April 8 2008. US Patent 7,354,380.Google Scholar
- Zi Wang, Yili Ren, Yingying Chen, and Jie Yang. Toothsonic: Earable authentication via acoustic toothprint. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.Google Scholar
- Metin C Yarici, Harry J Davies, Takashi Nakamura, Ian Williams, and Danilo P Mandic. Hearables: In-ear multimodal brain computer interfacing. In Brain-Computer Interface Research. Springer, 2021.Google ScholarCross Ref
Index Terms
- APG: Audioplethysmography for Cardiac Monitoring in Hearables
Index terms have been assigned to the content through auto-classification.
Recommendations
Pulse Arrival Time Based Cuff-Less and 24-H Wearable Blood Pressure Monitoring and its Diagnostic Value in Hypertension
Ambulatory blood pressure monitoring (ABPM) has become an essential tool in the diagnosis and management of hypertension. Current standard ABPM devices use an oscillometric cuff-based method which can cause physical discomfort to the patients with ...
Semantic Hearing: Programming Acoustic Scenes with Binaural Hearables
UIST '23: Proceedings of the 36th Annual ACM Symposium on User Interface Software and TechnologyImagine being able to listen to the birds chirping in a park without hearing the chatter from other hikers, or being able to block out traffic noise on a busy street while still being able to hear emergency sirens and car honks. We introduce semantic ...
Diabetic Macular Edema Screened by Handheld Smartphone-based Retinal Camera and Artificial Intelligence
AbstractOur aim was to assess the tomographic presence of diabetic macular edema in type 2 diabetes patients screened for diabetic retinopathy with color fundus photographs and an artificial intelligence algorithm. Color fundus photographs obtained with a ...
Comments