Dong Ma
Cecilia Mascolo
ABSTRACT
Smart earbuds are recognized as a new wearable platform for personal-scale human motion sensing. However, due to the interference from head movement or background noise, commonly-used modalities (e.g. accelerometer and microphone) fail to reliably detect both intense and light motions. To obviate this, we propose OESense, an acoustic-based in-ear system for general human motion sensing. The core idea behind OESense is the joint use of the occlusion effect (i.e., the enhancement of low-frequency components of bone-conducted sounds in an occluded ear canal) and inward-facing microphone, which naturally boosts the sensing signal and suppresses external interference. We prototype OESense as an earbud and evaluate its performance on three representative applications, i.e., step counting, activity recognition, and hand-to-face gesture interaction. With data collected from 31 subjects, we show that OESense achieves 99.3% step counting recall, 98.3% recognition recall for 5 activities, and 97.0% recall for five tapping gestures on human face, respectively. We also demonstrate that OESense is compatible with earbuds' fundamental functionalities (e.g. music playback and phone calls). In terms of energy, OESense consumes 746 mW during data recording and recognition and it has a response latency of 40.85 ms for gesture recognition. Our analysis indicates such overhead is acceptable and OESense is potential to be integrated into future earbuds.
- AirPods Pro. https://www.apple.com/uk/airpods-pro/, Online. (Accessed on May 19, 2021).Google Scholar
- Sony WF-1000XM3. https://www.sony.co.uk/electronics/truly-wireless/wf-1000xm3, Online. (Accessed on May 19, 2021).Google Scholar
- Bose QuietControl. https://www.bose.com/, Online. (Accessed on May 19, 2021).Google Scholar
- Andrea Ferlini, Alessandro Montanari, Cecilia Mascolo, and Robert Harle. Head motion tracking through in-ear wearables. In Proceedings of the 1st International Workshop on Earable Computing, pages 8--13, 2019.Google ScholarDigital Library
- Nam Bui, Nhat Pham, Jessica Jacqueline Barnitz, Zhanan Zou, Phuc Nguyen, Hoang Truong, Taeho Kim, Nicholas Farrow, Anh Nguyen, Jianliang Xiao, et al. eBP: A Wearable System For Frequent and Comfortable Blood Pressure Monitoring From User's Ear. In The 25th Annual International Conference on Mobile Computing and Networking, pages 1--17, 2019.Google ScholarDigital Library
- Jay Prakash, Zhijian Yang, Yu-Lin Wei, and Romit Roy Choudhury. STEAR: Robust Step Counting from Earables. In Proceedings of the 1st International Workshop on Earable Computing, pages 36--41, 2019.Google ScholarDigital Library
- Abdelkareem Bedri, David Byrd, Peter Presti, Himanshu Sahni, Zehua Gue, and Thad Starner. Stick it in your ear: Building an in-ear jaw movement sensor. In Proceedings of the 2015 ACM International Symposium on Wearable Computers, pages 1333--1338, 2015.Google Scholar
- Toshiyuki Ando, Yuki Kubo, Buntarou Shizuki, and Shin Takahashi. Canalsense: Face-related movement recognition system based on sensing air pressure in ear canals. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pages 679--689, 2017.Google ScholarDigital Library
- Alexis Martin and Jérémie Voix. In-ear Audio Wearable: Measurement of heart and breathing rates for health and safety monitoring. IEEE Transactions on Biomedical Engineering, 65(6):1256--1263, 2017.Google ScholarCross Ref
- Fahim Kawsar, Chulhong Min, Akhil Mathur, Alessandro Montanari, Utku Günay Acer, and Marc Van den Broeck. eSense: Open Earable Platform for Human Sensing. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, pages 371--372, 2018.Google ScholarDigital Library
- Yang Gao, Wei Wang, Vir V Phoha, Wei Sun, and Zhanpeng Jin. EarEcho: Using Ear Canal Echo for Wearable Authentication. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(3):1--24, 2019.Google ScholarDigital Library
- Jennifer R Kwapisz, Gary M Weiss, and Samuel A Moore. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter, 12(2):74--82, 2011.Google ScholarDigital Library
- Muhammad Farooq and Edward Sazonov. Accelerometer-based detection of food intake in free-living individuals. IEEE sensors journal, 18(9):3752--3758, 2018.Google ScholarCross Ref
- Abhinav Parate, Meng-Chieh Chiu, Chaniel Chadowitz, Deepak Ganesan, and Evangelos Kalogerakis. Risq: Recognizing smoking gestures with inertial sensors on a wristband. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services, pages 149--161, 2014.Google ScholarDigital Library
- Mohammad Omar Derawi. Accelerometer-based gait analysis, a survey. Nor Informasjonssikkerhetskonferanse NISK, 1, 2010.Google Scholar
- Xuhai Xu, Haitian Shi, Xin Yi, Wenjia Liu, Yukang Yan, Yuanchun Shi, Alex Mariakakis, Jennifer Mankoff, and Anind K Dey. EarBuddy: Enabling On-Face Interaction via Wireless Earbuds. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1--14, 2020.Google ScholarDigital Library
- Stefan Stenfelt. Acoustic and physiologic aspects of bone conduction hearing. In Implantable bone conduction hearing aids, volume 71, pages 10--21. Karger Publishers, 2011.Google ScholarCross Ref
- Roman Schlieper, Song Li, Stephan Preihs, and Jürgen Peissig. The relationship between the acoustic impedance of headphones and the occlusion effect. In Audio Engineering Society Conference: 2019 AES INTERNATIONAL CONFERENCE ON HEADPHONE TECHNOLOGY. Audio Engineering Society, 2019.Google Scholar
- Michael A Stone, Anna M Paul, Patrick Axon, and Brian CJ Moore. A technique for estimating the occlusion effect for frequencies below 125 hz. Ear and hearing, 35(1):49, 2014.Google ScholarCross Ref
- Stefan Stenfelt and Sabine Reinfeldt. A model of the occlusion effect with bone-conducted stimulation. International journal of audiology, 46(10):595--608, 2007.Google ScholarCross Ref
- Kévin Carillo, Olivier Doutres, and Franck Sgard. Theoretical investigation of the low frequency fundamental mechanism of the objective occlusion effect induced by bone-conducted stimulation. The Journal of the Acoustical Society of America, 147(5):3476--3489, 2020.Google ScholarCross Ref
- Librosa. https://librosa.org/, Online. (Accessed on May 19, 2021).Google Scholar
- Chloë Brown, Jagmohan Chauhan, Andreas Grammenos, Jing Han, Apinan Hasthanasombat, Dimitris Spathis, Tong Xia, Pietro Cicuta, and Cecilia Mascolo. Exploring automatic diagnosis of covid-19 from crowdsourced respiratory sound data. arXiv preprint arXiv:2006.05919, 2020.Google Scholar
- Honor Magic Earbuds. https://www.hihonor.com/global/products/accessories/honor-magic-earbuds/, Online. (Accessed on May 19, 2021).Google Scholar
- MINISO Marvel Earphones. https://www.miniso-au.com/en-au/product/145169/marvel-earphones/, Online. (Accessed on May 19, 2021).Google Scholar
- Microphone SPU1410LR5H-QB. https://www.mouser.com/datasheet/2/218/SPU1410LR5H-QB-215269.pdf, Online. (Accessed on May 19, 2021).Google Scholar
- ReSpeaker Voice Accessory HAT. https://wiki.seeedstudio.com/ReSpeaker_4-Mic_Linear_Array_Kit_for_Raspberry_Pi/, Online. (Accessed on May 19, 2021).Google Scholar
- Marília Barandas, Duarte Folgado, Letícia Fernandes, Sara Santos, Mariana Abreu, Patrícia Bota, Hui Liu, Tanja Schultz, and Hugo Gamboa. Tsfel: Time series feature extraction library. SoftwareX, 11:100456, 2020.Google ScholarCross Ref
- KL Yick, LT Tse, WT Lo, SP Ng, and J Yip. Effects of indoor slippers on plantar pressure and lower limb emg activity in older women. Applied ergonomics, 56:153--159, 2016.Google ScholarCross Ref
- David R Bassett, Lindsay P Toth, Samuel R LaMunion, and Scott E Crouter. Step counting: a review of measurement considerations and health-related applications. Sports Medicine, 47(7):1303--1315, 2017.Google ScholarCross Ref
- Frédéric Camps, Sébastien Harasse, and André Monin. Numerical calibration for 3-axis accelerometers and magnetometers. In 2009 IEEE International Conference on Electro/Information Technology, pages 217--221. IEEE, 2009.Google ScholarCross Ref
- Wenqiang Chen, Maoning Guan, Yandao Huang, Lu Wang, Rukhsana Ruby, Wen Hu, and Kaishun Wu. Vitype: A cost efficient on-body typing system through vibration. In 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pages 1--9. IEEE, 2018.Google ScholarDigital Library
- Mehul P Sampat, Zhou Wang, Shalini Gupta, Alan Conrad Bovik, and Mia K Markey. Complex wavelet structural similarity: A new image similarity index. IEEE transactions on image processing, 18(11):2385--2401, 2009.Google Scholar
- Billboard All-Time Top 100 Songs. https://www.billboard.com/articles/news/hot-100-turns-60/8468142/hot-100-all-time-biggest-hits-songs-list, Online. (Accessed on May 19, 2021).Google Scholar
- D Esteban, C Galand, Daniel Mauduit, and J Menez. 9.6/7.2 kbps voice excited predictive coder (vepc). In ICASSP'78. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 3, pages 307--311. IEEE, 1978.Google ScholarCross Ref
- Frank Angione, Colin Novak, Chris Imeson, Ashley Lehman, Ben Merwin, Tom Pagliarella, Nikolina Samardzic, Peter D'Angela, and Helen Ule. Study of a low frequency emergency siren in comparison to traditional siren technology. In Proceedings of Meetings on Acoustics 172ASA, volume 29, page 030008. Acoustical Society of America, 2016.Google Scholar
- Ming-Zher Poh, Kyunghee Kim, Andrew D Goessling, Nicholas C Swenson, and Rosalind W Picard. Heartphones: Sensor earphones and mobile application for non-obtrusive health monitoring. In 2009 International Symposium on Wearable Computers, pages 153--154. IEEE, 2009.Google ScholarDigital Library
- Denys JC Matthies, Bernhard A Strecker, and Bodo Urban. EarFieldSensing: A novel in-ear electric field sensing to enrich wearable gesture input through facial expressions. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 1911--1922, 2017.Google ScholarDigital Library
- Junjue Wang, Kaichen Zhao, Xinyu Zhang, and Chunyi Peng. Ubiquitous keyboard for small mobile devices: harnessing multipath fading for fine-grained keystroke localization. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services, pages 14--27, 2014.Google ScholarDigital Library
- Yanzhi Ren, Chen Wang, Jie Yang, and Yingying Chen. Fine-grained sleep monitoring: Hearing your breathing with smartphones. In 2015 IEEE Conference on Computer Communications (INFOCOM), pages 1194--1202. IEEE, 2015.Google ScholarCross Ref
- Jagmohan Chauhan, Yining Hu, Suranga Seneviratne, Archan Misra, Aruna Seneviratne, and Youngki Lee. reathPrint: Breathing acoustics-based user authentication. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, pages 278--291, 2017.Google ScholarDigital Library
- Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang, and Marco Gruteser. Snooping keystrokes with mm-level audio ranging on a single phone. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, pages 142--154, 2015.Google ScholarDigital Library
- Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. SoundWave: using the doppler effect to sense gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1911--1914, 2012.Google ScholarDigital Library
- Wei Wang, Alex X Liu, and Ke Sun. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking, pages 82--94, 2016.Google ScholarDigital Library
- Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th annual international conference on mobile systems, applications, and services, pages 15--28, 2017.Google ScholarDigital Library
- Marcos Serrano, Barrett M Ens, and Pourang P Irani. Exploring the use of hand-to-face input for interacting with head-worn displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3181--3190, 2014.Google ScholarDigital Library
- Takashi Kikuchi, Yuta Sugiura, Katsutoshi Masai, Maki Sugimoto, and Bruce H Thomas. EarTouch: turning the ear into an input surface. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, pages 1--6, 2017.Google ScholarDigital Library
- Juyoung Lee, Hui-Shyong Yeo, Murtaza Dhuliawala, Jedidiah Akano, Junichi Shimizu, Thad Starner, Aaron Quigley, Woontack Woo, and Kai Kunze. Itchy Nose: discreet gesture interaction using EOG sensors in smart eyewear. In Proceedings of the 2017 ACM International Symposium on Wearable Computers, pages 94--97, 2017.Google ScholarDigital Library
Index Terms
- OESense: employing occlusion effect for in-ear human sensing
Recommendations
Obtaining Binaural Room Impulse Responses From B-Format Impulse Responses Using Frequency-Dependent Coherence Matching
Measuring binaural room impulse responses (BRIRs) for different rooms and different persons is a costly and time-consuming task. In this paper, we propose a method that allows to compute BRIRs from a B-format room impulse response (B-format RIR) and a ...
Immersive auditory display system 'sound cask': three-dimensional sound field reproduction system based on the boundary surface control principle
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologySound cask was developed to realize the perfect 3D auditory display that creates 3D sound waves around the listener's head just the same as the primary sound field, based on the boundary surface control (BoSC) principle.
If we consider the sound ...
Theoretical analysis of linearly constrained multi-channel wiener filtering algorithms for combined noise reduction and binaural cue preservation in binaural hearing aids
Besides noise reduction, an important objective of binaural speech enhancement algorithms is the preservation of the binaural cues of all sound sources. For the desired speech source and the interfering sources, e.g., competing speakers, this can be ...
Comments