Posted on

Using your Brain for HCI

To further increase the bandwidth from humans to computers, we are investigating methods for sensing signals that users naturally give off while using a computer system. We use this data to augment the explicit input that the user provides through standard input devices.

Using a relatively new brain sensing tool called functional near-infrared spectroscopy (fNIRS), along with a more established brain sensing tool called electroencephalography (EEG), we can detect signals within the brain that indicate various cognitive states. These devices provide data on brain activity while remaining portable and non-invasive. The cognitive state information can be used as input to provide the user with a richer and more supportive environment, particularly in challenging or high workload situations such as management of unmanned aerial vehicles, driving, air traffic control, video games, health care, training, and anything involving information overload, interruptions or multitasking. It may also improve operation at the other end of the spectrum in highly automated systems that require little effort from the human, but that can result in boredom and low performance. In addition, while most of my research has focused on the broader population of healthy users, many of the results would benefit disabled users as well, by providing additional channels of communication in a lightweight manner.

Related Publications

  • R. Moradinezhad, E.T. Solovey. Integrating Brain and Physiological Sensing with Virtual Agents to Amplify Human Perception. In Proceedings of ACM CHI 2017 Workshop on Amplification and Augmentation of Human Perception.
  • J. Chan, P. Siangliulue, D. Q. McDonald, R. Liu, R. Moradinezhad, S. Aman, E.T. Solovey, K. Gajos & S.P. Dow. (2017). Semantically far inspirations considered harmful? Accounting for cognitive states in collaborative ideation. In Proceedings of 2017 ACM Conference on Creativity and Cognition. 
  • S. Keating, E. Walker, A. Motupali, E.T. Solovey. Toward Real-time Brain Sensing for Learning Assessment: Building a Rich Dataset. In Proc. ACM CHI 2016 Extended Abstracts. San Jose, CA. May, 2016. (To Appear)
  • D. Belyusar, B. Mehler, E.T. Solovey, & B. Reimer. The Impact of Repeated Exposure to a Multi-Level Working Memory Task on Physiological Arousal and Driving Performance, Transportation Research Record. Also appeared in (2015) Proceedings of The Transportation Research Board 94th Annual Meeting, Washington, DC.
  • M. Boyer, M.L. Cummings, L.B. Spence, E.T. Solovey. Investigating Mental Workload Changes in a Long Duration Supervisory Control Task, Interacting With Computers (2015). [link]
  • E.T. Solovey, D. Afergan, A. Venkat, D. Belyusar, B. Mehler, B. Reimer. “Enabling Adaptive Autonomy: Brain & Body Sensing for Adaptive Vehicles,” Proc. CHI 2015 Workshop on Autonomous Driving UX. (2015). (To Appear). [link]
  • E.T. Solovey, D. Afergan, E.M. Peck, S. Hincks, R.J.K. Jacob, Designing Implicit Interfaces for Physiological Computing: Guidelines and Lessons Learned using fNIRS, ACM Transactions on Computer-Human Interaction (TOCHI) Vol. 21, Iss. 6., 2015. [link]
  • C. Hoef, J. Davis, O. Shaer, E.T. Solovey, An In-Depth Look at the Benefits of Immersion Cues on Spatial 3D Problem Solving, ACM Symposium on Spatial User Interfaces (Poster), Honolulu, HI, October 4-5, 2014.
  • D. Belyusar, B. Reimer, B. Mehler, D. Afergan, J.F. Coughlin, E.T. Solovey, Utilizing functional near-infrared spectroscopy to identify cognitive processes contributing to workload in a dual-task environment, Society for Neuroscience Annual Meeting (Poster Presentation), Washington, D.C., November 18, 2014.
  • D. Afergan, E. Peck, E.T. Solovey, A. Jenkins, S. Hincks, E.T. Brown, R. Chang, R.J.K. Jacob. Dynamic Difficulty Using Brain Metrics of Workload. Proc. ACM Conference on Human Factors in Computing Systems CHI ’14, ACM Press (2014).Best Paper Nominee.[Acceptance Rate:5%] [link]
  • A. Girouard, E.T. Solovey, and R.J.K. Jacob, Designing a Passive Brain Computer Interface using Real Time Classification of Functional Near-Infrared Spectroscopy, International Journal of Autonomous and Adaptive Communications Systems (2013). [link]
  • E.T. Solovey, Real-time fNIRS Brain Input for Adaptive Robot Autonomy, Proc HRI Pioneers Workshop (2012). [Acceptance Rate: 23%]
  • E.T. Solovey. Real-time fNIRS Brain Input for Enhancing Interactive Systems. Ph.D. Dissertation, Computer Science Department, Tufts University, Medford, MA. (2012).
  • E.T. Solovey, P. Schermerhorn, M. Scheutz, A. Sassaroli, S. Fantini, R.J.K. Jacob, Brainput: Enhancing Interactive Systems with Streaming fNIRS Brain Input,” Proc. ACM Conference on Human Factors in Computing Systems CHI’12, ACM Press (2012). Best Paper Nominee.[Acceptance Rate:5%] [link]
  • E.T. Solovey, K. Chauncey, F. Lalooses, M. Parasi, D. Weaver, M. Scheutz, P. Schermerhorn, A. Sassaroli, S. Fantini, A. Girouard, R.J.K. Jacob, Sensing Cognitive Multitasking for a Brain-Based Adaptive User Interface, Proc. ACM Conference on Human Factors in Computing Systems CHI’11, ACM Press (2011). [Acceptance Rate: 26%] [link]
  • E.T Solovey, R.J.K. Jacob, Meaningful Human-Computer Interaction Using fNIRS Brain Sensing, Proc ACM CHI 2011 Workshop on Brain and Body Interfaces: Designing for Meaningful Interaction (2011). [link]
  • Peck, E.M., Solovey, E.T., Su, S., Jacob, R.J.K., and Chang, R. Near to the Brain: Functional Near-Infrared Spectroscopy as a Lightweight Brain Imaging Technique for Visualization. InfoVis 2011. (Poster Paper). [link]Best Poster Award
  • A. Girouard, E.T. Solovey, R. Mandryk, D. Tan, L. Nacke, R.J.K. Jacob, Brain, Body and Bytes: Psychophysiological User Interaction, Proc. ACM CHI 2010 Extended Abstracts. (2010). [link]
  • Peck, E. and Solovey, E.T. Neuroscience and Computing. ACM XRDS: Crossroads Magazine 18, 1 (2011), 5-5.[link]
  • Peck, E. and Solovey, E.t. The sensorium. ACM XRDS: Crossroads Magazine 18, 1 (2011). 14-17. [link]
  • A. Girouard, E. Solovey, L. Hirshfield, E. Peck, K. Chancey, A. Sassaroli, S. Fantini, and R. Jacob, From Brain Signals to Adaptive Interfaces: Using fNIRS in HCI, in Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction, ed. by A. Nijholt, Springer (2010). [link]
  • E.T. Solovey, R.J.K. Jacob, Using fNIRS to Support User Interfaces, fNIRS Conference, Cambridge, MA, Oct 15-17, 2010. (Poster Paper) [abstract] [poster]
  • E.T. Solovey, R.J.K. Jacob, Using fNIRS to Support User Interfaces, fNIRS Conference, Cambridge, MA, Oct 15-17, 2010. (Poster Paper) [abstract] [poster]
  • E. Peck, K. Chauncey, A. Girouard, R. Gulotta, F. Lalooses, E.T. Solovey, D. Weaver, and R. Jacob, From Brains to Bytes, XRDS: Crossroads, The ACM Magazine for Students, vol. 16, no. 4, pp. 42-47 (2010). [link]
  • E.T. Solovey. Using Your Brain for Human-Computer Interaction, Doctoral Symposium, ACM UIST 2009 Symposium on User Interface Software and Technology, ACM Press (2009). [link]
  • E.T. Solovey, A. Girouard, K. Chauncey, L.M. Hirshfield, A. Sassaroli, F. Zheng, S. Fantini , and R.J.K. Jacob, Using fNIRS Brain Sensing in Realistic HCI Settings: Experiments and Guidelines, ACM UIST 2009 Symposium on User Interface Software and Technology, ACM Press (2009). [Acceptance Rate: 18%] [link] 
  • L.M. Hirshfield, E.T. Solovey, A. Girouard, J. Kebinger, R.J.K. Jacob, A. Sassaroli, and S. Fantini , Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy, Proc. ACM CHI 2009 Human Factors in Computing Systems Conference, ACM Press (2009). [Acceptance Rate: 24.5%] [link]
  • L.M. Hirshfield, E.T. Solovey, A. Girouard, R.J.K. Jacob, J. Kebinger, M.S. Horn, O. Shaer, J. Zigelbaum, and R.J.K. Jacob, Using Brain Measurement to Evaluate Reality Based Interactions, Proc. ACM CHI 2009 Workshop on Challenges in Evaluating Usability and User Experience of Reality-Based Interaction (2009). [link]
  • A. Girouard, E.T. Solovey, L.M. Hirshfield, K. Chauncey, A. Sassaroli, S. Fantini, and R.J.K. Jacob, Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements, Proc. INTERACT 2009 Conference (2009). [Acceptance Rate: 29%] [link] 
  • L. Hirshfield, K. Chauncey, R. Gulotta, A. Girouard, E. Solovey, R. Jacob, A. Sassaroli, and S. Fantini, Combining Electroencephalograph and Near Infrared Spectroscopy to Explore Users’ Instantaneous and Continuous Mental Workload States, HCI International 2009 13th International Conference on Human-Computer Interaction, Springer (2009).[link]
  • A. Sassaroli, F. Zheng, M. Coutts, L. M. Hirshfield, A. Girouard, E. T. Solovey, R. J. K. Jacob,Y. Tong, B. deB. Frederick, and S. Fantini, Application of near-infrared spectroscopy for discrimination of mental workloads, SPIE Proceedings 7174 (2009). [link]
  • A. Girouard, L.M. Hirshfield, E. Solovey, and R.J.K. Jacob, Using functional Near-Infrared Spectroscopy in HCI: Toward evaluation methods and adaptive interfaces, Proc. ACM CHI 2008 Workshop on Brain-Computer Interfaces for HCI and Games (2008). [link]
  • A. Sassaroli, F. Zheng, L.M. Hirshfield, A. Girouard, E.T. Solovey, R.J.K. Jacob, and S. Fantini, Discrimination of Mental Workload Levels in Human Subjects with Functional Near-Infrared Spectroscopy, Journal of Innovative Optical Health Sciences (2008). [link]
  • A. Sassaroli, Y. Tong, L. M. Hirshfield, A. Girouard, E. T. Solovey, R. J. K. Jacob, S. Fantini, Real-time assessment of mental workload with near infrared spectroscopy: potential for human-computer interaction, OSA topical meeting, BIOMED, BMD14 (2008) [link]
  • E. T. Solovey. Using your brain for human-computer interaction, Grace Hopper Celebration of Women in Computing. Poster Session (2008).
  • A. Girouard, E. T. Solovey, L. M. Hirshfield, K. Chauncey, A. Sassoroli, S. Fantini, and R. J. K. Jacob, Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements, Technical Report 2008-3, Department of Computer Science, Tufts University, Medford, Mass. (2008).
  • L.M. Hirshfield, A. Girouard, E.T. Solovey, R.J.K. Jacob, A. Sassaroli, Y. Tong, and S. Fantini , Human-Computer Interaction and Brain Measurement Using Functional Near-Infrared Spectroscopy, ACM UIST 2007 Symposium on User Interface Software and Technology, ACM Press, Poster paper (2007). [link]

News and Related Articles

  • Corriere della Sera. (May 25, 2012). Il computer che sa metterti a riposo [link]
  • Discovery News. (May 21, 2012). Fix Stress Overload with a ‘Brainput’ System [link] [pdf]
  • Smartplanet Smart Takes. (May 20, 2012). Wearable brain sensor helps workers multitask [link][pdf]
  • IEEE Spectrum Tech Talk (May 17, 2012). Wearable Brain Scanner Tells Your Computer When You’re Overwhelmed [link] [pdf]
  • Haaretz (May 16, 2012). [link] [pdf] [Translated]
  • Wired UK. (May 15, 2012). MIT’s ‘Brainput’ offloads human multitasking to a computer [link] [pdf]
  • Engadget. (May 15, 2012). MIT’s Brainput reads your mind to make multitasking easier [link]  [pdf]
  • Technology Review. (May 14, 2012). A Computer Interface that Takes a Load Off Your Mind [link] [pdf]
  • Extremetech. (May 14, 2012). MIT’s Brainput boosts your brain power by offloading multitasking to a computer [link] [pdf]
  • Mind-reading Computers: It may sound like sci-fi, but one day a computer may sense when you’re stressed and tell you to take a break, Tufts Journal. Feb. 3, 2008. [link] [pdf]
  • Tufts researchers delve into the human brain with cutting-edge ‘light imaging’ technology, Tufts Daily. Oct. 18, 2007. [link] [pdf]
  • Mind controls computer, EE Times. Oct. 5, 2007. [link] [pdf]
  • Computer Can Tell How Hard You’re Working, Fox News. Oct. 3, 2007. [link] [pdf]
  • Mind-Reading Computers, BBC Focus Magazine. Oct. 3, 2007. [link] [pdf]
  • Building a computer that reads minds, MSNBC. Oct. 2, 2007. [link] [pdf]
  • Technology Could Enable Computers To ‘Read The Minds’ Of Users, Science Daily. Oct. 1, 2007. [link] [pdf]
  • J. C. Keller, Technology May Bridge Emotion Gap between Humans and Computers, Engineering eNews. May 2007. [link] [pdf]