Visualization and Intelligent Systems Laboratory



Contact Information

Winston Chung Hall Room 216
University of California, Riverside
900 University Avenue
Riverside, CA 92521-0425

Tel: (951)-827-3954

Bourns College of Engineering
NSF IGERT on Video Bioinformatics

UCR Collaborators:

Other Collaborators:
Keio University

Other Activities:
IEEE Biometrics Workshop 2019
IEEE Biometrics Workshop 2018
Worshop on DVSN 2009
Multibiometrics Book

Webmaster Contact Information:
Alex Shin

Last updated: July 1, 2017



Understanding Subtle Non-Social Facial Expressivity to Boost Learning and Computer Interaction

NSF Project ID: IIS 1911197

Principal Investigator

Bir Bhanu
Department of Electrical and Computer Engineering
University of California at Riverside
Riverside, CA 92521
Tel. (951)827-3954, Fax. (951)827-2425

Co-Principal Investigator

Aaron Seitz
Department of Psychology
University of California - Riverside
900 University Avenue
Riverside, CA 92521
Tel. (951) 827-6422, Fax. (951) 827-3985


Bhanu, Bir
Seitz, Aaron
Carrillo , Audrey
Li, Runze
Rakesh Kumar, Ankith Jain
Aguayo , Laura
Blencowe , Kristin
Cheung, Sierra
Huang , Meiyu
Lara-Alejandro , Cindy
Molina, Steven

Facial expressions play a significant role in everyday communication among humans. Computer understanding of these complex and subtle expressions will lead to highly capable interactive cyber-human systems with proactive computers that make more appropriate responses to human interactions. This award brings together an interdisciplinary team of investigators to address key challenges associated with spontaneous microexpression recognition in non-social scenarios. The project concentrates on generating bio-feedback from humans while learning skills, such as game playing and online learning, and being recorded and analyzed in continuous color and depth video streams. It will develop computer algorithms for human-machine synergy and test how this information can provide for superior learning when training applications are augmented with expression-informed bio-feedback in near real-time. This represents a significant step forward in training machines to recognize and classify facial microexpressions and maximizing the synergy of cyber-human systems that will improve the quality of life experiences. Understanding complex and subtle human facial expressions as captured in continuous video streams will have a profound impact on human-computer interaction. It will provide a computing environment within the reach of common people in which the interests or even the health of people can be detected and predicted, with significant impacts on skill learning, education and information retrieval.

The project develops a transformative approach to the understanding of complex and subtle facial microexpressions and bio-feedback where the synergy between cyber and human systems can be fully exploited. It addresses key challenges associated with computational understanding and modeling of intelligence in challenging, realistic contexts. It uses assessment and intervention based on facial microexpressions to maximize synergy of cyber and human systems for skill learning. First, it considers deep learning and closed-loop video analysis for optimized skill learning in a reinforcement learning framework. Second, it develops novel representation of facial microexpressions from color and depth video streams and use them for person independent emotion recognition as well as person-specific emotions recognition when a game play is adapted. Third, it exploits not only the color camera but also the integrated depth camera for precise measurements, which has not been used for microexpressions. The focus is to determine the extent to which real-time classification of microexpressions can provide for more appropriate interactivity that will facilitate human learning in real applications. The results will be broadly disseminated through a website that will have regular releases of databases and software tools by offering tutorials, workshops and demos at major professional meetings.


  • A.J.R. Kumar, B. Bhanu, C. Casey, S.C. Cheung and A. Seitz, “Depth videos for the classification of micro-expressions,” International Conference on Pattern Recognition, Milan, Italy, January 10-15, 2021. (submitted 7/15/2020) Link
  • W. Liu, R. Li, M. Zheng, S. Karanam, Z. Wu, B. Bhanu, R.J. Radke, O. Camps, “Towards visually explaining variational autoencoders,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, June 14-19, 2020. Link
  • Molina, Steven; Blencowe, Kristin; Huang, Meiyu; Lara-Alejandro, Cindy; Caldera, Laura; Seitz, Aaron (2020). Assessing Training Stimuli for an Emotion Recognition Neural Net. University of California, Riverside Undergraduate Research Symposium. Also at University of California, Riverside R'Psych Conference. University of California, Riverside. Link
  • Cheung, S.; Caldera-Aguayo, L; Seitz, A. (October, 2020). (2020). Emotional expressivity differences by gender using the facial action coding system and machine learning. 100th Western Psychological Association. San Francisco, CA. Link

RGBD Microexpressions Dataset in Social Context: We collected a dataset from 29 healthy adults, recruited from the UCR student population (mean age = 20.79 years; 18 females) using an RGB-D camera, while participants watched a series of videos directed to elicit emotions of different types (happy, surprise, fear, anger, disgust, contempt, and sadness). Videos have been hand-scored by at least 2 research assistants each and then results inspected and verified by a 3rd. We have used this dataset in our publication.

RGBD Microexpressions Dataset in Non-Social Context for Working Memory Training: We have also collected a dataset recorded while participants conducted a 10 session working memory training. We have 25 participants who conducted this study and we are examining the extent to which microexpressions predict either performance and/or changes in performance across time. Further we are comparing whether similar classes of microexpressions are used in this non-social dataset compared to the dataset collected in the social context. Currently, this dataset is being curated.

We plan to increase the number of subjects for these two datasets. These datasets and the corresponding software will be released in the future.