Selected projects
Spoken dialog with a virtual counselor
Our virtual health coaches deliver low-cost engaging effective motivational health interventions, accessible via the Internet or mobiles, to help people participate in their own healthcare and wellbeing. They adapt to current needs of diverse populations, and increase access to effective health interventions for people without such access.
Our virtual health coaches are 3-dimensional virtual agents who use their speaking abilities, dynamic expressive animations of subtle facial expressions and gestures to engage people in dialogs focused on healthy lifestyles. The dialog system processes the persons’ spoken answers and controls the flow of the conversation based on effective motivational healthcare interventions.
Our virtual health coaches can deliver a variety of interventions that educate people about what can place them at-risk (obesity, excessive alcohol consumption, unsafe sex). The virtual health coaches provide people with personalized feedback and advice about their health over multiple interactions based on their evolving readiness to change.
You can see Lola in action by clicking on this video.
Our virtual health coaches deliver low-cost engaging effective motivational health interventions, accessible via the Internet or mobiles, to help people participate in their own healthcare and wellbeing. They adapt to current needs of diverse populations, and increase access to effective health interventions for people without such access.
Our virtual health coaches are 3-dimensional virtual agents who use their speaking abilities, dynamic expressive animations of subtle facial expressions and gestures to engage people in dialogs focused on healthy lifestyles. The dialog system processes the persons’ spoken answers and controls the flow of the conversation based on effective motivational healthcare interventions.
Our virtual health coaches can deliver a variety of interventions that educate people about what can place them at-risk (obesity, excessive alcohol consumption, unsafe sex). The virtual health coaches provide people with personalized feedback and advice about their health over multiple interactions based on their evolving readiness to change.
You can see Lola in action by clicking on this video.
Since almost all psychosocial interventions on the web and other electronic modalities are delivered via text (or at best video), using supportive virtual health coaches is a novel modality for healthcare interventions. For many, virtual health coaches who converse with people over the long term about their intimate health issues and provide them with effective personalized counseling is still part of science fiction. This research has the potential to transform access to and delivery of effective healthcare interventions for diverse populations.
The research involved the integration of computer vision, dialog management, and dynamic animations for 3D virtual characters. The agent performed real-time recognition of the user’s facial expressions and speech during the dialog. Using reinforcement learning, the dialog manager controlled the flow of the conversation based on the user’s spoken responses to an effective motivational behavior change intervention. The agent used a novel computational model of nonverbal communication to decide what expressions and gestures to portray as it engaged the user on at-risk behaviors.
Our virtual coaches demonstrated that they can lower dropout rates. Our virtual health coaches are effective in increasing people’s ongoing interest in computer-based health interventions, thereby increasing their ability to change toward healthy lifestyles. Our virtual health coaches can help address the nation epidemic of behavioral issues, and help people find and maintain lifestyles.
This project is further explained in the following articles:
The research involved the integration of computer vision, dialog management, and dynamic animations for 3D virtual characters. The agent performed real-time recognition of the user’s facial expressions and speech during the dialog. Using reinforcement learning, the dialog manager controlled the flow of the conversation based on the user’s spoken responses to an effective motivational behavior change intervention. The agent used a novel computational model of nonverbal communication to decide what expressions and gestures to portray as it engaged the user on at-risk behaviors.
Our virtual coaches demonstrated that they can lower dropout rates. Our virtual health coaches are effective in increasing people’s ongoing interest in computer-based health interventions, thereby increasing their ability to change toward healthy lifestyles. Our virtual health coaches can help address the nation epidemic of behavioral issues, and help people find and maintain lifestyles.
This project is further explained in the following articles:
Ontology for Behavioral Health
Named-Entity Recognizers (NERs) are an important part of information extraction systems in annotation tasks. Although substantial progress has been made in recognizing domain-independent named entities (e.g. location, organization and person), there is a need to recognize named entities for domain-specific applications in order to extract relevant concepts.
Due to the growing need for smart health applications in order to ad- dress some of the latest worldwide epidemics of behav- ioral issues (e.g. over eating, lack of exercise, alcohol and drug consumption), we focused on the domain of behavior change, especially lifestyle change.
We have developed the first named-entity recognizer designed for the lifestyle change domain. It aims at enabling smart health applications to recognize relevant concepts.
We created an ontology for behavioral health based on which we developed an NER augmented with lexical resources. Our NER automatically tags words and phrases in sentences with relevant (lifestyle) domain-specific tags. For example, it can tag:
Named-Entity Recognizers (NERs) are an important part of information extraction systems in annotation tasks. Although substantial progress has been made in recognizing domain-independent named entities (e.g. location, organization and person), there is a need to recognize named entities for domain-specific applications in order to extract relevant concepts.
Due to the growing need for smart health applications in order to ad- dress some of the latest worldwide epidemics of behav- ioral issues (e.g. over eating, lack of exercise, alcohol and drug consumption), we focused on the domain of behavior change, especially lifestyle change.
We have developed the first named-entity recognizer designed for the lifestyle change domain. It aims at enabling smart health applications to recognize relevant concepts.
We created an ontology for behavioral health based on which we developed an NER augmented with lexical resources. Our NER automatically tags words and phrases in sentences with relevant (lifestyle) domain-specific tags. For example, it can tag:
- healthy food
- unhealthy food
- potentially-risky
- healthy activity,
- drugs,
- tobacco,
- alcoholic beverages.
Our ontology enables computer systems and smart health applications to derive further information for recognized named entities using semantic reasoners.
You can learn more about our NER and ontology for lifestyles by reading this article:
Yasavur U, Amini R, Lisetti C, Rishe N: Ontology-based Named Entity Recognizer for Behavioral Health. In: Proceedings of FLAIRS 2013. Association for the Advancement of Artificial Intelligence (www.aaai.org), 2013 [.PDF].
You can learn more about our NER and ontology for lifestyles by reading this article:
Yasavur U, Amini R, Lisetti C, Rishe N: Ontology-based Named Entity Recognizer for Behavioral Health. In: Proceedings of FLAIRS 2013. Association for the Advancement of Artificial Intelligence (www.aaai.org), 2013 [.PDF].
On-Demand Virtual Counselor for Motivational Dialogs on Lifestyles
There is a growing societal need to address the increasing prevalence, and even epidemics, of lifestyle behavioral health issues such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. These behavioral issues place people at risk of serious diseases. For example, obesity can lead to diabetes, alcoholism to cirrhosis, physical inactivity to heart disease.
In this project, we develop a novel modality for the computer-delivery of behavior change.
On-Demand VIrtual Counselor (ODVIC), accessed over the internet, is a multimodal Embodied Conversational Agent (ECA) that empathically delivers an evidence-based behavior change intervention. ODVIC adapts its verbal and nonverbal messages to the person's answers and emotional cues.
We currently focus our work on excessive alcohol consumption as a target behavior. Our approach is based on the DCU, a successful existing patient-centered brief motivational intervention for behavior change. With a simple text-only interface, the Drinker’s Check-Up(DCU) has been found effective in reducing alcohol consumption in problem drinkers by 50% at 12-month follow-up.
Our studies show a 31% increase in users' intention to reuse the intervention when the content of the DCU intervention is delivered by our ODVIC, compared to users' intention to reuse the text-only DCU.
Our work continues toward our long-term goals of on-demand conversations with virtual agents as personal health and well-being helpers for people in need. Our approach is also adaptable to other target behaviors (e.g., overeating, lack of exercise, narcotic drug use, non-adherence to treatment).
Our work is described in the following article: C. L. Lisetti, R. Amini, U. Yasavur, and N. Rishe (2013). I Can Help You Change! An Empathic Virtual Agent Delivers Behavior Change Health Interventions. ACM Transactions on Management Information Systems, Vol. 4, No. 4, Article 19, 2013 [pdf].
There is a growing societal need to address the increasing prevalence, and even epidemics, of lifestyle behavioral health issues such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. These behavioral issues place people at risk of serious diseases. For example, obesity can lead to diabetes, alcoholism to cirrhosis, physical inactivity to heart disease.
In this project, we develop a novel modality for the computer-delivery of behavior change.
On-Demand VIrtual Counselor (ODVIC), accessed over the internet, is a multimodal Embodied Conversational Agent (ECA) that empathically delivers an evidence-based behavior change intervention. ODVIC adapts its verbal and nonverbal messages to the person's answers and emotional cues.
We currently focus our work on excessive alcohol consumption as a target behavior. Our approach is based on the DCU, a successful existing patient-centered brief motivational intervention for behavior change. With a simple text-only interface, the Drinker’s Check-Up(DCU) has been found effective in reducing alcohol consumption in problem drinkers by 50% at 12-month follow-up.
Our studies show a 31% increase in users' intention to reuse the intervention when the content of the DCU intervention is delivered by our ODVIC, compared to users' intention to reuse the text-only DCU.
Our work continues toward our long-term goals of on-demand conversations with virtual agents as personal health and well-being helpers for people in need. Our approach is also adaptable to other target behaviors (e.g., overeating, lack of exercise, narcotic drug use, non-adherence to treatment).
Our work is described in the following article: C. L. Lisetti, R. Amini, U. Yasavur, and N. Rishe (2013). I Can Help You Change! An Empathic Virtual Agent Delivers Behavior Change Health Interventions. ACM Transactions on Management Information Systems, Vol. 4, No. 4, Article 19, 2013 [pdf].
HapFACS Open Source software/API for generating FACS-based facial expressions on 3D conversational agents
Our HapFACS open source software for facial expression generation is based on Ekman's (1992) Facial Action Coding Systems (FACS) - the current standard on facial coding of the human face - enables:
We already have over 30 research labs using our HapFACS open source software and API. Here's a link to the project site, with introductory and tutorial videos, and its forum.
Our work is further described in the following article:
R. Amini, C. L. Lisetti (2013) HapFACS: an Open Source API/Software to Generate FACS-Based Expressions for Embodied Conversational Agents Animation and for Corpus Generation. In Proceedings of the Affective Computing and Intelligent Interaction Internal Conference (ACII) , (Geneva, SWITZTERLAND, September, 2013).
Our HapFACS open source software for facial expression generation is based on Ekman's (1992) Facial Action Coding Systems (FACS) - the current standard on facial coding of the human face - enables:
- researchers on 3-dimensional animated characters - without expertise in computer graphics nor in FACS - to design a set of FACS-based facial expressions on the Haptek virtual character platform and to integrate their expressive avatars in their applications; and
- researchers on facial expression recognition to experiment with realistically generated expressions at the level of 49 individual Action Units (AUs), including bilateral (49 AUs) and unilateral (14 AUs).
We already have over 30 research labs using our HapFACS open source software and API. Here's a link to the project site, with introductory and tutorial videos, and its forum.
Our work is further described in the following article:
R. Amini, C. L. Lisetti (2013) HapFACS: an Open Source API/Software to Generate FACS-Based Expressions for Embodied Conversational Agents Animation and for Corpus Generation. In Proceedings of the Affective Computing and Intelligent Interaction Internal Conference (ACII) , (Geneva, SWITZTERLAND, September, 2013).
Signing Avatars
In this project, we design 3D Virtual Agents who can dynamically sign in American Sign Language (ASL).
We partner with the Institute for Disabilities Research and Training (IDRT) to generate real-time animations of 3D characters during real-time motion capture of human signers ASL gestures and facial expressions.
In this project, we design 3D Virtual Agents who can dynamically sign in American Sign Language (ASL).
We partner with the Institute for Disabilities Research and Training (IDRT) to generate real-time animations of 3D characters during real-time motion capture of human signers ASL gestures and facial expressions.
|
|
The goal of the project is to integrate these animations in IDRT Sign Generator, an English-to-ASL and ASL-to-English translation engine. Sign Generator’s robust database pairs specific signs - represented as graphics and video clips, and now 3D-characters - with over 30,000 English words, idioms, phrases, numbers, and symbols.
Emotion recognition from physiological bio-signals - Driver safety project
In this project, we simulate driving scenes and scenarios in a 3D virtual reality immersive environment to sense and interpret drivers' affective states (e.g. anger, fear, boredom).
Drivers wear non invasive bio-sensors while operating vehicles with haptic feedback (e.g. break failure, shaking with flat tire) and experience emotionally loaded 3D interactive driving events (e.g. driving in frustrating delayed New-York traffic, arriving at full speed in a blocked accident scene with failed breaks and pedestrian crossing).
Our results are discussed in our article on drivers' safety: F. Nasoz, C. Lisetti, A. Vasilakos. Affective intelligent and Adaptive Car Interfaces. Information Sciences, Vol. 180: 3817-3836. [.pdf]
In this project, we simulate driving scenes and scenarios in a 3D virtual reality immersive environment to sense and interpret drivers' affective states (e.g. anger, fear, boredom).
Drivers wear non invasive bio-sensors while operating vehicles with haptic feedback (e.g. break failure, shaking with flat tire) and experience emotionally loaded 3D interactive driving events (e.g. driving in frustrating delayed New-York traffic, arriving at full speed in a blocked accident scene with failed breaks and pedestrian crossing).
Our results are discussed in our article on drivers' safety: F. Nasoz, C. Lisetti, A. Vasilakos. Affective intelligent and Adaptive Car Interfaces. Information Sciences, Vol. 180: 3817-3836. [.pdf]
|
|
|
Cherry, the Little Red Robot with a Personality
Cherry, the Little Red Robot with a Personality (shown on the left), was one of the first fully integrated mobile robotic systems to test human-robot interaction in social contexts.
We built Cherry to provide guided tours of our Computer Science suite to visitors. Cherry had a map of the office floor, she could recognize faculty with machine vision and talk to visitors about each faculty's research. She would also get frustrated - and showed it - if she kept trying to find a professor whose door would always be closed…
We evaluated the reactions of people before they met Cherry and after they met her: the more people interacted with her, the more they liked her. When the project ended, many people told us they missed her roaming around our Computer Science floor, and asked if we could bring her back.
Cherry helped pave the way for further research in social robotics, and establish Human-Robot Interaction (HRI) as a necessary new field of study. Indeed, Human-Robot Interaction raises research questions that are different from Human-Computer Interaction and that need to be addressed accordingly.
You can read about our results in this following article:
C. Lisetti, S. Brown, K. Alvarez, A. Marpaung. A Social Informatics Approach to Human-Robot Interaction with a Service Social Robot. IEEE Transactions on Sytems, Man and Cybernetics, Vol. 34, N. 2: 195-209, 2004.
Cherry, the Little Red Robot with a Personality (shown on the left), was one of the first fully integrated mobile robotic systems to test human-robot interaction in social contexts.
We built Cherry to provide guided tours of our Computer Science suite to visitors. Cherry had a map of the office floor, she could recognize faculty with machine vision and talk to visitors about each faculty's research. She would also get frustrated - and showed it - if she kept trying to find a professor whose door would always be closed…
We evaluated the reactions of people before they met Cherry and after they met her: the more people interacted with her, the more they liked her. When the project ended, many people told us they missed her roaming around our Computer Science floor, and asked if we could bring her back.
Cherry helped pave the way for further research in social robotics, and establish Human-Robot Interaction (HRI) as a necessary new field of study. Indeed, Human-Robot Interaction raises research questions that are different from Human-Computer Interaction and that need to be addressed accordingly.
You can read about our results in this following article:
C. Lisetti, S. Brown, K. Alvarez, A. Marpaung. A Social Informatics Approach to Human-Robot Interaction with a Service Social Robot. IEEE Transactions on Sytems, Man and Cybernetics, Vol. 34, N. 2: 195-209, 2004.
AAAI Robot Competition: Hors d'Oeuvres Anyone?
We won the Nils Nilsson Award for Integrating AI Technologies and the Technical Innovation Award when we competed at the AAAI Robot Competition: Hors D'oeuvre Anyone at the National Artificial Intelligence Conference in 2000 organized by the Association for the Advancement of Artificial Intelligence (AAAI). The task was to use robots to entertain, engage and serve hors d'oeuvres at the conference main reception, to as many guests as possible - without running them over!
We were the first team to introduce a pair of collaborating robots - Butler (taller, shown on the left) and its assistant (right): Butler moved around the crowd offering hors d'oeuvres on its tray; a laser sensor enabled it to know when a treat was taken, hence to determine when the tray was getting low on food and needed refilling. Sonars enabled them to avoid obstacles such as guests... The assistant stood by the hors d'oeuvres refill station until called by Butler to bring a new full tray for tray exchange.
Moreover, they were designed with an emotion-based three-layer architecture which simulated some of the roles of emotions during human decision-making: e.g. if the assistant was too slow, Butler's frustration would increase (and be expressed) with time, until it decided to get the tray itself. If you're wondering about the armadillo theme, the competition was in Texas...
You can this article to learn about how we implemented this work: R. Murphy, C. Lisetti, R. Tardif, L. Irish, and A. Gage. Emotion-Based Control of Cooperating Heterogeneous Mobile Robots. IEEE Transactions on Robotics and Automation, Vol. 18, No. 5, October 2002.
We also won a red Amigobot robot, which led to our Cherry robot project (shown above).
We won the Nils Nilsson Award for Integrating AI Technologies and the Technical Innovation Award when we competed at the AAAI Robot Competition: Hors D'oeuvre Anyone at the National Artificial Intelligence Conference in 2000 organized by the Association for the Advancement of Artificial Intelligence (AAAI). The task was to use robots to entertain, engage and serve hors d'oeuvres at the conference main reception, to as many guests as possible - without running them over!
We were the first team to introduce a pair of collaborating robots - Butler (taller, shown on the left) and its assistant (right): Butler moved around the crowd offering hors d'oeuvres on its tray; a laser sensor enabled it to know when a treat was taken, hence to determine when the tray was getting low on food and needed refilling. Sonars enabled them to avoid obstacles such as guests... The assistant stood by the hors d'oeuvres refill station until called by Butler to bring a new full tray for tray exchange.
Moreover, they were designed with an emotion-based three-layer architecture which simulated some of the roles of emotions during human decision-making: e.g. if the assistant was too slow, Butler's frustration would increase (and be expressed) with time, until it decided to get the tray itself. If you're wondering about the armadillo theme, the competition was in Texas...
You can this article to learn about how we implemented this work: R. Murphy, C. Lisetti, R. Tardif, L. Irish, and A. Gage. Emotion-Based Control of Cooperating Heterogeneous Mobile Robots. IEEE Transactions on Robotics and Automation, Vol. 18, No. 5, October 2002.
We also won a red Amigobot robot, which led to our Cherry robot project (shown above).
I have recently started a collaboration with the FIU Discovery Lab on mobile assistive tele-operated robots. The goal of the project is to study what type of interaction can a mobile robot tele-operated by a disabled law enforcement or police officer have with the public (e.g. apprehended individuals, standbys).
In this context, a disabled law enforcement or police officer can control the movements of the robot from a distance, and continue to contribute to society in spite of motor disabilities. The officer wears equipment feeding him or her high-definition video and audio data captured in real-time by sensors on the TeleBot. Seeing and hearing what the robot sees, the officer remotely controls the movements and actions of the robot to roam an area and engage with individuals for enhanced safety.
My specific research in this project involves studying and creating means for the tele-operated mobile robot to interact with the public in an ethical and acceptable manner in such as way that the public will be in favor of such assistive technologies.
In this context, a disabled law enforcement or police officer can control the movements of the robot from a distance, and continue to contribute to society in spite of motor disabilities. The officer wears equipment feeding him or her high-definition video and audio data captured in real-time by sensors on the TeleBot. Seeing and hearing what the robot sees, the officer remotely controls the movements and actions of the robot to roam an area and engage with individuals for enhanced safety.
My specific research in this project involves studying and creating means for the tele-operated mobile robot to interact with the public in an ethical and acceptable manner in such as way that the public will be in favor of such assistive technologies.
