|
|
Research Directions
Overview
Human-centered signal processing (HCSP) is the science of decoding human behavior. HCSP is part of the larger emerging field of Behavioral Signal Processing (BSP). BSP seeks to provide a computational account of aspects of human behavior ranging from interaction patterns to individual emotion expression using techniques from both signal processing and machine learning. HCSP encompasses a subset of the BSP domain including emotion production and perception, the coordination of verbal and non-verbal behavior, turn-taking behavior, user preferences, and user judgments. Results from these sub-domains can be integrated to develop quantitative models of user state.
My current area of focus is in the quantitative analysis of emotion, a sub-domain of HCSP. The goals of this research are motivated by the complexities of human emotion perception. We seek to provide a computational account of how humans perceive emotional utterances ("emotion perception") and combine this with knowledge gleaned from perception estimation studies ("emotion recognition") to develop a system capable of interpreting naturalistic expressions of emotion utilizing a new quantification measure ("emotion profiles").
Emotion Quantification
Emotion profiles (EP) describe the emotions present in an utterance. This makeup is characterized not in black or white semantic labels (e.g, the speaker is angry), but instead through the estimation of the degree of presence or absence of multiple emotional components. These components can either be defined by conventional semantic labels (e.g., angry, happy, neutral, sad) or based on unsupervised clustering of the feature space. These representations, which we refer to as profiles, are a multi-dimensional description of the emotional makeup of an utterance.
Publications:
| o | Mower et al. | [soon] | "A Framework for Automatic Human Emotion Classification Using Emotional Profiles," 2010. |
| o | Mower et al. | [soon] | "Robust Representations for Out-of-Domain Emotions Using Emotion Profiles," 2010 |
| o | Mower et al. | [soon] | "A Cluster-Profile Representation of Emotion Using Agglomerative Hierarchical Clustering," 2010 |
| o | Mower et al. | [pdf] | "Interpreting Ambiguous Emotional Expressions," 2009. |
Emotion Classification Studies
Engineering models provide an important avenue through which to develop a greater understanding of human emotion. These techniques enable quantitative analysis of current theories, illuminating features that are common to specific types of emotion perception and the patterns that exist across the emotion classes. Such computational models can inform design of automatic emotion classification systems from speech, and other forms of emotion-relevant data.
Publications:
| o | Wu et al. | [pdf] | "Speech Emotion Estimation in 3D Space," 2010. |
| o | Mower et al. | [pdf] | "Evaluating Evaluators: A Case Study in Understanding the Benefits and Pitfalls of Multi-Evaluator Modeling," 2009. |
| o | Lee et al. | [pdf] | "Emotion recognition using a hierarchical binary decision tree approach," 2009. |
| o | Grimm et al. | [pdf] | "Primitives based estimation and evaluation of emotions in speech," 2007 |
| o | Grimm et al. | [pdf] | "Combining categorical and primitives-based emotion recognition," 2006. |
Perceptual Studies
The proper design of affective agents requires an understanding of human emotional perception. Such an understanding provides designers with a method through which to estimate how an affective interface may be perceived given intended feature modulations. However, human perception of naturalistic expressions is difficult to predict. This difficulty is partially due to the mismatch between the emotional cue generation (the speaker) and cue perception (the observer) processes and partially due to the presence of complex emotions, emotions that contain shades of multiple affective classes.
An understanding of the mapping between signal cue modulation and human perception can facilitate design improvements both for emotionally relevant and emotionally targeted expressions for use in human-computer and human-robot interaction. This understanding will further human-centered design, necessary for wide-spread adoption of this affective technology.
Publications:
| o | Mower et al. | [pdf] | Human Perception of Audio-Visual Synthetic Character Emotion Expression in the Presence of Ambiguous and Conflicting Information," 2009. |
| o | Mower et al. | [pdf] | "Selection of Emotionally Salient Audio-Visual Features for Modeling Human Evaluations of Synthetic Character Emotion Displays," 2008. |
| o | Mower et al. | [pdf] | Joint-processing of audio-visual signals in human perception of conflicting synthetic character emotions," 2008. |
| o | Mower et al. | [pdf] | "Human perception of synthetic character emotions in the presence of conflicting and congruent vocal and facial expressions," 2008. |
|
