Assessing Self-awareness and Transparency when Classifying a Speaker’s Level of Certainty
Heather Pon-Barry, Stuart Shieber, Harvard University
This paper is about using prosody to automatically detect one aspect of a speaker's internal state: their level of certainty. While past work on classifying level of certainty used the perceived level of certainty as the value to predict, we find that this quantity often differs from a speaker's actual level of certainty as gauged by self-reports. In this work we build models to predict a speaker's self-reported level of certainty using prosodic features. Our data is a corpus of single-sentence utterances that are annotated with (1) whether the statement is correct or incorrect, (2) the perceived level of certainty, and (3) the self-reported level of certainty. Knowing the self-reported level of certainty, in conjunction with the perceived level of certainty, allows us to assess what we will refer to as the speaker's transparency. Knowing the self-reported level of certainty, in conjunction with the correctness of the answer, allows us to assess what we will refer to as self-awareness. Our models, trained on prosodic features, correctly classify the self-reported level of certainty 75\% of the time. Intelligent systems can use this information to make inferences about the user's internal state, for example whether the user of a system has a misconception, makes a lucky guess, or needs encouragement.