Human-Computer Interaction Institute Thesis Proposal

  • Remote Access - Zoom
  • Virtual Presentation - ET
Thesis Proposals

Agency in AAC

Augmentative and alternative communication (AAC) devices enable speech-based communication, but generating speech is not the only resource needed to have a successful conversation. Augmented communicators (ACs) who use AAC devices to speak also face conversational constraints beyond their device and individual abilities,  such as when they can speak, what they can say, and who they can address.  This doctoral thesis presents conversational agency as a new frame to study AAC and investigates different design materials that can enable ACs to easily exercise their conversational agency.

To introduce conversational agency in AAC as a design framework for AAC I first carried out a study investigating AC's conversational agency with different conversational tasks between four triads of expert ACs, their close conversation partners, and a third party. I defined metrics to analyze AAC conversational agency quantitatively and qualitatively and outline specific conversational and  AAC constraints that can impact an AC's agency.  For example, ACs participate more when addressed directly by their partners (participation-shift constraint), may lose opportunities to say what they want to say due to fast-changing topics (relevance constraint), ACs work together with their close conversation partners to communicate with others (ritual constraint), and may get interrupted while composing a message as people may not perceive they are typing (one-speaker constraint). In addition to these conversational constraints, ACs communicated differently based on how close they were to their conversation partners (relationship constraint) and expressed their agency differently depending on their device set up and the conversational task at hand (device and task constraints).
Identifying the factors that impact ACs' agency led to identifying design opportunities to address existing constraints. Technologies that can help balance participation in a group and signal to partners that an AC is still making use of their turn could help address the participation-shift and one-speaker constraint. Technologies that can retrieve content from a conversation and clarify misunderstandings could provide ACs with more control over the relevance and ritual constraints. I take a research through design approach to examine how these different technologies may be used to give ACs more control over the conversational constraints that impact their agency. I begin by presenting a co-design workshop exploring how motion displayed through socially assistive robots, framed as ``sidekicks,'' might provide ACs with a nonverbal channel of communication.  I complement this investigation by designing and evaluating a working sidekick with an AC co-designer to understand how sidekicks can support conversational agency in face-to-face and online communication.

In my proposed work, I will explore how to loosen the relevance and ritual constraints using natural language processing and machine learning techniques using a four-step design process. So far I have discovered opportunities for NLP to support context-retrieval that can help in connecting an AC's messages to the existing context in the conversation. I will continue this exploration by generating a set of storyboards and evaluating them with ACs to define design guidelines for a potential prototype. I will then use these findings to develop and evaluate a prototype of a working system in a user study to understand how it might impact the expression of ACs agency when talking with different partners.

Thesis Committee:
Jeffrey P. Bigham Co-Chair)
Henny Admoni (Co-Chair)
Jodi Forlizzi
Jeffrey Higginbotham (State University of New York at Buffalo)

Zoom Participation. See announcement.

For More Information, Please Contact: