Language Technologies Institute Colloquium

  • Posner Hall A35 and Zoom
  • In Person and Virtual ET
  • X Consortium Assistant Professor
  • Department of Electrical Engineering and Computer Science and CSAIL
  • Massachusetts Institute of Technology

Toward Natural Language Supervision

In the age of deep networks, "learning" almost invariably means "learning from examples". Image classifiers are trained with large datasets of (labeled or unlabeled) images, machine translation systems with corpora of translated sentences, and robot policies with demonstrations. When human learners acquire new concepts and skills, we often do so with richer supervision, especially in the form of language---we learn new concepts from exemplars accompanied by descriptions or definitions, and new skills from demonstrations accompanied by instructions. In natural language processing, recent years have seen a number of successful approaches to learning from task definitions and other forms of auxiliary language-based supervision. But these successes have been largely confined to tasks that also involve language as an input and an output---what will it take to make language-based training useful for the rest of the machine learning world?

In this talk, I'll present two recent applications of natural language supervision to tasks outside the traditional domain of NLP: using language to guide inductive program synthesis and visuomotor policy learning. In these applications, natural language annotations reveal latent compositional structure in the space of programs and plans, helping models discover reusable abstractions for processing data and interacting with the environment. This kind of compositional structure is present in many tasks beyond program synthesis and policy learning, and I'll conclude with a brief discussion of how these techniques might be more generally applied.

Jacob Andreas is the X Consortium Assistant Professor at MIT. His research aims to build intelligent systems that can communicate effectively using language and learn from human guidance. Jacob earned his Ph.D. from UC Berkeley, his M.Phil. from Cambridge (where he studied as a Churchill scholar) and his B.S. from Columbia. As a researcher at Microsoft Semantic Machines, he
founded the language generation team and helped develop core pieces of the technology that powers conversational interaction in Microsoft Outlook. He has been the recipient of a Sony Faculty Innovation Award, a Kolokotrones for teaching at MIT, and paper awards at NAACL and ICML.

The LTI Colloquium is generously sponsored by Abridge.

In Person and Zoom Participation. See announcement.

For More Information, Please Contact: