COMPUTATIONAL RESEARCH in BOSTON and BEYOND (CRIBB)

What would it mean for AI to understand skilled human activity? In augmented reality (AR), a person wearing smart glasses could quickly pick up new skills with a virtual AI coach that provides real-time guidance. In robot learning, a robot watching people in its environment could acquire new dexterous manipulation skills with less physical experience. Realizing this vision demands significant advances in video understanding, in terms of the degree of detail, viewpoint flexibility, and proficiency assessment. In this talk I’ll present our recent progress tackling these challenges. This includes 4D models to anticipate human activity in long-form video; video-language capabilities for generating fine-grained descriptions and constructive commentary; and cross-view representations able to bridge the exocentric-egocentric divide---from the view of the teacher to the view of the learner. I’ll also illustrate the impact of these ideas for AI coaching prototypes that guide users through new skills or provide feedback on their physical performance, transforming how-to videos into personalized AI assistants.