Description

This project aims to develop an intelligent AI agent to facilitate adaptive learning in human-robot interaction (HRI) contexts in which the robot has to adapt to both the environment and the human collaborator. The core of the agent will be a large language model (LLM), enabling natural communication, contextual understanding, and reasoning capabilities. The agent will integrate multimodal sensory inputs: visual cues processed via visual language models (VLMs), audio input from human speech, and sensor feedback from the robot’s arms.

The work is under the scope of the following projects:

• SOCIAL PIA: Cooperative Social PIA model for Cybernetics Avatars (Moonshot Research and Development Program) (web)
• LENA: Lifelong navigation learning using human-robot interaction (web)
• TORNADO: foundaTion mOdels for Robots that haNdle smAll, soft and Deformable Objects (web)