Project Description

Project PID2023-151351NB-I00 funded by MCIN/ AEI /10.13039/501100011033 and by ERDF, UE

Temporal sequence encoding (TSE), the task of representing temporal sequences, is crucial for perception, action and learning, and it is considered a hallmark of human cognitive abilities. A fundamental aspect of TSE is that we perceive continuous temporal observations into discrete units, called events, and we use the temporal organization of events in our memory accumulated along a lifetime as prior knowledge to understand the present in the context of the past and to make inference about the future. Consequently, performing correctly TSE requires a very high level of temporal abstraction, that is, the idea of representing and reasoning about events, actions and changes at different levels of granularity and duration. Currently, this represents a major challenge for AI based systems.

The proposed methodological developments will be put to test on real-world data in the challenging computer vision tasks of temporal action segmentation, localization, and anticipation in untrimmed videos that require high level of temporal abstraction and would largely benefit of the use of prior knowledge about the unfolding of events. Given that Language, together with Vision, is a fundamental modality through which human beings acquire knowledge about the world to guide their future actions, we will leverage them to acquire prior knowledge.