Project Description

Project PID2023-151184OB-I00 funded by MCIN/ AEI /10.13039/501100011033 and by ERDF, UE

The GRAvatar project is a cutting-edge research initiative aimed at advancing the field of individual and collective human digitization. The primary objective is to develop innovative algorithms and methodologies that enable the creation and manipulation of high-fidelity 3D avatars in uncontrolled and challenging real-world scenarios. The project is structured into several tasks, each addressing specific challenges in the realm of human digitization.

Task 1 focuses on enhancing 3D human digitization in uncontrolled settings. The team aims to overcome the limitations of existing methods by drawing inspiration from recent advancements in generative modeling, particularly denoising diffusion probabilistic models. These models have shown success in 2D image generation, and the project seeks to extend these capabilities to the realm of 3D reconstruction.

Task 2, "Bridging the Gap in 3D-Aware Editing Tools," addresses the limitations in controlling the content of generated 3D avatars. The project aims to develop conditioning schemes for diffusion-based approaches to enforce geometry consistency during the generation process. Simultaneously, the team will explore novel approaches for editing pre-trained Neural Radiance Fields, offering improved controllability in 3D avatar generation.

Task 3 focuses on learning precise human dynamics. The objective is to advance approaches for fine-grained modeling of human pose and motion by harnessing natural language descriptions. The team aims to establish meaningful connections between human pose and motion descriptions and underlying body geometry, with applications in personalized coaching, physical therapy, and novel image retrieval approaches based on natural language.

Task 4, "Modeling Interaction with the Environment," delves into how contextual information influences the digital avatar. The team will develop novel approaches to address challenges such as inferring human shape under clothing, intrinsic decomposition of avatar appearance, and modeling the impact of the environment on human motion.

Task 5 addresses uncertainty in modeling, incorporating uncertainty models into each task to enhance robustness in non-controlled environments. The team will explore approaches such as normalizing flows for implicit functions and leverage Bayesian Transformers for modeling uncertainties in human motion dynamics.

Task 6 focuses on resource-efficient algorithms for accessibility. The team aims to minimize computational resources by considering strategies such as synthetic datasets and theoretical advancements in neural rendering and diffusion models.

The methodology involves a systematic approach to each task, combining innovative techniques, established methodologies, and rigorous experimentation. The team emphasizes the development of algorithms that demand minimal computational resources, ensuring efficiency and applicability in diverse scenarios. The main objectives we pursue are commercially and socially relevant technologies, as endorsed by our EPOs from the health (Crisalix), content production (Kognia and Arquimea) and automotive (IDIADA) industries. Overall, the GRAvatar project represents a forward-looking exploration into the future of human digitization, where advancements in algorithms and methodologies have the potential to reshape our interaction with digital representations of individuals and groups in diverse and uncontrolled environments.