Silvia Izquierdo Badiola, Gerard Canal, Carlos Rizzo, Guillem Alenyà

Abstract: “Hey, robot. Let's tidy up the kitchen. By the way, I have back pain today". How can a robotic system devise a shared plan with an appropriate task allocation from this abstract goal and agent condition? Classical AI task planning has been explored for this purpose, but it involves a tedious definition of an inflexible planning problem. Large Language Models (LLMs) have shown promising generalisation capabilities in robotics decision-making through knowledge extraction from Natural Language (NL). However, the translation of NL information into constrained robotics domains remains a challenge. In this paper, we use LLMs as translators between NL information and a structured AI task planning problem, targeting human-robot collaborative plans. The LLM generates information that is encoded in the planning problem, including specific subgoals derived from an NL abstract goal, as well as recommendations for subgoal allocation based on NL agent conditions. The framework, PlanCollabNL, is evaluated for a number of goals and agent conditions, and the results show that correct and executable plans are found in most cases. With this framework, we intend to add flexibility and generalisation to HRC plan generation, eliminating the need for a manual and laborious definition of restricted planning problems and agent models.

Paper: Open pdf file.



PlanCollabNL: an LLM is used to generate and filter subgoals from an abstract goal and the current environment (1), and later to reason about the subgoal allocation among the agents, favouring or disfavouring subgoals for each agent based on their conditions (2). This information is encoded into the PDDL planning problem, translating the LLM output subgoals into PDDL goals, and modifying the action costs based on the LLM recommendations. The complete planning problem is given to a planner to produce grounded and executable collaborative plans that consider the agent states (3).

Videos

ICRA24 Presentation Video

The following video describes the important concepts of the paper, as presented at ICRA 2024.

Video download here.

Diagrams

Subgoal Generation and Allocation

From Natural Language to Structured Planning Language PDDL

Poster

Explanatory poster presented at ICRA24: Open pdf file.

Datasets

Planning subgoals generation from an abstract NL goal

The dataset used to evaluate the generation of planning subgoals from an abstract NL goal can be found here: Open csv file.

Subgoal allocation reasoning based on NL agent conditions

The dataset used to evaluate the plan generation with appropriate subgoal allocation in the plan based on NL agent conditions can be found here: Open csv file.

Planning Domain and Problem

The PDDL planning domain and problems defined for the system can be found here: Open domain file. Open problem file.

Note that the domain is fixed for all scenarios in the application. The problem goal and the functions related to the action costs are determined by the LLM for each specific scenario.

LLM Prompts

The prompts designed for the system can be found here. We utilise n-shot learning, where the prompt includes n previous examples. For each new scenario, the templated query (found after the examples) is completed by the system with the current situation.

Planning subgoals generation from an abstract NL goal

Subgoal generation prompt
Query:
Goal: {goal}
Predicates: {predicates}
Objects: {objects}
Locations: {locations}
Subgoals:

Subgoal filter prompt - common-sense
Query:
A {object} is {predicate} in {location}. Does it make sense?

Subgoal filter prompt - goal contribution
Query:
The goal is: {goal}. Would the following subgoal contribute to the goal? Answer yes or no.
{subgoal}

Subgoal allocation reasoning based on NL agent conditions

Subgoal favour/disfavour prompt
Query:
I have the following tasks:
{subgoal}
{subgoal}
The agent {condition}.
The agent should {favour/disfavour} these tasks:

Code

Coming soon.