Fixing robot plans with natural language feedback

Robots would benefit from the ability to incorporate natural language feedback to change their behavior. A recent paper on proposes to use natural language instructions as input to directly modify the planning purpose of the robot.

Giving instructions to the robot.

Giving instructions to the robot. Image credits: Amber Case, CC BY-NC 2.0 via Flickr

The researchers propose a learning model that maps visual observation and a natural language correction to a residual cost function.

Users can modify or clarify the purpose of the robot or introduce additional constraints to the motion optimization process at any time during execution. The framework is seamlessly integrated with commonly used motion planner costs, such as collision avoidance, joint limits and easiness.

Evaluation of the model in a variety of real-world settings incorporating non-templated natural user commands, cluttered sequences, and new object types demonstrated that the language interface can successfully correct local planner failures.

When humans design cost or target specifications for robots, they often produce specifications that are unclear, unspecified, or beyond the planners’ ability to solve. In these cases, the improvements provide a valuable tool for human-in-the-loop robot control. Improvements can take the form of new target specifications, new obstacles (eg specific objects to avoid), or prompts for planning algorithms (eg to go to specific waypoints). Existing correction methods (eg using joysticks or direct manipulation of the end effector) require full teleoperation or real-time interaction. In this paper, we explore natural language as an expressive and flexible tool for robotic improvement. We describe how to map from natural language sentences to transformations of cost functions. We show that these changes enable users to fix goals, update robot motions to accommodate additional user preferences, and recover from planning errors. These improvements can be leveraged to achieve 81% and 93% success rates on tasks where the original planner failed, with one or two language improvements. Our method makes it possible to create multiple constraints in simulated environments and real-world environments, and to generalize unseen scenes, objects, and sentences.

Research Article: Sharma, P., “Improving robot schemes with natural language feedback”, 2022. Link:

Leave a Reply

Your email address will not be published.