Authors: Jared Buabeng

arXiv: your arxiv link.

Abstract:
This thesis studies the challenge of enabling customization of robot motion behavior by non- expert users via natural language. These natural language instructions naturally refer to objects and semantic features of the environment that are not historically included in robot motion planning. To address this gap between non-expert human communication and robot motion planning, this thesis proposes a novel framework, Object-centric Reasoning for Behavior with Instruction-conditioned Trajectories (ORBIT) which utilizes scene graphs and Graph Neural Networks (GNNs) to map natural language instructions to motion planning constraints. By representing the environment as a graph, the model can identify and prioritize relevant objects for a given instruction, enabling more precise, context-aware navigation. Additionally, a Reinforcement Learning from Human Feedback (RLHF) pipeline is used to improve the robot's trajectories based on personal user preferences, balancing hard task constraints alongside soft human desires. Finally, in addition to the overall framework and model architecture, this work includes the creation of a novel dataset linking natural language instructions to scene graph annotations used to derive motion constraints. 

paste your bibtex citation here