Contact-Guided Exploration for Non-Prehensile Locomanipulation

Abstract

Non-prehensile manipulation offers versatile skills for moving and rearranging heavy or bulky objects, particularly when combined with a mobile manipulation platform. However, both model-based and model-free approaches struggle with the complex hybrid dynamics and the sparsity of the contact in these tasks.

To address these challenges, we propose a contact-guided exploration strategy implemented within a Multi-Critic Reinforcement Learning (RL) framework. A dedicated exploration critic is trained with a dense contact-seeking reward that guides the end-effector toward meaningful contact points; its influence is progressively decayed to recover a task-optimal policy. We obtain candidate interaction points from a general-purpose grasping algorithm, enabling the exploration mechanism to generalise across various object geometries.

We evaluate the approach on multiple tasks, including box pushing, chair transportation, and a dishwasher opening task. Finally, we validate the chair transportation policy through extensive experiments on a quadrupedal mobile manipulator, demonstrating deployable non-prehensile manipulation in the real world.

Tip: Hover over diagram elements to read more.

Exploration Bottleneck in Non-Prehensile Locomanipulation

Non-prehensile manipulation depends on establishing, and often sustaining, unilateral contact, a rare event under random exploration. In model-free RL, the task reward $r_t^{\text{task}}$ stays at zero until contact happens, so a standard single-critic PPO hits an exploration bottleneck: smoothness and energy penalties dominate early gradients, and the agent settles into a static local minimum before ever finding contact.

Common fixes rely on task-specific shaping or demonstration data, which scale poorly across object geometries. We instead bias exploration toward informative regions of the interaction space, without hand-designing that bias per task or object, and without permanently distorting the objective the agent ends up optimising.

Contributions

A contact-guided exploration strategy implemented as a decay schedule over a dedicated exploration critic head, guiding the end-effector toward meaningful contact points before phasing out in favour of a task-optimal policy.
A systematic evaluation on box pushing and chair transportation, with a qualitative study on dishwasher opening.
Hardware validation of chair transportation on the ALMA quadrupedal mobile manipulator, showing robustness to varying object geometries and masses.

Multi-Critic Reinforcement Learning Formulation

Rather than combining heterogeneous objectives into a single scalar reward, we use a Multi-Critic PPO architecture. The environment returns a vector-valued reward decomposed into three streams — task progress, exploration guidance, and regularisation:

$$\mathbf{r}_t = \begin{bmatrix} r_t^{\text{task}} & r_t^{\text{exp}} & r_t^{\text{reg}} \end{bmatrix}^T$$

Each stream is estimated by its own value head sharing a common LSTM backbone, and their advantages are combined with independently scheduled weights:

$$A_t = w_{\text{task}} A_t^{\text{task}} + w_{\text{exp}}(t)\, A_t^{\text{exp}} + w_{\text{reg}}(t)\, A_t^{\text{reg}}$$

Separating the value functions this way keeps contact-seeking exploration from tangling with task performance, so each can be weighted and later decayed on its own terms.

Contact Candidates & Scheduled Decay

Candidate interaction points are proposed by a general-purpose grasping algorithm applied to the object mesh. At each episode reset, one candidate is sampled as a target for the end-effector. The exploration weight $w_{\text{exp}}(t)$ is annealed down over training, so the policy discovers contact early and then shifts to task-optimal physics without being anchored to a fixed contact point.

Results

The full method exceeds 90% success on both simulated tasks, cutting missed-contact and tipover failures relative to single-critic and fixed-weight baselines. On hardware, the chair-transportation policy transfers zero-shot to unseen IKEA furniture, recovers from failed contact attempts, and stays stable under external disturbances and payloads beyond the manipulator's rated static capacity.

Paper

For full hyperparameter tables, network architecture diagrams, and results, please refer to the paper.

Download Paper PDF

BibTeX

@article{tolomei2025contactguided,
  author    = {Tolomei, Simone and Mittal, Mayank and Angelini, Franco and Garabini, Manolo and Salaris, Paolo and Hutter, Marco},
  title     = {Contact-Guided Exploration for Non-Prehensile Locomanipulation with Multi-Critic RL},
  journal   = {IEEE Robotics and Automation Letters},
  year      = {2025},
}