Manipulation Is Intelligence in Contact

A friend who works on humanoids told me acrobatics is basically solved and manipulation is the hard part. The first claim is overstated; the second is right, and the reason is structural rather than a matter of effort or data. Acrobatics and manipulation are different problem classes, and the difference shows up in the math.

What "solved" means here

The strong version of "acrobatics is solved" is wrong. Robust real-world acrobatics with novel terrain, adversarial disturbances, and unscripted routines is not solved. The honest version is narrower: low-contact whole-body control of a known articulated system is well-characterized. We know how to formulate the problem, we know which methods work, and we can scale them.

A humanoid doing a backflip is, mechanically, a controlled trajectory through configuration space. The dynamics are smooth, $\dot{x} = f(x, u)$, where $x$ is the robot's state, $u$ is the joint torque vector, and $f$ is differentiable almost everywhere. State estimation is good because the relevant variables (joint angles, velocities, base orientation) are directly measurable through encoders and IMUs. The objective is fixed in advance. Trajectory optimization, model-predictive control, and reinforcement learning all have mature pipelines for this regime, and the sim-to-real gap, while real, is tractable.

This is not trivial. Stabilizing a nonlinear underactuated system under flight phases requires real engineering. But it is engineering on a problem with a clean mathematical structure.

Where contact breaks the structure

Manipulation is a different object. Once a controlled body has to maintain sustained contact with an external object, three things change at once.

Dynamics become non-smooth. Contact introduces complementarity constraints. For a single contact point with gap $\phi(q) \geq 0$ between the gripper and the object, normal force $\lambda_n$, and tangential force $\lambda_t$, the contact conditions are

with Coulomb friction adding $|\lambda_t| \leq \mu \lambda_n$, where the inequality is tight during slip and slack during stick. This is a linear complementarity problem embedded inside the equations of motion. The resulting dynamics are piecewise-smooth with switching surfaces at every contact event. Gradients with respect to $u$ are discontinuous at those surfaces, which breaks naive gradient-based trajectory optimization and forces methods that either smooth the contacts (introducing bias) or enumerate the modes explicitly (combinatorial explosion).

Observability collapses. The robot's own state is observable. The contact state is not. Friction coefficients, contact normals, and slip onset must be inferred from sparse tactile signals and proprioception. Formally, manipulation is a POMDP where the hidden component includes the contact mode, contact location, and object pose. Belief-state planning is the right framework, and it is computationally hard. Acrobatics, by contrast, is essentially fully observed; the MDP approximation is reasonable.

Actuation is mediated. In acrobatics the controlled object is the actuated object. In manipulation the controlled object is reached through contact forces, which lie in a constrained set (the friction cone) and depend on geometry, materials, and contact history. Let $J_c(q)$ be the Jacobian relating joint velocities to contact-point velocities. The map from commanded torques to object accelerations factors through $J_c^\top \lambda$, where $\lambda$ is itself constrained to the friction cone and determined implicitly by the LCP above. The relevant Jacobian is rank-deficient whenever the contact cannot transmit the required force, which is most of the time. The configuration space the agent cares about (the object's pose) is underactuated through a non-smooth interface.

Why this is not a benchmark problem

A common reading is that manipulation benchmarks have not saturated because the methods are immature. The structural reading is that the problem class admits less help from the standard toolkit. Smooth optimization inherits bias from contact smoothing. Reinforcement learning suffers from the same non-smoothness in the form of discontinuous reward landscapes around contact events, plus the partial observability that makes credit assignment harder. Sim-to-real transfer is harder because contact dynamics are precisely where simulators diverge most from physics: contact stiffness, friction models, and integration schemes all introduce artifacts that policies overfit to.

This is consistent with what the contact-rich manipulation literature has been finding. Methods that work tend to either (a) constrain the problem until contact becomes mild and predictable (parallel-jaw grasping of rigid objects in known poses), (b) accept high data costs and learn contact-aware policies end-to-end (large-scale teleoperation datasets, behavior cloning at scale), or (c) introduce dense tactile sensing and treat contact as a perceptual problem to be observed rather than predicted. None of these are tricks. They are responses to the structural fact that contact dynamics do not yield to the methods that work for free-flight control.

What the original line was pointing at

Ray's compressed version was that acrobatics is solved and manipulation is the hard part. The accurate version is closer to this: low-contact whole-body control sits inside a problem class we have good tools for, and rich-contact manipulation sits inside a problem class we do not. Better data and bigger models help with the second class, but they do not change its structure. Hybrid dynamics, partial observability of contact state, and underactuation through constrained interfaces are properties of the problem, not of the methods we throw at it.

What this changes for me is how I read humanoid demos. A robot doing flips is a demonstration that we can solve the easier problem class at scale. It is genuinely hard control, and it is not nothing, but the difficulty lives inside the system and the system is fully under the agent's authority. A robot reliably picking up a novel object, rotating it in hand, and placing it precisely is a demonstration that we can solve the harder class. We mostly cannot, yet, and the reason is in the math.

The reframing is small but it changes what counts as progress. Flips are control at scale. Manipulation is competence under non-smooth, partially observed, mediated dynamics. Different problems, different tools, and the second one is where the work is.