Simulation and Sim-to-Real

Training a reinforcement learning policy on a physical robot for a walking task might require millions of trial steps. At one step per second, that is 11 days of continuous operation — per training run. Physics simulators solve this by running thousands of virtual robots in parallel, at hundreds of times real speed, generating millions of steps in hours on a GPU cluster. But simulation is an approximation of reality. The policy trained in simulation must still run on a robot made of metal, rubber, and electronics subject to friction, temperature, wear, and all the imprecision of the physical world. Bridging this gap — making simulation-trained policies work on real hardware — is one of the central engineering challenges of modern robot learning.

Physics Simulators: What They Model and What They Omit

Modern robot physics simulators (MuJoCo, IsaacGym/IsaacSim, Gazebo, PyBullet, CARLA for autonomous driving) model three things: rigid body dynamics, contact and collision physics, and sensor rendering. Rigid body dynamics tracks how robot links and objects move under applied forces and torques, governed by Newton-Euler equations integrated forward in time. This part of simulation is well-understood and highly accurate for most robot structures. Contact and collision physics is harder. When a gripper contacts an object, micro-deformations, friction, and the geometry of contact edges all affect outcomes. Simulators approximate contact with simplified models (spring-dampers, Coulomb friction) that differ from real contact behavior. This is the primary source of sim-to-real gap in manipulation tasks. Sensor rendering: simulating what a camera, LiDAR, or force-torque sensor would observe requires additional models. Simulated RGB images are rendered by graphics engines that do not capture real sensor noise, lens distortion, bloom, or the subtle texture differences between real surfaces and their simulated counterparts. Simulated LiDAR may miss the multi-path reflections and beam divergence of real sensors. The result is a simulation fidelity spectrum. Proprioceptive signals (joint angles, angular velocities) simulate very accurately because they depend only on rigid body dynamics. Contact forces simulate moderately well. Visual sensor output simulates poorly. This is why RL locomotion controllers (which depend mostly on proprioceptive feedback) transfer more readily to hardware than manipulation policies (which depend heavily on contact and vision).

Domain Randomization: Teaching Robustness Through Variety

If the simulation will never perfectly match reality, one strategy is to make the policy robust to a wide range of simulation parameters — so wide that reality falls somewhere within the distribution. This is domain randomization. In domain randomization, simulation parameters are randomly sampled from broad distributions during each training episode: - Mass and inertia of robot links: ±30% of nominal - Friction coefficients on surfaces: uniform sample from [0.3, 1.5] - Motor torque constants: ±20% - Camera position and orientation: small random offsets - Object visual appearance: random textures, lighting conditions, colors - Sensor noise: random Gaussian noise added to observations A policy trained across this enormous variety of simulated worlds cannot rely on any single set of physical parameters being correct. Instead, it learns behaviors that work despite parameter uncertainty — and these behaviors tend to work on real hardware because real hardware parameters fall within the randomized range. OpenAI's Dactyl project (2019) used extensive domain randomization to train a robotic hand to manipulate a Rubik's cube entirely in simulation. The randomized parameters included 100 different physical attributes. When transferred to the real Shadow Hand hardware, the policy generalized successfully — the first time a physical robot hand solved a Rubik's cube with a learned policy. Domain randomization does have limits. If the real robot's parameters fall outside the randomization range, the policy will fail. And overly aggressive randomization can prevent the policy from converging at all, because the task becomes too hard under extreme parameter combinations.

System Identification and Adaptive Methods

An alternative to randomization is system identification: carefully measuring the real robot's physical parameters and configuring the simulation to match them as closely as possible. Techniques include: Measuring mass and inertia by observing how the robot responds to known torques. Estimating friction by recording joint motion under gravity and fitting a friction model. Identifying motor characteristics (torque constant, back-EMF) from electrical measurements under controlled loads. System identification produces a 'digital twin' — a simulation tuned to match a specific physical robot. Policies trained on a well-tuned digital twin can transfer with much less domain randomization because the simulation is already accurate. Boston Dynamics uses digital twins extensively in their development pipeline. A more powerful approach is adaptive transfer: train the policy in simulation with a learned context variable that captures the current physical parameters. At deployment, the robot quickly infers its context variable by observing its own behavior for a few seconds, then uses the matched policy. This is sometimes called meta-RL or rapid motor adaptation (RMA, developed at CMU, 2021). RMA-trained ANYmal robots adapted their gait to mud, sand, and carrying payloads within seconds of deployment — without any prior real-world training on those surfaces.