Learning from Data

A robot navigating a warehouse must recognize hundreds of product types, estimate where each one sits in three-dimensional space, and predict whether a given grasp will succeed before committing to it. None of these tasks can be specified with explicit rules. All three are learned from data. Supervised learning — the branch of machine learning in which a model is trained on labeled input-output pairs — is the engine behind most robot perception systems operating in the world today. This lesson develops supervised learning precisely as it applies to robotics, with attention to the data pipelines, model architectures, and failure modes specific to physical robots.

Supervised Learning: The Formal Setup

In supervised learning, we have a dataset D = {(x1, y1), (x2, y2), ..., (xN, yN)} where each xi is an input observation and each yi is the corresponding label or target value. The goal is to learn a function f such that f(x) approximates y for new inputs x not in D. For a robot, the inputs might be: - A 640x480 RGB image from a wrist-mounted camera - A point cloud of 10,000 3D measurements from a LiDAR sensor - A 6-axis force-torque reading at the robot's wrist - A concatenated vector of joint angles and joint velocities The output targets might be: - The class label of the object in the image (classification) - The 6-DoF pose of an object (position plus orientation in 3D space) (regression) - The probability that a proposed grasp will succeed (binary classification) - The next joint-velocity commands to track a trajectory (regression) The critical move in applying supervised learning to robotics is the translation: you must define precisely what the input representation will be, what the output target means, how labels will be obtained, and what loss function will measure prediction quality. Errors in this translation — choosing the wrong input representation, using noisy or systematically biased labels — propagate through the learned model and cause failures that are very difficult to diagnose after the fact.

How Robots Obtain Labeled Data

Data collection is the most expensive and robot-specific part of the supervised learning pipeline. Three dominant strategies exist. Strategy 1 — Human annotation of recorded observations. A robot records sensor streams (camera, LiDAR, force-torque) while a human teleoperates it through a task. Humans then annotate the recordings: drawing bounding boxes around objects, labeling grasp outcomes as success or failure, marking where the robot's gripper was in each frame. This is high-quality but labor-intensive. Companies like Boston Dynamics and large warehouse robotics firms employ teams of annotators. Strategy 2 — Simulation-based labeling. Labels that are expensive to collect physically — such as the exact 6-DoF pose of every object in a scene — can be obtained for free in simulation, where the ground truth is known by construction. The robot operates in a physics simulator (Gazebo, MuJoCo, Isaac Sim) and every label is automatically generated. The resulting dataset is large and perfectly labeled, but may not transfer to the real world because simulated sensors differ from real ones. This gap is called the sim-to-real domain gap and is addressed in Lesson 5. Strategy 3 — Self-supervised and weakly supervised labeling. In some settings, the robot can generate its own labels from structure in the data, without human annotation. A robot arm that attempts thousands of grasps can label each attempt as success or failure from a force sensor reading — no human needed. A camera-equipped robot can use the known motion of its camera between frames to compute depth labels from stereo or motion cues. These self-supervised approaches allow scaling to millions of examples that would be prohibitively expensive to label by hand.

Key Architectures for Robot Perception

The choice of neural network architecture depends heavily on the input modality. Convolutional Neural Networks (CNNs) remain the backbone of image-based robot perception. Their key inductive bias — that nearby pixels are more related than distant ones, and that the same feature detector applies everywhere in the image — is well-matched to visual data. Architectures like ResNet-50, EfficientDet, and YOLO have been deployed on robot platforms from surgical robots (Da Vinci) to autonomous vehicles (Waymo) to quadrupeds (Spot). Point cloud processing requires architectures designed for unordered 3D data. PointNet (2017, Stanford) was a landmark: it processes each 3D point independently then aggregates features with symmetric functions (max pooling) to achieve invariance to point order. Later architectures like PointNet++ and 3D sparse convolutions improved on local structure capture. Transformer-based architectures have emerged as universal backbones. Vision Transformers (ViT) treat an image as a sequence of patches and apply self-attention across them. More recently, architectures like RoboAgent and RT-2 (Google DeepMind, 2023) use large vision-language transformer backbones to give robots semantic understanding that generalizes across tasks — a robot that understands what 'pick up the apple' means across many different apples and scenes. For time-series data (joint angles over time, force profiles during manipulation), recurrent architectures (LSTM, GRU) and temporal convolutional networks are used, though transformers have increasingly replaced these as well.