How Self-Driving Cars See: Sensors, Cameras, and Radar

In 2023, Waymo's self-driving taxis completed over 700,000 paid trips in San Francisco without a human driver at the wheel. Each mile those cars traveled, they processed roughly 1 gigabyte of sensor data every second — analyzing it all fast enough to brake, steer, and accelerate in real time. How does a car with no eyes actually see the world? The answer is a combination of cameras, laser scanners, and radio waves working together in a system engineers call a perception pipeline.

By the end of this lesson, you will be able to: • Explain what cameras, lidar, radar, and ultrasonic sensors each detect • Describe why self-driving cars use multiple sensor types at the same time • Identify real-world situations where each sensor type performs best or worst • Explain the concept of sensor fusion in plain language

Cameras on self-driving cars work a lot like human eyes — they capture light and turn it into images. A typical self-driving car has 8 to 12 cameras pointing in every direction, giving it a full 360-degree view around the vehicle. Cameras are the best tool for reading things that encode meaning in color or text: stop signs (red octagon with white letters), traffic lights (red, yellow, or green), painted lane markings, and pedestrian crosswalks. They can identify hundreds of object types — cars, cyclists, dogs, children — using AI systems trained on millions of images. The big weakness of cameras: they struggle in low light (nighttime, tunnels) and are blinded by direct glare from the sun or oncoming headlights. Rain and fog scatter light, making images blurry. And like photographs, cameras produce flat 2D images — they cannot directly measure how far away an object is.

Lidar stands for Light Detection And Ranging. A lidar unit fires millions of laser pulses per second in every direction. Each pulse travels at the speed of light, bounces off an object, and returns to the sensor. By measuring how long the round trip takes — often just nanoseconds (billionths of a second) — the sensor calculates the exact distance to that object. The result is a point cloud: a three-dimensional map made of millions of dots, each representing one laser pulse return. This map shows the exact size, shape, and position of every object around the car — buildings, pedestrians, traffic cones, even the edges of potholes in the road. Waymo's cars use a spinning lidar unit on the roof that generates a full 3D scan of the environment 10 times per second. The main weakness: lidar units are expensive (early units cost $75,000 each, though prices have dropped sharply) and they struggle in heavy rain or fog because water droplets scatter the laser beams.

Radar works like lidar but uses radio waves instead of laser beams. Radio waves pass right through rain, fog, snow, and dust — conditions that blind cameras and scatter lidar pulses. Radar is especially good at measuring the speed of moving objects using the Doppler effect, and it can detect large objects like cars and trucks at long distances even in terrible weather. However, radar has low resolution: it can detect that something large is ahead, but it cannot tell you whether it is a car, a bicycle, or a shopping cart. Ultrasonic sensors work like sonar — similar to how bats navigate. They emit sound pulses and measure the echo. They are most useful at slow speeds: for parking, detecting objects close to the bumper, and ensuring the car does not back into a trash can. They work reliably in any weather but have very limited range (only a few meters).

No single sensor type handles every situation, so self-driving cars use sensor fusion — the process of combining data from all sensors simultaneously to build one complete, reliable picture of the environment. How it works in practice: Imagine it is raining at night, and a person in a dark jacket is crossing the road. • The cameras see the pedestrian's shape but the image is dark and blurry. • The lidar detects a 3D human-shaped object but scatters on heavy raindrops. • The radar clearly detects a moving mass ahead. • Ultrasonic confirms something is close to the vehicle. The car's computer fuses all four data streams, assigns confidence levels to each, and concludes: 99.8% probability that a human pedestrian is crossing — apply brakes now. The redundancy is intentional: if any one sensor fails or gives a wrong reading, the others keep the system safe. This is fundamentally different from human driving, where we have only two eyes and one brain.