Skip to main content
Artificial intelligence

Self-driving cars: lights, camera, action!

Two men and another vehicle as seen from the front window of a car and on a screen in the car’s cockpit.

Matthias Birk and Andreas Schulz are teaching vehicles to see. The camera for self-driving cars they helped develop uses AI to ensure more reliable object recognition. Their job is at the crossroads of mind-bending theory and eye-popping practice.

Realizing the vision of autonomous cars

Matthias Birk and Andreas Schulz with a tablet next to a vehicle with an self-driving car camera standing on a test track.
Matthias Birk and Andreas Schulz at the Bosch test track in Renningen.

A clear idea of the present can help point the way forward. At least in the case of Matthias Birk and Andreas Schulz, two Bosch engineers who are working on a multi purpose camera. Known as MPC3, it enables vehicles to recognize their surroundings more quickly and reliably than ever before. The camera is helping solve current problems with surround sensors — and in doing so, propelling the development of automated driving forward.

The new MPC3 multi purpose camera — better than the human eye

Together with colleagues, the project managers Birk and Schulz coordinated the 20 or so teams that were involved in the camera’s development. But these teams haven’t only combined a lot of knowledge. They are also jointly responsible for a daunting task. “We want to make the car a better driver. This is why the camera-based technology has to work more reliably than people,” Birk says. The MPC3 already does this: “Just because people look, it doesn’t necessarily mean they grasp what they’re seeing. This is where we speak about ‘looked-but-failed-to-see errors,’” Schulz says. By contrast, the video camera mounted behind the windshield between the headliner and rear-view mirror sees things very clearly indeed, triggering the appropriate response in an instant. And it never tires. Even after a long drive, it remains as alert and responsive as it was during the journey’s first few kilometers.

A view of a vehicle ceiling. The multifunctional camera is installed above the rear view mirror.
The camera sits above the vehicle’s rearview mirror.
“We no longer need road markings to keep a self-driving car safely on track.”
Andreas Schulz, project manager, Chassis Systems Control

Drives safely, brakes on its own

Andreas Schulz and Matthias Birk watch with satisfaction as a vehicle equipped with an MPC3 camera brakes to avoid hitting another vehicle.
The car brakes as it should. Even so, Andreas Schulz and Matthias Birk take a closer look.

Vibrant shapes and surfaces flicker across the dashboard monitor as Birk and Schulz demonstrate the camera’s functions on the test track at the Bosch location in Renningen. The multi purpose camera scans the road, processing information about what it sees with remarkable speed. Responding to an obstacle in its path, the car brakes hard.

An AI-enabled milestone

The MPC3 marks a major stride toward autonomous vehicles, an advance impelled mainly by artificial intelligence. To develop the camera, the Bosch team took a multi-path approach. Its engineers and programmers created a software architecture that combines conventional image-processing algorithms with AI-driven methods, and embedded it on a high-performance system-on-chip (SoC) with an integrated microprocessor. “This enables peerless scene understanding and reliable object recognition,” Schulz says.

Andreas Schulz and Matthias Birk use a computer to adapt the system directly in the cockpit of the test car for automated driving.
A computer in the test vehicle’s cockpit allows adjustments to be made on the spot.

Three paths to a reliable result

Measuring 12 by 6.1 by 3.6 centimeters, the MPC3 employs three paths to identify objects.

Path 1: The classifier

The first path: how the MPC3 recognizes and classifies objects and pedestrians.

The first path is the conventional approach already in use. With the help of machine learning, classifiers are trained to recognize and classify specific objects such as vehicles and pedestrians.

Path 2: The optical flow

The second path: the MPC3 multifunctional camera uses optical-flow and structure-from-motion imaging to detect objects bounding the road.

In the second path, the camera taps into the optical-flow and structure-from-motion (SfM) imaging to detect raised objects bounding the road, such as curbstones. It computes three-dimensional structures based on corresponding points in images, the motion sequences of which are tracked.

“What’s more, the images’ resolution is higher than ever. Instead of roughly one megapixel, we now have roughly 2.5. This allows us to detect objects even earlier and more precisely,” Birk says. The camera’s angle of view has also been widened to around 100 degrees. “This is more than we humans can normally see sharply with a focused gaze,” he says.

Path 3: Artificial intelligence

The third path: the system uses neural networks and semantic segmentation to distinguish road surfaces from roadsides.

The third path puts artificial intelligence to good use. For another first in object detection, this new generation of multi purpose camera can distinguish the roadway from the shoulder or identify discrete objects.

It uses neural networks and semantic segmentation that assigns each pixel of an image to a defined category. This is a huge advantage. “We no longer need road markings to keep a self-driving car safely on track,” Schulz says. By combining all three detection paths, errors can be minimized.

The ace in the hole — an agile project framework

 A bird’s-eye view of a team at a conference table.
Teamwork as it should be — agile, independent, and very exciting.

Keeping things on track was very much on the managers’ minds during this challenging project’s development phase. “We want to be able to react quickly to changes,” Schulz says, adding: “That’s why it’s important for us to have a lot of flexibility in how we work.” A contact person in each team allows the project managers to coordinate teams working independently of each other. Even so, no day can be planned with total precision, which makes the job both exciting and challenging. “There is no master plan for the technology we’re working on; practically everything about it is uncharted territory,” Birk says.

Leading the effort to usher in autonomous vehicles

“As an engineer, it’s great to be able to work in such proximity to the real world to make the lives of many people safer and less stressful,” Birk says. Schultz seconds that notion: “Our work has a direct impact on the everyday lives of many people, which is motivating.

It’s a privilege to be able to play a leading role in co-creating the future at such an early stage in our careers.” The next milestone on the road to the self-driving system — the fourth generation of the multi purpose camera — already awaits the duo’s attention.


Matthias Birk and Andreas Schulz look into the trunk of a test car equipped with an integrated camera for self-driving cars.

Matthias Birk and Andreas Schulz

Project managers, Function and Algorithms, Video Perception, Cross-Domain Computing Solutions, Bosch

We innovate. That means being open to every sort of change, every day.

Matthias Birk and Andreas Schulz graduated from the Karlsruhe Institute of Technology (KIT). Birk studied electrical engineering with a focus on system-on-chip technology. Schulz majored in mathematics with a minor in computer science. After interning at the Bosch Research and Technology Center in Palo Alto, California, Birk earned a doctorate at KIT with his work on accelerated medical image reconstruction using parallel heterogeneous hardware platforms. Schulz opted to do his PhD in image processing at Bosch, where he is now working to chart the future of driving.

Share this on: