From cloud to concrete: How Bosch is unlocking AI for the physical world

First, why embed AI into products at all? Why not just connect everything to a powerful AI in the cloud? The answer is that we will ultimately need both: a centralized Cloud AI and local, or Edge AI, which operates directly within a device. This Edge AI can react reliably in milliseconds and operate securely on local data.

Many scenarios demand this local intelligence. For example, an autonomous vehicle cannot wait for a cloud round-trip to brake for a pedestrian. Likewise, collaborative humanoid robots require instant decision-making to work safely alongside people.

These scenarios highlight several key drivers for Edge AI:

• Latency and reliability: Edge AI provides the split-second, dependable response times essential for safety-critical systems like automated driving.
• Connectivity: The application may be in a location with poor or no internet access.
• Data bandwidth: Modern autonomous systems generate enormous amounts of data. Processing it locally is far more efficient and affordable than sending it all to the cloud.
• Security and privacy: Processing sensitive data on-device reduces the attack surface for cyber threats.
• Cost: Relying on the cloud for every single calculation can lead to significant, recurring operational costs.

Given the need for Edge AI, why not just run a cloud model on a local chip? The core of the problem lies in two challenges:

• First, a strategic challenge: relying on a single chip family for both cloud and edge can limit design freedom and supply chain resilience.
• Second, a technical mismatch: hardware for training AI in the cloud is built for flexibility, while embedded systems on a chip (SoCs) are built for maximum efficiency – performing a specific task with the least amount of power.

This has created a vibrant market of specialized SoCs for Edge AI. While their manufacturers provide toolchains to help deploy AI models, these tools are, by necessity, model-agnostic. They cannot be pre-optimized for the unique architecture and data characteristics of a specific customer’s AI model. This is the source of the hardware-software performance gap.

Imagine trying to pack a suitcase. A generic approach might be to just toss your clothes in, which works but leaves a lot of wasted space. But if you are an expert at folding each specific item, you can fit dramatically more inside. That is precisely what we do with AI.

At Bosch, we have developed that expert packing technique for AI. It is an AI inference optimization toolchain, the result of a co-creation effort between Bosch Research and our Bosch Mobility division. Our toolchain serves as a sophisticated engine for hardware-software co-optimization. It analyzes both the AI model and the target chip’s architecture, then automatically restructures the model for maximum efficiency.

For AI-powered perception in advanced driver-assistance systems (ADAS), our toolchain has achieved up to a six times inference speed-up on the same hardware compared to a standard deployment and without compromising the model's required output accuracy. This means the chip can deliver more powerful features, or that a smaller, cost-effective SoC can be used for the application. We have successfully showcased these capabilities with several leading automotive SoCs.

This holistic approach liberates us from being locked into a single hardware supplier, enabling software-defined systems, ensuring supply chain resilience, and giving us a global competitive advantage.

Our toolchain’s key principle is that we use AI to optimize AI. This is achieved through a suite of automated methods:

• Hardware-Aware Neural Architecture Search (HW-NAS): Instead of designing an AI model in a vacuum, our algorithms automatically search for an architecture that is perfectly suited to the target hardware’s strengths and weaknesses from the start.
• Quantization-Aware Training (QAT): To drastically reduce model size and improve speed, we convert models to use lower-precision mathematics (quantization). Crucially, this is not just a post-processing step. It is integrated into a specialized training process (pruning and quantization-aware training) that ensures the model outputs remain highly accurate even after compression.
• Graph-level deployment optimizations: We analyze the complex chain of mathematical calculations inside a neural network and intelligently reorder them to match the way the specific chip processes information, dramatically improving runtime without affecting the model's output.
• AI-driven optimization loop: We take automation a step further by using intelligent agents to control the optimization process itself. These agents gather performance data from the target hardware (or its simulation) and use that information to reason about the best optimization strategy.

The journey of AI from the cloud into the physical world is one of today’s most exciting frontiers. Our vision extends to enabling “liquid intelligence” across the edge-cloud continuum, where algorithms can be deployed on any suitable silicon. However, mastering the future of edge AI is a team sport. The path forward lies in building open ecosystems where automakers, suppliers, and technology companies collaborate through open standards and deep partnerships. Our toolchain is a key enabler for this collaborative future, supporting the development of a multi-vendor chiplet ecosystem crucial for a modular and sovereign European compute landscape. By providing a clear path to targeted AI optimization, Bosch is committed to driving this transformation with our partners.