DeepInspect

Bosch Research expert Matthias Kayser in a modern model factory hall (ARENA2036 in Stuttgart), pointing at a screen which, with the help of Bosch’s DeepInspect system, shows him whether or not his component has been correctly installed.

A youngster who enjoys scale model-building has three bowls with pre-sorted clip-together pieces in front of him in his bedroom and, next to them, a booklet with detailed instructions for building a scale model of a cool racing car. He reaches for the necessary parts one by one and carefully puts them together. Until he picks up the wrong piece and puts it in the wrong place, that is. The boy becomes frustrated and it takes him a while to figure out where he went wrong so that he can get back on track.

There are around 1,000 manual assembly stations at Bosch’s plants. If errors occur at one of them or at one of our customers’ plants, this not only causes frustration but also leads to increased production costs. And the “models” are usually more complex than the sets which children build. An assembly guide which tells plant employees straight away if they have put the right part in the right position during a particular step would be extremely helpful.

Deep learning allows robust object recognition

Bosch Research expert Matthias Kayser (left) in front of a monitor at ARENA2036 in Stuttgart and pointing at a screen which shows whether or not the component on the table below – a battery – has been installed correctly. The installation is part of a battery assembly pilot project carried out by Bosch Research together with Mercedes-Benz. The DeepInspect system shows the worker in real time precisely whether or not the component has been installed correctly. — Bosch Research expert Matthias Kayser demonstrating how DeepInspect works at ARENA2036 in Stuttgart as part of a joint pilot project with the Mercedes-Benz Group AG relating to battery assembly: The system shows the worker in real time precisely whether or not the component has been installed correctly.

Bosch Research engineers are currently working on such a solution. DeepInspect is a high-performance, intelligent and easy-to-train assembly assistance system designed to provide help at manual assembly stations. In spite of automation, there is still some important manual work to be carried out at the plants. This includes production tasks which must be performed by hand because they require a high degree of dexterity, procedures involving items that come in small batch sizes where building a fully automated assembly line would not make economic sense, and processes requiring human flexibility which a machine cannot provide. However, this does not mean that machines and/or software cannot be used to help people during their work. This is where DeepInspect comes in. “Deep” here refers to the concept of deep learning, i.e. machine learning on the basis of deep neural networks, for example for image analysis. “Inspect” means insight or checking with reference to the technology used. “Using an intelligent camera at the production site, the assistance system can check whether or not the assembly process is going to plan while production is taking place,” says Eduardo Monari, Head of the Cognitive Industrial Vision research group, explaining the use of DeepInspect. He adds: “DeepInspect is like a digital assembly guide which visualizes the next assembly step in order to assist employees.”

In order to do this, an intelligent camera observes the step-by-step assembly of the product while it is taking place. If a step has been carried out correctly, the system automatically moves on to the next step. “It only observes and checks the products and components, not the assembly workers,” says Monari. The assistance system is also not designed to recognize assembly errors which could be traced back to particular employees. Instead, it is designed to look ahead to ensure correct assembly. “This perspective is important to us,” says Monari. “It focuses on what matters: Ensuring from the start that as few errors as possible actually occur and providing the best possible constructive support for employees in accordance with data protection requirements.”

Camera-based object recognition: Learning components made easy (easier)

Various systems designed to ensure that parts are put together correctly during complex assembly processes are available on the market. A range of solutions attempt to track and record the movements of people themselves in order to draw conclusions regarding the quality of the assembly steps undertaken. However, according to Eduardo Monari, users quickly find that this approach has its limits. Firstly, the processes involved can be too complex. Secondly, this approach falls short if the focus needs to be on positioning complex components correctly in a complex environment rather than ensuring that movements are ergonomically correct. Observing movements would therefore involve little more than an indirect check as to whether the assembly work was being carried out correctly. What is more, it would be neither efficient nor successful. “Some systems would also lead to critical questions regarding European data protection laws as they usually record whole people,” he adds.

Other solutions already rely on camera-based intelligent object recognition, though the team headed-up by Monari would like to optimize this with DeepInspect. After all, “training” is a complicated, time-consuming process for people and machines. With conventional methods, several hundred different images of a component are needed to train the assembly assistance system in order to obtain a reliable result. The human trainers must then spend a long time filtering out any images that are not relevant and linking relevant images to so-called image annotations. “The required image annotation has always been the stumbling block when it comes to acceptance and using the system cost-effectively in industrial assembly – but at Bosch Research, we have worked on this successfully,” says Project Manager Matthias Kayser.

With the help of an intelligent image evaluation system, the new DeepInspect software filters out any images that are not relevant while the component in question is being photographed and annotated automatically. “This way, it takes just 15 to 20 minutes to prepare the training data,” says Kayser. Another advantage is that because the process takes much less time, the trainers and, consequently, the system technicians at the plant can work iteratively to achieve the optimum result. They first produce 100 images, test to see whether this is enough and, if not, produce additional images until the result enables reliable object recognition and checks. What is more, DeepInspect offers the benefits of a “smart, interactive assembly guide” from the very first iteration.

Bosch Research engineer Matthias Kayser at a test table at ARENA2036 preparing components for the camera-based image recognition on which the DeepInspect system is based. — Bosch Research engineer Matthias Kayser preparing components at the test table for image-based object recognition.

The Four DeepInspect Steps

Graphic illustrating the process step of image capture and automatic mapping as part of Bosch Research's DeepInspect system.

Step 1: Capturing and annotating images

An adequate number of training images are first captured for each component. The DeepInspect tool starts annotating the images automatically. Ideally, the automatically annotated images will be so good that the trainers do not need to correct anything manually. A detector can then be trained for this component. DeepInspect offers suitable software components for this. The result is a neural network which enables the object captured to be detected. From this point on, the system will recognize this component, and the assembly frame for it will be determined as soon as it appears on camera . Step 1 is repeated for each component to be assembled and all detectable objects are registered in the so-called object catalog.

Graphic illustrating the process step of component recognition as part of Bosch Research’s DeepInspect system.

Step 2: Detecting the component type and defining the correct position

The so-called object detector, an image analysis module which is able to recognize the relevant components for the particular assembly process in the image, now comes into play. It is used in the second step and provides real-time information regarding the component, e.g. the current position and the component type in the field of vision, for each camera image.

Graphic illustrating the process step in which the correct assembly sequence is taught as part of Bosch Research’s DeepInspect system.

Step 3: Designing and setting up the workflow

To ensure that the system knows which components need to be assembled in which order and in which position, the assembly workflow has to be defined and taught. The detector which is already in use is involved here too. The detector system recognizes the exact positions of each object in the image, which means that the assembly workflow can be taught quickly and easily. As such, the system technician sets up the product once and the detector then automatically recognizes the components which are to be assembled in each case. During each assembly step, the system technician confirms the target position and configures the correct step in the assembly guide. This can be done simply by clicking on the frame shown around the recognized component. As a result, the assembly workflow with the steps for assembly in the correct order is defined interactively. Once Step 3 has been concluded , the set-up procedure is complete and assembly assistance during the production process can begin.

Graphic illustrating the assistance mode process step as part of Bosch Research’s DeepInspect system, in which production workers learn whether or not they have assembled the component correctly. Only if they have does assembly continue.

Step 4: Using the workflow correctly with the help of the assistance system

The DeepInspect software now switches to assistance mode. Here, the assembly workflow is shown in the form of assembly steps. The employee is prompted to assemble the next component in the position shown. The workflow only continues if the relevant component has been assembled correctly and in the right position and has been recognized by the detector. If the detector does not recognize the step as being correct due to an incorrect component or incorrect positioning, the workflow does not allow the worker to continue. As a result, the employee notices that something is wrong. The system then requests that the assembly be carried out correctly and no explicit error recognition is required.

First prototypes in use

The picture shows a test set-up at ARENA2036 in Stuttgart with an electric car battery at the bottom and a screen at the top on which Bosch Research’s DeepInspect system shows whether or not the battery has been installed correctly. — In collaboration with the Mercedes-Benz Group AG, Bosch Research has tested DeepInspect as part of a pilot project at the ARENA2036 research factory in Stuttgart using the example of a battery assembly system.

Bosch Research engineers are currently testing prototypes of DeepInspect. Together with the Mercedes-Benz Group AG, they demonstrated the assembly assistance system at the ARENA2036 research campus in Stuttgart using a battery assembly system as an example and tested it at a manual assembly station in a laboratory in Renningen. “We are currently developing specific pilot applications for four Bosch plants. These focus on very different production areas at Bosch,” says Monari. In the future, engineers at the plants will be able to train the interactive assembly guide for their product themselves with the help of the software – for happy employees at manual assembly stations.

Eduardo Monari

Dr. Eng. Eduardo Monari received his PhD in electrical engineering from the Karlsruhe Institute of Technology in Germany in 2011. Since 2018, he has been working at Bosch Research as Head of the Cognitive Industrial Vision research group (CR/APA2). He is also responsible for coordinating projects on AI in production systems at Bosch Research. His research background is in 2D/3D vision and signal processing, especially under uncontrolled and complex conditions. In this context, his research interest is always linked to the question as to how cognitive machines and systems can be trained or taught quickly, easily and with minimal effort.

Matthias Kayser

Dr. Matthias Kayser earned his doctorate in the field of image-based object recognition for traffic monitoring at Goethe University Frankfurt. When he joined Bosch’s central research department, his main focus was the detection and localization of objects in the environment using artificial intelligence in production technology and industrial robotics. As part of the DeepInspect project, Matthias Kayser is researching neural networks that recognize industrial components for the assembly assistance system and methods for training these networks with limited and automatically annotated data. In addition to the technical challenges, he also enjoys the interdisciplinary collaboration involved and networking with other Bosch plants around the world, which has enabled the system to be tested in many different areas.