Skip to main content
Item picking robots in warehouses

Camera-based item picking: Robotic colleagues help with packing

Bosch Research has developed a piece picking system for industrial robots which makes order picking in warehouses much easier.

The picture shows the Smart Item Picking System from Bosch Research and Bosch Rexroth in an industrial environment. The robotic arm picks up an object from the conveyor belt.

It is half past nine in the morning in a typical warehouse. There are tens of thousands of different products on shelves several meters high. Forklift trucks are on the move in the aisles between them. From time to time they lift a crate out of the compartment and take it to the packing or order picking station. Staff then put together the packages for shipping in accordance with the orders placed - with all of the work being done by hand. “When it comes to order picking, automation still doesn’t play any significant role,” said Christoph Marx, a research engineering at Bosch Research focusing on assembly technology and industrial robotics. As a result, selecting products in boxes, so-called item picking, accounts for around 50 percent of the costs of order processing in e-commerce.

Unlike the previous automation solutions in production environments which the Bosch team would usually focus on, order picking in warehouses is a totally new world, one which the robotics experts believe offers great potential. “In industrial production, most of our robotic applications are specially designed to grasp an individual component quickly and precisely and then assemble it elsewhere with great precision and quality. The same component is grasped millions of times. The robotic grippers and processes are therefore precisely tailored to this,” explained Marx.

The picture shows the Smart Item Picking System from Bosch Research and Bosch Rexroth in an industrial environment. The robotic arm picks up an object from the conveyor belt.
Camera-based, the robotic arm of the Smart Item Picker grabs the desired part from the crate and thus helps with automated picking.

When it comes to order picking in warehouses, engineers face very different challenges. Here, a single robotic system at the commissioning station needs to handle an unbelievably wide range of products of different sizes, weights and forms. Picking the goods from the approaching boxes, so-called bin picking, must also be possible without additional information on the products being provided beforehand. “Given the sheer quantity of products and the ever changing product range, we can’t maintain a database with product information regarding typical properties or even CAD data,” said the robotics expert. Instead, the team has developed generalized methods for grasping techniques and object segmentation for example which are used for a very wide range of goods but without knowing what the robot will find in a box.

Model-free “bin picking” – the smart revolution in warehouses

A swiveling robotic arm from Bosch Rexroth can be seen here in a factory environment. A camera is mounted above it. Using a suction pad on the gripper arm, the robotic arm picks a small box out of a crate containing a single type of goods and places it on the conveyor belt.
With the help of a camera, the robotic arm of the Smart Item Picker from Bosch Rexroth picks the desired item out of the crate.

In recent years, Bosch Research developed a new intelligent solution with industrial robots, one which was designed to fill this automation gap. As a result, day-to-day work in warehouses could look something like this in the future: A customer orders for example two windscreen wipers and a packet of brake linings. In accordance with the customer’s order, the robot is provided with a box of windscreen wipers and a box of brake linings at the order picking station. This is equipped with a 3D camera and a robotic arm. The camera photographs the contents of the box from above. The integrated software then records the contents, segments individual products on that basis and provides the robot with information regarding the exact position of the desired goods and the optimum “grasp”. The robot calculates an ideal path allowing it to reach into the box without collisions, picks out the desired product and places it accurately in the shipping box provided. For the second windscreen wiper in the order, the camera takes a new picture and the process begins once again from the start.

AI meets classic methods

This sequence is the result of a great deal of research work on the part of Bosch Research experts for industrial robotics.

A graphic in five steps explains the process for recognizing, gripping and placing an object with the help of the Smart Item Picking system from Bosch Research.
Process graphic for recognizing, gripping and placing objects using the Smart Item Picking system from Bosch Research.
Graphic which explains the object recognition process used by the Smart Item Picking system from Bosch Research. With the help of a camera, the Smart Item Picking system from Bosch Research recognizes objects without “knowing” what is in the crate beforehand. A classic image processing procedure is first used to search for primitive shapes in the box. For individually shaped objects, the engineers also use deep learning processes.

Firstly: Object recognition.

Firstly: Object recognition. “The robot must be able to identify the contents of a crate without knowing what’s in the crate beforehand,” said Marx, explaining the challenge. A classic image processing procedure is used to search for primitive shapes in the box. For individually shaped objects, the engineers also use deep learning processes. In order for this to be possible, the algorithm must literally be fed with images from a goods portfolio in advance. “We define various object classes which are contained in the range of products for the particular customer applications and train our network with these data,” said AI expert Philipp Schillinger from the Bosch Center for Artificial Intelligence (BCAI). This network can then be used in the various use cases if the product range does not deviate excessively.

Graphic which shows how the robotic arm of the Smart Item Picker from Bosch Research now picks the appropriate object out of the crate.

Secondly: Item picking knowledge.

With the help of algorithms, the robot knows which goods in the box it should remove first. In order for this to be possible, the object poses found are compared on the basis of properties such as the size of the object, the quality of object segmentation and the position of the object in the box.

Graphic showing the collision-free path planning for the robot arm of the Smart Item Picker from Bosch Research.

Thirdly: Smart collision-free path planning.

This prevents the robot becoming caught on obstacles when reaching into the box or on the way back. It therefore always finds the best way to the product. The algorithm on which the system is based searches for possible movement paths in a multi-dimensional solution space and checks for possible collisions between the robotic arm and its environment. At the end, a path for the robot is chosen which is as short and thus as efficient as possible for the robotic arm in order to save precious cycle time.

Graphic which explains how the Smart Item Picking system from Bosch Research selects the right gripper and how suitable suction pads are activated or deactivated in order to grip the target object correctly.

Fourthly: The grippers.

In addition to the computing models, the engineers developed special grippers which can cope with a wide variety of products without damaging the products to be gripped in the process. They are also designed to grasp components in corners and thus actually empty the whole box. Depending on the size of the object to be grasped, individual suction pads on the universal gripper are activated or deactivated.

Graphic which explains how Smart Item Picking first grips the object and then places it in the target crate as the final step in the process.

Fifthly: Placing the object in the shipping box in a targeted manner.

There are various possibilities here, for example dropping or placing the object in a box in a targeted manner. One aim of “placing” is to make the best possible use of the space in the box. At this stage in the project, however, “dropping” is mainly used. Nevertheless, there are definite plans to expand the “placing” feature as part of current development planning.

Explanatory video: the Smart Item Picking System from Bosch Research and Bosch Rexroth

Bosch Research experts work hand in hand with product developers at Bosch Rexroth

Bosch Research came up with the idea of addressing the automation gap in e-commerce around three years ago. Slightly later, Bosch Rexroth realized that it actually had a need for such products. One of its customers was looking for an automated order picking system. As a result, developing the Smart Item Picking robot became a joint project involving Bosch Research and Bosch Rexroth.

The image shows a robotic arm from Bosch Rexroth which is picking a small package out of a crate.
Bosch Research and Bosch Rexroth are working hand in hand to develop the Smart Item Picking system.
The robotic arm of the Smart Item Picking system can be seen surrounded by crates with small objects at the Bosch Rexroth Model Factory in Ulm.
The Smart Item Picking system at Bosch Rexroth’s Model Factory in Ulm.
The robotic arm of the Smart Item Picking system can be seen surrounded by crates with small objects at the Bosch Rexroth Model Factory in Ulm.
The Smart Item Picking system at Bosch Rexroth’s Model Factory in Ulm.
The robotic arm of the Smart Item Picking system can be seen surrounded by crates with small objects at the Bosch Rexroth Model Factory in Ulm.
The Smart Item Picking system at Bosch Rexroth’s Model Factory in Ulm.
/

The ten-person project team developed a first prototype for the Bosch Rexroth Model Factory in Ulm. Rexroth’s job was then to bring the prototype into mass production. “From the very start, we had the idea of launching a construction kit system which would meet the needs of customers from specific sectors openly,” said Jörg Heckel from Bosch Rexroth. Depending on the product portfolio and workstation, Bosch puts together a suitable robotic arm, a 3D sensor and a gripper and equips a suitable software package with the necessary algorithms. At the end of May 2022, the project team will officially unveil the new custom-made system at the Logimat logistics trade fair in Stuttgart.

With the help of various algorithms developed by Bosch, item picking robots are able to recognize specific products in boxes, pick them out and place them in shipping packages. The various USPs which are offered by the system and which were developed through to the market launch stage by Bosch Research and Bosch Rexroth can be seen here:

A graphic showing the six major benefits of the Smart Item Picking system from Bosch Research, such as “deep item picking knowledge” or the possibility of combining various robots and sensors.
The six major benefits of the Smart Item Picking system from Bosch Research.

Unique Selling Propositions of the Smart Item Picking system

Bosch Research expert and mechatronics engineer Christoph Marx dealing with robotics in the industrial context at the Bosch Research Campus in Renningen, Germany.

Christoph Marx

Christoph Marx studied mechanical engineering and mechatronics at Esslingen University of Applied Sciences in Germany. During his studies, he was already involved in industrial robotics and software development and contributed to the development of the automatic production assistant "APAS" in his final thesis at Bosch Research. Since 2016, Christoph Marx has been working as a research engineer at Bosch Research. Until 2019, he and his team dealt with robotic systems in production. The team's focus here was always to simplify the programming of robots using specific path planning algorithms and 3D image processing methods. Another focus has been two-arm robot systems that mimic humans and their use in typical manual workstations in Bosch production. Since 2019, Marx and his colleagues have been working on automating picking tasks in warehouses, which is becoming increasingly important due to the steady growth in e-commerce. Marx has also recently started representing Bosch in the non-profit association "euRobotics" of the European robotics community, where, among other things, influence is exerted on future robotics topics of the European Research Framework Programs.

Share this on: