Camera-based item picking: Robotic colleagues help with packing
Bosch Research has developed a piece picking system for industrial robots which makes order picking in warehouses much easier.
It is half past nine in the morning in a typical warehouse. There are tens of thousands of different products on shelves several meters high. Forklift trucks are on the move in the aisles between them. From time to time they lift a crate out of the compartment and take it to the packing or order picking station. Staff then put together the packages for shipping in accordance with the orders placed - with all of the work being done by hand. “When it comes to order picking, automation still doesn’t play any significant role,” said Christoph Marx, a research engineering at Bosch Research focusing on assembly technology and industrial robotics. As a result, selecting products in boxes, so-called item picking, accounts for around 50 percent of the costs of order processing in e-commerce.
Unlike the previous automation solutions in production environments which the Bosch team would usually focus on, order picking in warehouses is a totally new world, one which the robotics experts believe offers great potential. “In industrial production, most of our robotic applications are specially designed to grasp an individual component quickly and precisely and then assemble it elsewhere with great precision and quality. The same component is grasped millions of times. The robotic grippers and processes are therefore precisely tailored to this,” explained Marx.
When it comes to order picking in warehouses, engineers face very different challenges. Here, a single robotic system at the commissioning station needs to handle an unbelievably wide range of products of different sizes, weights and forms. Picking the goods from the approaching boxes, so-called bin picking, must also be possible without additional information on the products being provided beforehand. “Given the sheer quantity of products and the ever changing product range, we can’t maintain a database with product information regarding typical properties or even CAD data,” said the robotics expert. Instead, the team has developed generalized methods for grasping techniques and object segmentation for example which are used for a very wide range of goods but without knowing what the robot will find in a box.
Model-free “bin picking” – the smart revolution in warehouses
In recent years, Bosch Research developed a new intelligent solution with industrial robots, one which was designed to fill this automation gap. As a result, day-to-day work in warehouses could look something like this in the future: A customer orders for example two windscreen wipers and a packet of brake linings. In accordance with the customer’s order, the robot is provided with a box of windscreen wipers and a box of brake linings at the order picking station. This is equipped with a 3D camera and a robotic arm. The camera photographs the contents of the box from above. The integrated software then records the contents, segments individual products on that basis and provides the robot with information regarding the exact position of the desired goods and the optimum “grasp”. The robot calculates an ideal path allowing it to reach into the box without collisions, picks out the desired product and places it accurately in the shipping box provided. For the second windscreen wiper in the order, the camera takes a new picture and the process begins once again from the start.
AI meets classic methods
This sequence is the result of a great deal of research work on the part of Bosch Research experts for industrial robotics.
Loading the video requires your consent. If you agree by clicking on the Play icon, the video will load and data will be transmitted to Google as well as information will be accessed and stored by Google on your device. Google may be able to link these data or information with existing data.
Explanatory video: the Smart Item Picking System from Bosch Research and Bosch Rexroth
Bosch Research experts work hand in hand with product developers at Bosch Rexroth
Bosch Research came up with the idea of addressing the automation gap in e-commerce around three years ago. Slightly later, Bosch Rexroth realized that it actually had a need for such products. One of its customers was looking for an automated order picking system. As a result, developing the Smart Item Picking robot became a joint project involving Bosch Research and Bosch Rexroth.
The ten-person project team developed a first prototype for the Bosch Rexroth Model Factory in Ulm. Rexroth’s job was then to bring the prototype into mass production. “From the very start, we had the idea of launching a construction kit system which would meet the needs of customers from specific sectors openly,” said Jörg Heckel from Bosch Rexroth. Depending on the product portfolio and workstation, Bosch puts together a suitable robotic arm, a 3D sensor and a gripper and equips a suitable software package with the necessary algorithms. At the end of May 2022, the project team will officially unveil the new custom-made system at the Logimat logistics trade fair in Stuttgart.
With the help of various algorithms developed by Bosch, item picking robots are able to recognize specific products in boxes, pick them out and place them in shipping packages. The various USPs which are offered by the system and which were developed through to the market launch stage by Bosch Research and Bosch Rexroth can be seen here:
Unique Selling Propositions of the Smart Item Picking system
With the help of classic computer vision algorithms and modern AI methods, individual objects in a crate can be recognized regardless of whether they have been stacked in an orderly or random fashion. No prior knowledge or CAD models of the articles are needed for object recognition purposes.
The robot plans a path for reaching into the box to pick out an object without colliding with obstacles in the environment or other objects in the box. With the help of the collision-free path planner, an optimum path for the robotic arm and the best possible gripping action can be determined.
Because Smart Item Picking was developed as a construction kit, a range of industrial robots with various reaches and load handling capacities can be combined with different types of 3D vision sensors, whether they be high-precision or low-cost models.
“Which object should the robotic arm pick first from the objects it has recognized in the box?” A great deal of thought went into answering this question and numerous tests were carried out. This is part of Bosch Research’s core know-how in the area of bin picking applications.
A universal gripper is used so that numerous different components can be handled without having to change the gripper in between. Ensuring that this gripper can be used for a wide range of objects yet can reliably grasp specific parts is a major challenge.
In addition to software algorithms and AI-methods from Bosch Research for the Smart Item Picking system, Bosch Rexroth also has its own system engineering and construction facilities – something which makes Bosch unique among its competitors and start-up in this field.
Christoph Marx
Christoph Marx studied mechanical engineering and mechatronics at Esslingen University of Applied Sciences in Germany. During his studies, he was already involved in industrial robotics and software development and contributed to the development of the automatic production assistant "APAS" in his final thesis at Bosch Research. Since 2016, Christoph Marx has been working as a research engineer at Bosch Research. Until 2019, he and his team dealt with robotic systems in production. The team's focus here was always to simplify the programming of robots using specific path planning algorithms and 3D image processing methods. Another focus has been two-arm robot systems that mimic humans and their use in typical manual workstations in Bosch production. Since 2019, Marx and his colleagues have been working on automating picking tasks in warehouses, which is becoming increasingly important due to the steady growth in e-commerce. Marx has also recently started representing Bosch in the non-profit association "euRobotics" of the European robotics community, where, among other things, influence is exerted on future robotics topics of the European Research Framework Programs.