Thread by @Lawrence_Y_Chen

Can a robot open a crumpled plastic bag and pack objects into it? We’re developing algorithms for autonomous bagging. @UCBerkeley BAIR @berkeley_ai, @CMU_Robotics, and @ToyotaResearch #ICRA2023 🧵👇 (1/11)

🔎 Thin plastic bags are ubiquitous. “Bagging” is useful for many tasks such as grocery shopping, cleaning, recycling, and packing. (2/11)

While easy for humans, bagging is very challenging for robots. Bags can deform in many ways, creating complex self-occlusions. Furthermore, many plastic bags are reflective, or translucent, making perception challenging. (3/11)

Bags are also difficult to manipulate because different parts of the bag move differently, making it difficult to orient. Moreover, a one-millimeter difference in the gripper height can also significantly affect grasping. (4/11)

💡 To address these challenges, the robot learns a novel representation of plastic bags with segmentation of bag handles and rim. Inspired by All You Need is LUV (arxiv.org/abs/2203.04566), we use UV light to generate labeled training data under self-supervision: (5/11)

🎨 We paint a set of training bags with UV-fluorescent paints. The bag looks normal under regular lighting. But under UV lights, the paints glow with unique colors, allowing the robot to collect high-accuracy ground-truth labels automatically 🌟. (6/11)

🤖🦾 The robot systematically explores the bag state space by sampling actions to manipulate the bag into diverse configurations. This makes data collection efficient and scalable. (7/11)

We use the collected data to train a segmentation model that recognizes the bag rim and handles at execution time without UV lights. We propose two metrics to quantify how large and how round the bag opening is. (8/11)

With the perception module, our algorithm divides the bagging task into 3 stages: (1) Orienting the bag upward, (2) enlarging the opening, and (3) inserting the objects and lifting the bag. (9/11)

Physical experiments are encouraging 🔥 (10/11)

💥 This is a collaboration with @BaiyuShi147 @Ken_Goldberg @UCBerkeley @AUTOLab_Cal + @daniel_t_seita @davheld @CMU_Robotics + Richard Cheng @tkollar @ToyotaResearch.
👀 Paper: arxiv.org/abs/2210.17217
🌐 Project website: sites.google.com/view/autobag (11/11)