Robot design is a complex and time-consuming process that requires specialized expertise. Gaining a deeper understanding of robot design data can enable various applications, including automated design generation, retrieving example designs from text, and developing AI-powered design assistants. While recent advancements in foundation models present promising approaches to addressing these challenges, progress in this field is hindered by the lack of large-scale design datasets. In this paper, we introduce RobotDesign1M, a large-scale dataset comprising 1 million samples. Our dataset features multimodal data collected from scientific literature, covering various robotics domains. We propose a semi-automated data collection pipeline, enabling efficient and diverse data acquisition. To assess the effectiveness of RobotDesign1M, we conduct extensive experiments across multiple tasks, including design image generation, visual question answering about designs, and design image retrieval. The results demonstrate that our dataset serves as a challenging new benchmark for design understanding tasks and has the potential to advance research in this field. RobotDesign1M will be released to support further developments in AI-driven robotic design automation.
We design a semi-automated data collection pipeline. Specifically, we collect and extract images, texts from scientific documents. In order to filter images related to robot design, we manually annotate 32K images and train a classification model. Next, we clean the extracted images with the trained model, resulting in robot design image-text pairs. In addition, we construct visual-instruction tuning data by using a high-performing open-sourced large language model to generate over 1M question-answer pairs.
RobotDesign1M significantly covers a wide range of keywords and outperforms related datasets in terms of vocabulary size.
A glimpse of image–text pairs from RobotDesign1M. Each sample pairs a robot design image with a generated question and answer. Use the arrows to browse left and right through the examples.
QWhat components make up the foot board design as shown in the figure?
AThe foot board design includes the forefoot and the hind paw.
QHow are the fly wheels attached to the bicycle as illustrated in the figure?
AThe fly wheels are connected by a developed fly wheel cage.
QWhat is the purpose of the rods in the figure?
AThe rods are lead screws, which can extend and shorten, leading to the posture change of the footplate.
QWhat mechanism is utilized in the system to achieve easy handling and positioning as shown in the figure?
AThe system uses a lock system to achieve easy handling and positioning.
QHow are the twist angles of opposite links related in the figure?
AOpposite links have equal twist angle.
QWhat is the difference in length between the leftwards and rightwards pillars in the picking head?
AThe leftwards pillar is longer than the rightwards pillar.
QHow many degrees of freedom does each robot arm have as illustrated in the figure?
AEach arm has 4-DOF.
QWhat type of fastener is used in the target design to reduce assembly complexity?
AA rotationally activated tensile fastener is used.
QWhat type of components are mounted on the base along with the motors in the figure?
AThe figure shows gearboxes mounted on the base along with the motors.
QWhat type of mechanisms are used to design the gripper of the snake-like robot as shown in the figure?
AThe gripper is designed using two parallel four-bar mechanisms.
@inproceedings{le2026robotdesign1m,
title={RobotDesign1M: A Large-scale Dataset for Robot Design Understanding},
author={Le, Tri and Nguyen, Toan and Tran, Quang and Nguyen, Quang and Huang, Baoru and Nguyen, Hoan and Vu, Minh Nhat and Ta, Tung D. and Nguyen, Anh},
booktitle={2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year={2026},
organization={IEEE}
}