Our lab has been leading and participating in several effort for enabling performance quantification and comparison in robotic manipulation.
With the YCB object and model set, which is current being administered by UMass Lowell NERVE Center, is maintained since 2015, when the effort is kicked off.
We have also been editing a special issue on benchmarking in robotic manipulation, the collection of which is summarized here.
We have also published a benchmark for robotic pick-and-place in clutter by adapting the well-known box-and-blocks test to robotics assessment. This work was done in collaboration in collaboration with Yale University.
We have also been participating in the organization of robotic manipulation competitions over the years. Robotic Grasping and Manipulation Challenge and the OCRTOC has been conducted in ICRA, IROS and remotely in collaboration with University of South Florida, National Institutes of Standards and Technology (NIST), and UMass Lowell. You can find the discussions on these competitions in the papers below.
We are collaborating with Institut de Robòtica i Informàtica Industria and UMass Lowell to form a cloth manipulation object set. We are aiming for a set of cloth objects with different properties and sizes to be used to design benchmarking protocols.
Moreover, for the robotics waste sorting research, we have been collecting and disseminating datasets for recycling. More information on this effort can be found in the corresponding project page here.
Papers related to benChmarking and Datasets
ZeroWaste Dataset: Towards Deformable Object Segmentation in Extreme Clutter
D. Bashkirova, M. Abdelfattah, Z. Zhu, J. Akl, F. Alladkani, P. Hu, V. Ablavsky, B. Calli, S. A. Bargal, and K. Saenko
Conference on Computer Vision and Pattern Recognition (CVPR), 2022
[Paper] [Dataset]
Household Cloth Object Set: Fostering Benchmarking in Deformable Object Manipulation
I. Garcia-Camacho, J. Borràs, B. Calli, A. Norton, and G. Alenyà
accepted to IEEE Robotics and Automation Letters, 2022
[Paper]
Research Challenges and Progress in Robotic Grasping and Manipulation Competitions
Y. Sun, J. Falco, M. A. Roa and B. Calli,
IEEE Robotics and Automation Letters, 2022
[Paper]
OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation
Z. Liu, W. Liu, Y. Qin, F. Xiang, S. Xin, M.A. Roa, B. Calli, H. Su, Y. Sun, and P. Tan
IEEE Robotics and Automation Letters, vol. 7, no. 1, pp. 486-493, Jan. 2022
[Paper]
Guest Editorial: Introduction to the Special Issue on Benchmarking Protocols for Robotic Manipulation
B. Calli, A. Dollar, M. A. Roa, S. Srinivasa, and Y. Sun
IEEE Robotics and Automation Letters, 6(4), pp.8678-8680, 2021
[Paper]
“Benchmarking Cluttered Robot Pick-and-Place Manipulation with the Box and Blocks Test”,
S. Morgan, K. Hang, W. Bircher, F. Alladkani, A. Gandhi, B. Calli, A. M. Dollar,
IEEE Robotics and Automation Letters, 2019.
[Paper] [Supplementary] [Video]