There is significant overlap between robotics competitions and machine learning competitions. In general, the involvement of physical hardware in robotics competitions brings extra challenges for competition organisers and participants.
Our blog post on Robotics Competitions at ICRA 2023 took a detailed look at 12 recent robotics competitions at a top conference. In this post we summarise some themes from these competitions, and robotics competitions in general.
Motivation
Firstly, why should robotics competitions happen at all? Organising a competition often involves an enormous amount of work, as does participating.
From the competition organisers’ perspectives, there are numerous good reasons:
- To foster standardisation, create benchmarks, and stimulate collaboration in specific types of academic or industrial problems. This seemed to be a strong driver behind the quadruped challenge, the grasping and manipulation challenges, the PUB.R challenge, the physically assistive robot challenge, and the BARN challenge.
- As an educational tool, to provide an accessible starting point for practical robotics or give participants experience with expensive hardware that they might not otherwise have access to. The manufacturing robotics challenge, BARN challenge, and F1Tenth seemed to share this goal.
- To increase awareness of a particular problem in the research community. Vanderlande stated this as their reasoning for sponsoring the stacking challenge, which was directly related to a core problem their business deals with.
- To increase awareness of and familiarity with particular tools or products. The humanoid robot wrestling competition introduced participants to the webots simulator, and the virtual manipulation challenge shared improvements that Fraunhofer IPA had made to widely-used simulators for use in certain problems.
Participants, also, benefit from competing in various ways:
- The obvious academic kudos or tangible prize associated with winning or performing well in a competition. Some of the competitions at ICRA 2023 had a prize pool of $10,000.
- The ability to compare their solution to a particular problem against others, in a standardised way. The academic literature can only capture so much of the experimental design, and in-person comparison on standardised problem sets gives significant insight. Unlike pure software-based disciplines, where tools like Docker containers greatly aid reproducibility, the variability of the real-world can make it difficult to demonstrate progress in robotics. In addition to benchmarking, comparing notes with other teams working on exactly the same problem is a great way to do collaborative research — as we saw in the grasping and manipulation challenges.
- To get experience with expensive hardware. The Clearpath Jackal robots in the BARN challenge and the Kuka arms in the manufacturing robotics challenge cost tens of thousands of dollars each. By giving participants the chance to run their own software on them for a few days (after having to do some work in simulation, or otherwise showing aptitude), people who would not otherwise have a chance to use these robots can gain practical experience, without necessarily needing to be in a university or company lab.
- The pressure of a competition can be a useful motivator for pushing forward on a particular problem and gaining practical hands-on experience. At ICRA 2023, several competition participants mentioned that their day-to-day research was more theoretical, and they chose to take part in a competition to better understand the problems faced when bringing robotics applications into the real world. As one of them put it, half jokingly, “we thought it would be less work than it was!”
- Competitions can be very useful for teams that are aiming to commercialise their technology. Both as a test case that is somewhat closer to the real-world — the uncontrolled environment of a conference hall brings unexpected challenges not usually faced in the lab — and as a way to showcase capabilities and gain recognition. This was clearly part of the motivation for some teams in the PUB.R challenge and the stacking challenge.
Types of Competitions
There is no one template for robotics competitions, but there are a few common ways to run them. We’ve categorised them into the following buckets:
- In-person: Live, in-person evaluation. Many competitions fall into this bucket, for example F1Tenth and the quadruped challenge.
- Simulation-based: Fully software-based. For example, the virtual manipulation challenge or humanoid robot wrestling .
- Remote (centralised): Physical robots, based in the organisers’ lab, with participants able to control them remotely. For example Robomaster University Sim2Real.
- Remote (distributed): Physical robots, based in the participants’ lab, with organisers conducting evaluation remotely. For example, the grasping and manipulation challenges.
Additionally, while most competitions have weeks or months of asynchronous preparation followed by a few days of evaluation, some follow a hackathon format — a few days of intense synchronous work during the competition event.
Choosing the right way to run a particular competition is critical.
For example, simulation-based competitions have the lowest barrier to entry, and are great when the number of participants or diversity of solutions is more important than the real-world fidelity of the solutions.
For competitions requiring lots of expensive and hard-to-transport hardware, a remote approach is likely to work best — unless the challenging in-person environment is core to the challenge, and teams are willing to make the investment to compete in-person.
On remote evaluation — while centralised remote evaluation is better for direct comparison and ensuring a level playing field, competitions involving highly custom hardware are likely to be better suited to distributed remote evaluation (see the cloth manipulation challenge for more discussion on this).
Key Events
We list ongoing robotics competitions on our competition listing page.
These are some regular events that host interesting robotics competitions:
- IROS; the International Conference on Intelligent Robots and Systems. The 2023 edition is in Detroit, October 1-5.
- RoboCup; the 2023 edition is in Bordeaux, July 4-10.
- ICRA; the 2024 edition will take place in Yokohama, Japan, from 13-17 May 2023.
- NeurIPS, a machine learning conference, also tends to have a few robotics conferences in its competition track. The 2023 edition is in New Orleans, December 10-16.
About ML Contests
For over three years now, ML Contests has provided a competition directory and shared insights on trends in the competitive machine learning and robotics space. To receive occasional updates with key insights like this conference summary, subscribe to our mailing list. You can also follow us on Twitter, or join our Discord community.