Accompanying resource for the paper AI Competitions and Benchmarks: Competition Platforms. Current status: draft; features still to be verified.
Top Platform Features
| Criteria | AICrowd | Codabench | CodaLab | DrivenData | EvalAI | Kaggle | Tianchi | Zindi |
|---|---|---|---|---|---|---|---|---|
| Design Support | ✓ | - | - | ✓ | - | ✓ | - | ✓ |
| Data Support | ✓ | - | - | ✓ | ✓ | ✓ | - | ✓ |
| Code sharing | ✓ | - | ✓ | - | ✓ | ✓ | ✓ | - |
| Code evaluation | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - |
| Custom metrics | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Staged contests | ✓ | ✓ | ✓ | ✓ | ✓ | - | - | - |
| Private evaluation | - | ✓ | ✓ | - | ✓ | - | - | - |
| RL-friendly | ✓ | ✓ | ✓ | - | ✓ | ✓ | - | - |
| Judging panel | ✓ | ? | ? | ✓ | ? | ✓ | ? | ? |
| HITL evaluation | ✓ | - | - | - | ✓ | ? | ? | - |
| Run for free | - | ✓ | ✓ | - | ✓ | ✓ | - | ✓ |
| Open-source | - | ✓ | ✓ | - | ✓ | - | - | - |
Competition Platforms
Broader comparison. Users as of end of 2023. Number of competitions and total prize money represents full year 2023.
For more information, see State of Competitive ML in 2023.
| Platform | Founded | Users | Competitions | Total prize money |
|---|---|---|---|---|
| AIcrowd | 2017 | 123k+ | 17 | $198,000 |
| Bitgrit | 2017 | 35k+ | 2 | $6,000 |
| Codabench | 2023 | 3.5k+ | 3 | $10,000 |
| CodaLab | 2013 | 55k+ | 71 | $218,000 |
| DrivenData | 2014 | 100k+ | 11 | $1,762,000 |
| DS Works | 5 | $127,000 | ||
| EvalAI | 2017 | 40k+ | 33 | $96,000 |
| Grand Challenge | 2010 | 94k+ | 22 | $40,000 |
| Hugging Face | 2023 | 5 | $4,000 | |
| Humyn.ai | 2023 | 14k+ | 5 | $41,000 |
| Kaggle | 2010 | 17m+ | 41 | $2,358,000 |
| Lab42 | 2022 | ~500 | 3 | $87,000 |
| Onward | 2022 | 3.2k+ | 3 | $110,000 |
| Signate | 2014 | 80k+ | 12 | $163,000 |
| Solafune | 2020 | 3 | $30,000 | |
| Tianchi | 2014 | 61 | $852,000 | |
| Trustii | 2020 | 1.5k+ | 2 | $60,000 |
| Zindi | 2018 | 72k+ | 21 | $134,000 |
| Other | 47 | $1,518,000 |