Paper: https://arxiv.org/pdf/2506.00103

Difficulty: Very Hard

Notes: Full solution should include a pipeline to generate pairwise training samples from existing LLMs+rubrics or public sources, an environment to train GenRM, and an environment to train main model using GenRM. Requires some creative decision-making; discuss proposal with Will before getting too deep into it, will give sufficient compute for train experiments conditional on implementation progress.

Paper: https://arxiv.org/abs/2410.07095

Code: https://github.com/openai/mle-bench

Owner: https://x.com/creet_z

Difficulty: Very Hard

Paper: https://arxiv.org/abs/2504.01848

Code: https://github.com/openai/preparedness/tree/main/project/paperbench

Owner: https://x.com/_manan2005

Difficulty: Very Hard

Paper: https://openpipe.ai/blog/art-e-mail-agent

Code: https://github.com/OpenPipe/ART/tree/main/examples

Owner: https://x.com/dhruvrnaik

Difficulty: Easy-Hard

Notes: Bounty: $200 - 800. Easy = ART-E agent task, Hard = 2-way portability between ART <-> verifiers tasks

Paper: https://arxiv.org/abs/2507.20534

Owner: https://x.com/kalocide

Difficulty: Hard

Code: https://github.com/web-arena-x/webarena/tree/main

Owner: https://x.com/fido20222

Difficulty: Hard

Owner: https://x.com/NoisyTails

Difficulty: Hard

Paper: https://pebble-potato-fc6.notion.site/SWE-Swiss-A-Multi-Task-Fine-Tuning-and-RL-Recipe-for-High-Performance-Issue-Resolution-21e174dedd4880ea829ed4c861c44f88

Code: https://github.com/zhenyuhe00/SWE-Swiss

Dataset(s): https://huggingface.co/SWE-Swiss

Train/Eval?: Train

Owner: https://x.com/SrgntSaltNPepa

Difficulty: Medium-Hard

Notes: Bounty: $400 - 800. $400 = existing RL dataset, $800 = full auto-gen pipeline

Paper: https://arxiv.org/abs/2503.00096

Code: https://github.com/Future-House/BixBench

Difficulty: Hard