Open source on GitHubWork in a realistic environment
Agents get a realistic environment to work in, with project state, context, and access to the development tools they need.
Evaluating agentsacross Supabase
We evaluate model experiments across the Supabase developer journey, from building and deploying to investigating and resolving production issues, with real project context.
| Agent | Total | ||||
|---|---|---|---|---|---|
 Claude Code / Opus 5.5 (medium) | 93% | ||||
| Claude Code / Sonnet 5.5 (medium) | 86% | ||||
  OpenCode / Kimi K3 | 86% | ||||
  Grok Build / Grok 4.7 (high) | 83% | ||||
  Codex / GPT-6.1 sol (medium) | 81% | ||||
| Codex / GPT-6 luna (medium) | 77% |
Work through a task
Agents take on a task from somewhere along the Supabase developer journey, from building and deploying to investigating and resolving issues.
Results are evaluated
Scoring draws on SQL checks, client calls made as real users, and the files the agent creates. When broader assessment is needed, an LLM judge reviews the result.
Why these experiments
- Claude Code / Opus 5.5 (medium)
Claude Code's default model on most plans, at its default effort.
- Claude Code / Sonnet 5.5 (medium)
Claude Code's Sonnet model for daily coding tasks, at its default effort.
- Codex / GPT-6 luna (medium)
A cost-effective model for everyday tasks, and the one behind the Supabase dashboard assistant.
- Codex / GPT-6.1 sol (medium)
OpenAI's recommended Codex model for paid ChatGPT plans, run at medium effort.
- Grok Build / Grok 4.7 (high)
Grok Build's default model, at its default effort.
- OpenCode / Kimi K3
Included as an open-weight option in OpenCode.
Experiment upload dates
| Claude Code / Opus 5.5 (medium) | |
|---|---|
| Claude Code / Sonnet 5.5 (medium) | |
| Codex / GPT-6 luna (medium) | |
| Codex / GPT-6.1 sol (medium) | |
| Grok Build / Grok 4.7 (high) | |
| OpenCode / Kimi K3 |