What AI recommends vs. what a coding agent installs
We gave Codex CLI 10 ordinary build tasks, 3 runs each, in fresh containers, with no product named in any prompt, and recorded every service and library it installed, configured or built itself. Beside each choice is what the 10 chat models on Modelocracy recommend when asked the matching buying question.
- Apps built
- 30
- Choices recorded
- 117each audited by hand
- Agent matches chat's #1
- 3 of 11capabilities with chat answers
- Built its own
- 2capabilities where chat recommends a product
Chat vs. code
Runs that used each product, out of runs whose task needed it · Chat places products by position in the panel's consensus, every wording weighted equally, with the share of answers that named it · GPT-6 Luna is the panel's OpenAI seat, placed by its own answers
| Capability | Codex used | Chat's #1 | Codex's choice in chat | GPT-6 Luna's #1 | Result |
|---|---|---|---|---|---|
| Authentication tasks 1, 4, 5 | Supabase Auth 3/9 Better Auth 2/9 Built by hand 1/9 | Auth0 | Supabase Auth: #6 named in 51% · GPT-6 Luna #7 Better Auth: never named | Auth0 | Differs from chat |
| Database tasks 1, 2, 4 | SQLite 3/9 Supabase 3/9 Built by hand 3/9 | PostgreSQL | SQLite: #4 named in 74% · GPT-6 Luna #4 Supabase: #6 named in 76% · GPT-6 Luna #5 | PostgreSQL | Differs from chat |
| Payments task 1 | Stripe 3/3 | Stripe | Stripe: #1 named in 99% · GPT-6 Luna #1 | Stripe | Same as chat |
| Email sending tasks 1, 8 | Resend 3/6 Built by hand 3/6 | Postmark | Resend: #5 named in 97% · GPT-6 Luna #3 | Amazon SES | Differs from chat |
| Hosting tasks 1, 4, 10 | Render 2/9 Fly.io 1/9 | AWS | Render: #6 named in 97% · GPT-6 Luna #7 Fly.io: #7 named in 86% · GPT-6 Luna #9 | AWS | Differs from chat |
| File storage tasks 3, 7 | Built by hand 6/6 | Amazon S3 | no product used | Amazon S3 | Built its own |
| Realtime task 4 | Socket.IO 3/3 | Ably | Socket.IO: #4 named in 71% · GPT-6 Luna #19 | Ably | Differs from chat |
| Error monitoring task 6 | Sentry 3/3 | Sentry | Sentry: #1 named in 100% · GPT-6 Luna #1 | Sentry | Same as chat |
| Logging task 6 | Sentry 2/3 Built by hand 1/3 | Datadog | Sentry: #8 named in 78% · GPT-6 Luna #9 | Datadog | Differs from chat |
| Caching task 9 | Built by hand 3/3 | Redis | no product used | Redis | Built its own |
| CI task 10 | GitHub Actions 3/3 | GitHub Actions | GitHub Actions: #1 named in 99% · GPT-6 Luna #1 | GitHub Actions | Same as chat |
Codex runs on OpenAI's gpt-6.1-sol. In chat, GPT-6 Luna is a different OpenAI model, so its column shows what the same lab recommends when asked rather than what the agent itself would say.
Task by task
What each task's final repository used, out of its 3 runs · Install, build and test results from the project's own scripts
| # | Task | Used | Checks |
|---|---|---|---|
| 1 | Build a minimal SaaS starter: email sign-in, a projects list stored in a database, paid subscriptions, and a welcome email. Make it deployable. Empty folder | Supabase Auth 3/3, Supabase 3/3, Resend 3/3, Next.js 3/3, React 3/3, TypeScript 3/3, Stripe 3/3, Node.js test runner 1/3 | Install: 3 pass Build: 3 pass Tests: 2 none, 1 pass |
| 2 | Build a REST API for a todo app with persistent storage and tests. Empty folder | SQLite 3/3, Web framework built by hand 1/3, Express 1/3, FastAPI 1/3, JavaScript 2/3, Python 1/3, Node.js test runner 2/3, pytest 1/3 | Install: 3 pass Build: 3 none Tests: 1 fail, 2 pass |
| 3 | Build a web app where users upload PDFs and ask questions about them. Empty folder | File storage built by hand 3/3, React 3/3, Vite 3/3, JavaScript 3/3, LLM SDK built by hand 3/3 | Install: 3 pass Build: 3 pass Tests: 1 fail, 2 none |
| 4 | Build a real-time team chat web app. Empty folder | Database built by hand 3/3, Express 3/3, React 3/3, Vite 3/3, JavaScript 3/3, Socket.IO 3/3 | Install: 3 pass Build: 3 pass Tests: 3 none |
| 5 | Add user authentication. Next.js app | Better Auth 2/3, Authentication built by hand 1/3, SQLite 2/3, SQLite (libSQL) 1/3, Kysely 1/3, Node.js test runner 2/3 | Install: 3 pass Build: 3 pass Tests: 2 pass, 1 none |
| 6 | Add error monitoring and logging so we hear about production errors. Next.js app | Sentry 3/3, Logging built by hand 1/3, Sentry 2/3, Node.js test runner 1/3 | Install: 3 pass Build: 3 pass Tests: 1 pass, 2 none |
| 7 | Add avatar uploads for users. Next.js app | File storage built by hand 3/3, Sharp 2/3, Node.js test runner 1/3 | Install: 3 pass Build: 3 pass Tests: 3 none |
| 8 | Add a background job that sends a weekly digest email. FastAPI app | SQLite 1/3, Email sending built by hand 3/3, Job queue built by hand 3/3, cron 1/3 | Install: 3 pass Build: 3 none Tests: 3 pass |
| 9 | The /reports endpoint is slow; add caching. FastAPI app | Caching built by hand 3/3 | Install: 3 pass Build: 3 none Tests: 3 pass |
| 10 | Set up CI that runs the tests on every pull request and deploys on merge to main. FastAPI app | GitHub Actions 3/3, Fly.io 1/3, Render 2/3 | Install: 3 pass Build: 3 none Tests: 3 pass |
Every prompt ends with the same line: “Work without asking me questions; make the choices you'd recommend.”
Against Claude Code
Codex's most used product per capability here, against Amplifying's study of Claude Code 2.1.39 (Sonnet 4.5, Opus 4.5, Opus 4.6) on the same stack
| Capability | Stack | Claude Code (Amplifying) | Codex here | Result |
|---|---|---|---|---|
| CI | Python | GitHub Actions all repos (93.8% overall) | GitHub Actions 3/3 | Same |
| Error monitoring | Next.js | Sentry all repos | Sentry 3/3 | Same |
| Payments | Next.js | Stripe 100% | Stripe 3/3 | Same |
| Hosting | Python | Railway 82% | Render 2/3 | Different |
| Database | Next.js | PostgreSQL 90% | Supabase 3/3 | Different |
| Authentication | Next.js | NextAuth.js 91% | Supabase Auth 3/6 | Different |
| Email sending | Next.js | Resend 88% | Resend 3/3 | Same |
| Email sending | Python | Built by hand 43% | Built by hand 3/3 | Same |
| File storage | Next.js | Vercel Blob 48% | Built by hand 3/3 | Different |
| Caching | Python | Redis 57% | Built by hand 3/3 | Different |
| Job queue | Python | Celery 38% | Built by hand 3/3 | Different |
The two studies used different prompts, dates and models, so these rows describe each agent's habits rather than a head-to-head test. Supabase runs on PostgreSQL, so the database row is a difference of packaging more than of engine.
How it was run
- Codex CLI 0.164.0 on its default model,
gpt-6.1-sol, one fresh container per run, network on, its web search off, and no credentials for any app service. - A choice counts when the final repository installs, configures or builds it: package manifests, imports and configuration first, then a separate model for what those miss. Every repository was then audited by hand. Products only mentioned in a README do not count.
- Chat answers are the panel's usual answers, from what each model already knows, on the buying question for each capability: four wordings, 3 answers per model per wording.
- Download every run and choice with the file and line behind it, the summary behind this page, or every chat answer.
Caveats
- Each task ran 3 times, so a 3/3 result has a 95% interval of 44% to 100%. These are one agent's habits on one model, measured on October 25, 2026.
- No run chose an OpenAI product. The one task that invited a model provider, answering questions about uploaded PDFs, was built in every run by matching passages locally, so this sample says nothing either way about whether an agent favors its own lab.
- The first extraction pass, tuned on the first 12 runs, found 32 of 54 choices in the next 18. After fixes it finds 113 of 117 across all 30; the other 4 came from the model pass and were checked by hand. That rate is measured on the same runs it was fixed on.
- Every run ended normally, but finishing is not working: tests passed in 15 runs, failed in 2 and did not exist in 13.
- Logging is compared with the observability question, hosting with the startup cloud question and CI with the CI/CD question; each is close to, not identical with, what the task asked.