The test spec
Every builder receives the same short brief, versioned as booking-app-v1. The spec is intentionally small so that runs stay comparable over time:
- A public landing page listing three services
- A booking form (name, email, date, service) that stores submissions
- A confirmation email to the customer
- An admin view listing all bookings
- Usable layout down to 390 px wide
- A publicly reachable deployment
Payment, authentication and multi-language are out of scope for v1. They change the difficulty enough to make results incomparable, so they move to a separate spec version.
What is measured
| Metric | Method |
|---|---|
| Time to a working build | Timestamps recorded from first prompt to successful deploy |
| Prompt iterations | Counted from the run log |
| Cost | Taken from the account's credit usage after the run |
| Feature pass / fail | Automated browser test against the deployed URL |
| Layout breakage | Screenshots at 390 / 768 / 1440 px plus overflow check |
| Performance and accessibility | Lighthouse run against the live page |
| Export and handover | Can the code be exported, and is it usable outside the platform |
What we do not measure
We do not score taste, design quality, or "how it feels". Those are real buying factors, but they cannot be reproduced, and a score nobody can check is a marketing claim. If we mention them, we label them as a judgement, not a measurement.
Verification
Every published claim is linked to the artifact that supports it: a log, a screenshot, a metrics file or a billing line. A page cannot ship while a factual claim lacks a reference — the build rejects it.
Results are re-run when a builder ships a major change, and each run is dated. An old result stays visible with its date rather than being silently overwritten.
Corrections
If a builder changes behaviour after publication, the run is repeated and the page is updated with a changelog entry. If we got something wrong, the correction is noted in the same entry — not quietly deleted.