EVIDENCE
What we measure, and how.
We do not claim to beat the largest AI models at everything. We claim that for operational work under rules, Relayn OS is a cheaper and more predictable alternative. These are the numbers behind that claim, and what they do not show.
01 / In production
Most requests no longer need a language model.
Source: 4,026 requests from our own production use, July 2026.
95% lower spending on AI inference (Feb–Apr 2026 vs our earlier model-first version). Our own billing, not a controlled test.
02 / Benchmark
Close to frontier models, on one graphics card.
| Model | Accuracy | Typical response time | Full run |
|---|---|---|---|
| Claude Sonnet 4.5 | 91.8% | 2,663 ms | 1,564 s |
| Claude Opus 4.5 | 90.9% | 2,462 ms | 1,504 s |
| Claude Haiku 4.5 | 89.5% | 1,285 ms | 716 s |
| Relayn, first layer | 89.3% | 1,136 ms | 636 s |
| Amazon Nova 2 Lite | 88.2% | 1,035 ms | 529 s |
| Qwen2.5-14B, run locally | 85.0% | 888 ms | 646 s |
- Relayn was the best of the six at recognising when no action fits (92.1%).
- It finished the full run in under half the time of the most accurate model.
- It was 8.5 to 10 points behind where several tools could apply. We are investigating; part of the gap is Relayn asking a question where the test expects an action.
03 / Limits
What these numbers do not show.
- The benchmark tested the first layer only.
- A clarifying question counts as a failure, even when asking is right.
- It does not measure wrong actions from the full system.
- Hosted model times include network round trip.
Download the brief (PDF, November 2026) · Ask a technical question