EVIDENCE

What we measure, and how.

We do not claim to beat the largest AI models at everything. We claim that for operational work under rules, Relayn OS is a cheaper and more predictable alternative. These are the numbers behind that claim, and what they do not show.

01 / In production

Most requests no longer need a language model.

How 4,026 production requests were handled: 71% by rules, 18% with one model call, 9% with web research, 2% by a multi-step agent.

Source: 4,026 requests from our own production use, July 2026.

95% lower spending on AI inference (Feb–Apr 2026 vs our earlier model-first version). Our own billing, not a controlled test.

02 / Benchmark

Close to frontier models, on one graphics card.

Accuracy of six systems on 440 function-calling cases; Relayn's first layer 89.3%, the best frontier model 91.8%.
ModelAccuracyTypical response timeFull run
Claude Sonnet 4.591.8%2,663 ms1,564 s
Claude Opus 4.590.9%2,462 ms1,504 s
Claude Haiku 4.589.5%1,285 ms716 s
Relayn, first layer89.3%1,136 ms636 s
Amazon Nova 2 Lite88.2%1,035 ms529 s
Qwen2.5-14B, run locally85.0%888 ms646 s
  • Relayn was the best of the six at recognising when no action fits (92.1%).
  • It finished the full run in under half the time of the most accurate model.
  • It was 8.5 to 10 points behind where several tools could apply. We are investigating; part of the gap is Relayn asking a question where the test expects an action.

03 / Limits

What these numbers do not show.

  • The benchmark tested the first layer only.
  • A clarifying question counts as a failure, even when asking is right.
  • It does not measure wrong actions from the full system.
  • Hosted model times include network round trip.

Download the brief (PDF, November 2026) · Ask a technical question