Benchmarks & Evidence

Q: Why does this page not contain EDDI benchmark numbers?

Because we have not run the harness yet, and publishing numbers we have not measured is how benchmark pages lose their credibility. What this page does publish is the evidence that the underlying mechanisms work, the conditions that determine how much they help, and the exact measurement plan, so that when the numbers land you can check them against a methodology that was fixed in advance.

Q: How should EDDI's cascade cost figure be read?

EDDI quotes the range measured in production, 31% to 58% lower inference cost at held quality, rather than the benchmark ceiling. Published savings span roughly 30% to 98% depending on how many queries are genuinely easy, how well calibrated the escalation signal is, and how far apart the model tiers are priced. Your number depends on your workload, and nobody can quote it to you in advance.

Q: Are virtual threads always faster?

No. The literature is consistent that virtual threads improve throughput on blocking, I/O-bound workloads and impose overhead on CPU-bound ones. One study found them slightly slower than platform threads on a CPU-bound parallel task, and a Quarkus-specific study found virtual threads did not match Quarkus-reactive in a resource-constrained container. EDDI's workload is I/O-bound, which is the case the mechanism suits, but that is an argument about workload fit rather than a universal claim.

Q: Can we run these benchmarks ourselves?

That is the intent. EDDI is Apache 2.0 and self-hosted, so nothing prevents you measuring it on your own hardware with your own workload. Given that cost savings depend heavily on workload composition, your own measurement is worth more than ours.

Why this page is shaped like this

Vendor benchmark pages have a credibility problem, and it is deserved. The numbers are produced by the party they flatter, on hardware nobody names, against a baseline chosen after the fact.

So this page separates three things that usually get blended together:what independent published research establishes about the mechanisms EDDI uses, what EDDI claims and how those claims should be read against that evidence, and what has not been measured, stated plainly rather than omitted.

Every citation below resolves to a DOI, and each is labelledpeer-reviewed or preprint. Several of the most widely quoted cascade results, including FrugalGPT, are arXiv preprints. That does not make them wrong, but you should weigh them accordingly, and we would rather label them than have you discover it. Several of these results also cut against EDDI, and they are included for that reason.

Model cascading: what the evidence supports

Model cascading sends easy queries to a small model and escalates hard ones. The mechanism is well established and the savings are real. The range is very wide, and where you land in it is a property of your workload, not of the platform.

Study	Source	Reported result	Setting
FrugalGPT (2023)	arXiv preprint	Up to 98% cost reduction while matching the best single model	LLM cascade across commercial APIs; the widely cited upper bound, and an upper bound rather than a typical case
RouteNLP (2026)	arXiv preprint	58% cost reduction in production; 40 to 85% across a six-task benchmark	8-week pilot at an enterprise customer-service division, ~5K queries/day, 91% response acceptance maintained
Mixture-of-Thoughts cascade (2023)	arXiv preprint	Performance comparable to the stronger model at 40% of its cost	Six reasoning benchmarks, GPT-3.5-turbo escalating to GPT-4
SATER (2025)	arXiv preprint	Over 50% cost reduction, over 80% lower cascade latency	Three small language models across six datasets of varying complexity
Hybrid LLM (2024)	arXiv preprint	Up to 40% fewer calls to the large model with no drop in response quality	Quality-aware router with a tunable quality target at test time
UCCI (2026)	arXiv preprint	31% cost reduction (95% CI: 27 to 35%) at micro-F1 = 0.91	Production named-entity-recognition workload, 75,000 queries, 4B and 12B models on H100 GPUs, end-to-end routing on real outputs and measured latency
TensorOpera Router (2024)	EMNLP 2024, Industry Track peer-reviewed	Up to 30% cost reduction, up to 40% better query efficiency	Multi-model router across domain-specialist LLMs

Read the methodology, not just the headline

The UCCI authors make a point worth carrying into any cascade claim you evaluate, including ours: their results use end-to-end routing on actual model outputs and measured GPU latency, not simulated routing from global accuracies or nominal API prices. A large share of published cascade savings come from simulation. Simulated savings are systematically more optimistic than measured ones, which is part of why the most carefully measured figure in the table above is also the lowest.

What actually determines your number

How heterogeneous the workload is

Cascading only saves money when a large share of queries are genuinely easy. RouteNLP reports over 70% of queries at its enterprise partner were routine tasks well within a smaller model's capability. A workload where every query is hard has nothing to route away, and cascading will cost more than a single model because it pays for the first attempt twice.

How well calibrated the escalation signal is

A unified analysis of routing and cascading identifies quality estimators as the critical factor determining whether either strategy helps at all. UCCI makes the same point empirically: replacing an uncalibrated confidence score with a calibrated one cut expected calibration error from 0.12 to 0.03 and beat entropy thresholding and a FrugalGPT-style learned threshold at the same operating point.

How far apart the tiers are priced

Savings are bounded by the price gap between the cheap and expensive model. FrugalGPT observed fees differing by two orders of magnitude across commercial APIs, which is what makes 98% reachable at all. Two similarly priced models cannot produce a large saving no matter how good the router is.

How to read EDDI's own figure

EDDI's feature pages quote the two production measurements in the table:31% to 58% lower inference cost at held quality. We chose them over the benchmark results on purpose. They are the lowest numbers here, and the ones measured on real traffic rather than simulated, so they are the fairer expectation for a deployment you have not measured yet.

The honest version is a conditional: if a large share of your traffic is routine, if your escalation signal is calibrated, and if there is a wide price gap between your tiers, savings in that range or above are achievable and the literature supports it. If any of those three is absent, expect substantially less. We would rather you arrive at your own number than adopt ours.

JVM concurrency: what the evidence supports

EDDI runs on the JVM and uses virtual threads, on the argument that agent workloads are I/O-bound: they spend most of their time waiting on LLM responses, vector store queries, and tool calls. The literature supports that argument for that workload shape, and is equally clear about where it stops holding.

Study	Source	Reported result	Setting
Virtual threads in a Spring Boot portal (2026)	IRJAEM peer-reviewed	75% reduction in memory overhead, improved responsiveness under load	Data-intensive dashboard at peak loads of 2,000 concurrent users, versus platform threads
Adaptive thread type selection (2023)	IEEE APSEC 2023 peer-reviewed	Significant throughput improvement on blocking-operation workloads, but overheads on CPU-bound ones	JVM framework study; the authors build an adaptive switcher precisely because neither thread type wins everywhere
Virtual threads in Spring Boot 3 (2024)	Vestnik NKU peer-reviewed	About 20% performance improvement on concurrent network requests	1,000 concurrent network requests versus traditional threads
Virtual threads in Quarkus (2023)	ACM DEBS 2023 peer-reviewed	Virtual threads did NOT match Quarkus-reactive in a resource-constrained container	Typical container environment with scarce resources; attributed to a mismatch between Netty's assumptions and virtual threads
Memory constraints of virtual threads (2026)	IEEE MIPRO 2026 peer-reviewed	Scalability pressure shifts from CPU toward memory as concurrency rises	Fixed JVM heap sizes with concurrency increased until failure; heap configuration becomes the binding constraint
Virtual vs. platform threads in parallel work (2025)	IEEE MIPRO 2025 peer-reviewed	Virtual threads were slightly SLOWER than platform threads	CPU-bound parallel task (prime counting), not an I/O-bound workload

The inconvenient one

A study of virtual thread integration in Quarkus, the framework EDDI is built on, found that virtual threads did not perform as well as Quarkus-reactive in a resource-constrained container, attributing it to a mismatch between Netty's assumptions and virtual threads. That is a directly relevant negative result and it belongs on this page. It bounds the claim: virtual threads buy a simpler programming model at competitive throughput, not a guaranteed win over every alternative concurrency design.

Two further qualifications the evidence supports. Virtual threads impose overhead on CPU-bound work, so "millions of threads" is a statement about I/O-bound capacity and not a general throughput claim. And as concurrency rises,memory rather than CPU becomes the binding constraint, which means heap configuration is part of any honest capacity number.

Retrieval quality: chunking dominates

EDDI supports 8 embedding providers and 6 vector stores. The evidence says that choice matters far less than how documents are split, which is why ourRAG guide tells you to vary chunking before swapping models.

Study	Source	Reported result	Setting
Cross-domain chunking evaluation (2026)	arXiv preprint	Content-aware chunking reached nDCG@5 ~0.459 versus <0.244 for fixed-length splitting	36 segmentation methods, six knowledge domains, five embedding models
Chunk size for long-document retrieval (2025)	arXiv preprint	Smaller chunks (64 to 128 tokens) suit fact-based answers; larger (512 to 1024) suit contextual questions	Fixed-size strategies across short-form and long-form datasets, multiple embedding models
Adaptive chunking for clinical decision support (2025)	Bioengineering (MDPI) peer-reviewed	87% accuracy versus 50% for a fixed token baseline	Four otherwise identical RAG pipelines differing only in chunking strategy
Upstream document quality (2026)	medRxiv preprint	Under 20% of baseline chunks reached practical utility thresholds	Clinical protocol manuals; the authors conclude document quality constrains retrieval, "challenging assumptions regarding plug-and-play RAG deployment"
Chunking on academic texts (2026)	arXiv preprint	Cluster-based semantic chunking did NOT outperform simpler strategies	Long structured academic theses; a useful counterweight to the assumption that fancier chunking always wins

Note the counterweight in that table: cluster-based semantic chunking didnot beat simpler strategies on academic theses. More sophisticated chunking is not automatically better, which is another reason to measure on your own corpus rather than adopt a recommended default.

Governance: where the rules are enforced

EDDI's security model puts its rules in the engine, not in the prompt: tool approvals, allow-lists, vault references and the SSRF-safe HTTP client apply at the boundary where a tool call leaves the model. The research on prompt injection is the reason. Instructions hidden in a ticket, a web page or a tool result reach the model as ordinary text, and a model cannot reliably tell them apart from yours.

Unsafe tool actions under injected instructions, by defence (NetInjectBench, 240 attacks, preprint). *The static allowlist also blocked every approved change, so it was unusable.

Study	Source	Reported result	Setting
NetInjectBench (2026)	arXiv preprint	0 of 240 injected attacks led to an unsafe action behind a policy gate at the tool call, against 82.5% with no defence and 10 to 26% with prompt-level defences	130 network-operations scenarios on three 7 to 8B open models. A static allowlist reached 5%, but blocked every legitimate change
Indirect prompt injection (2023)	ACM AISec 2023 peer-reviewed	Retrieved content becomes instructions: data theft, worming, and control over which APIs an application calls	Demonstrated against real systems, including Bing Chat and code-completion engines
HouYi (2023)	arXiv preprint	31 of 36 commercial LLM-integrated applications were vulnerable to prompt injection	Black-box attack against deployed applications; 10 vendors confirmed the findings
Adaptive attacks on agent defences (2025)	arXiv preprint	All eight evaluated defences were bypassed, with attack success above 50%	Indirect prompt injection defences for tool-using agents, tested against attacks adapted to each defence
CaMeL (2025)	arXiv preprint	77% of tasks solved with provable security, against 84% undefended	AgentDojo. Control and data flow come from the trusted request, so untrusted data cannot change what the program does

How this maps to EDDI, and what it does not prove

EDDI's per-tool approval gates, glob allow and exempt lists, and vault-only credentials are execution-time boundaries of the kind these studies favour, and an approval binds to the resolved request rather than to a tool name. We havenot measured EDDI against these benchmarks, and a gate is only as good as the policy behind it: an allow-list that is too wide lets the attack through, and one that is too narrow blocks the work, as the static allowlist above did.

Human oversight: few gates, well placed

Human approval gates are one of EDDI's core controls. The literature says they work, and also that human attention is a finite resource. A gate that fires on everything trains people to click Approve.

Share of AI-assisted research runs with a critical failure (280 runs, preprint).

Study	Source	Reported result	Setting
Human decision gates (2026)	arXiv preprint	Critical failures fell from 72% to 16% of runs (Fisher's exact test, p < 0.001)	280 AI-assisted research runs with the same model and prompts; the gated version adds deterministic computation and three binding human decisions
Oversight has a capacity (2026)	arXiv preprint	Realized safety is an inverted U in the escalation rate: past a point, more human review makes the system less safe	125 hand-labelled agent actions with a fatiguing reviewer; reviewers agreed only moderately on what is risky (Fleiss' kappa 0.52)
Dynamic intervention (2026)	Discover Artificial Intelligence peer-reviewed	Task success statistically equivalent to full oversight, with a person reviewing 14.5% of steps and 89% lower latency	5,000 synthetic enterprise automation tasks; oversight allocated by a confidence score rather than at fixed checkpoints
Effective human oversight (2024)	ACM FAccT 2024 peer-reviewed	Oversight works only when the overseer has causal power, access to the relevant facts, self-control and fitting intentions	Interdisciplinary analysis, tested against Article 14 of the EU AI Act

That is why EDDI sets approval friction per endpoint and per tool, lets a gate time out to approve, reject, abort or wait, and detects a loop of identical pauses instead of asking the same person the same question again.

Multi-agent deliberation: diversity and the vote

EDDI's group conversations let several agents discuss, vote and decide. The research is more mixed than the enthusiasm around multi-agent debate suggests, and it shapes how the feature is built.

Study	Source	Reported result	Setting
Multiagent debate (2023)	arXiv preprint	Debate improved mathematical and strategic reasoning and reduced hallucinated facts	Several instances of the same black-box model proposing and critiquing answers over rounds
Debate or vote (2025)	arXiv preprint	Majority voting alone accounts for most of the gain usually attributed to debate	Seven NLP benchmarks, with a proof that debate by itself does not raise expected correctness
If debate is the answer, what is the question? (2025)	arXiv preprint	Debate methods often failed to beat single-agent baselines, but mixing different models consistently helped	Five debate methods, nine benchmarks, four models
Heterogeneous debate (2025)	J. King Saud Univ. Computer and Information Sciences peer-reviewed	4 to 6 points more accurate than standard debate, and over 30% fewer factual errors	Agents with distinct roles, dynamic routing and a learned consensus, across six benchmarks
Persuasion as an attack (2026)	Scientific Reports peer-reviewed	One persuasive adversarial agent cut group accuracy by 10 to 40% and raised agreement on wrong answers by over 30%	LLM-to-LLM debate; adding agents or rounds did not reliably help

Three design choices follow. Members can run on different models, since diversity is what consistently helps. Decisions go to a formal vote with the tally and the losing side's dissent recorded, since voting carries most of the gain. And ballots are cast independently of each other, so one confident, persuasive member cannot talk the room into its answer during the vote itself.

Conversation memory

Study	Source	Reported result	Setting
LongMemEval (2024)	arXiv preprint	Commercial assistants and long-context models lost about 30% accuracy recalling information across sustained interactions	500 questions embedded in long user-assistant histories
Recursive summarization (2023)	arXiv preprint	Recursively summarized memory produced more consistent responses in long conversations	Open and closed models; complements long-context and retrieval-based approaches
Mem0 (2025)	arXiv preprint	91% lower p95 latency and over 90% lower token cost than sending the full conversation	LOCOMO benchmark. The authors sell a memory product, so weigh the comparison accordingly

EDDI keeps the full conversation in storage and controls what the model sees: long-term properties that outlive the context window, token-aware windowing, rolling summaries with a recall tool to drill back in, and Dream consolidation of persistent memory in the background. We have not benchmarked this against LongMemEval or LOCOMO.

What we have not measured

Not yet published

EDDI has no published first-party benchmark numbers. Rather than fill this section with figures produced under conditions chosen to flatter them, here is the harness we intend to run and publish, fixed in advance so the methodology cannot be tuned to the result.

Dimension	Metric	Why this metric
Concurrent conversation throughput	Sustained conversations per second at fixed p95 latency, virtual threads versus a platform-thread pool, same build	The claim is about I/O-bound concurrency, so the measurement has to hold the LLM provider constant and vary only the threading model.
Memory at concurrency	Resident memory and heap pressure as concurrent conversations rise, to the point of failure	The literature says memory, not CPU, becomes the binding constraint. A throughput number without the memory curve hides where the wall is.
Model cascading cost	Measured spend per 1,000 conversations, cascade versus single frontier model, at matched answer-acceptance rate	Cost claims are only meaningful at a fixed quality bar. Measuring spend without holding acceptance constant measures nothing.
Cascade escalation rate	Share of turns escalated, and calibration error of the escalation signal	This is the number that predicts whether the saving transfers to another workload. It travels; a headline percentage does not.
Tool-call and RAG latency	p50/p95/p99 added latency for httpCall tools and for retrieval, separated from model latency	Orchestration overhead is what EDDI is responsible for. Model latency is not, and mixing them flatters the platform.
Audit ledger overhead	Throughput and latency delta with the HMAC-SHA256 audit ledger enabled versus disabled	Governance features have a cost. Publishing it is more useful than implying there is none.

Alongside the numbers, every run will publish the things that make a benchmark checkable:

Exact EDDI version, JVM version, and JVM flags.
Hardware, including instance type, core count, and memory.
Database engine and version, and whether it ran in or out of the cluster.
The workload generator and the query distribution, published so it can be replayed.
The baseline configuration, and why it is a fair comparison rather than a strawman.
Raw results, not only summary statistics.

Measure it yourself

The strongest argument for self-hosted, Apache 2.0 software in a benchmarks context is that you do not have to take our word for it. EDDI runs wherever Docker runs, so you can measure it on your hardware, with your workload, at your quality bar.

Given how strongly cascade savings depend on workload composition, a measurement you run is worth considerably more than one we publish. TheDocker Compose guide gets you to a working deployment in about fifteen minutes.

Frequently asked

Why does this page not contain EDDI benchmark numbers?

Because we have not run the harness yet, and publishing numbers we have not measured is how benchmark pages lose their credibility. What this page does publish is the evidence that the underlying mechanisms work, the conditions that determine how much they help, and the exact measurement plan, so that when the numbers land you can check them against a methodology that was fixed in advance.

How should EDDI's cascade cost figure be read?