How cheap can a reliable citation model be?

While building a retrieval harness for agents, we ran into a practical question: once a tool returns the right evidence, can the model cite it faithfully?

For Mempipe, a citation does not stop at the document or even the page. It can point to an individual block of content within a document. The tool returns the complete citation_url; the model only needs to carry it into the answer intact and format it as a valid Markdown link.

That sounds simple. It also sounds like something a model should understand when told to use a “standard Markdown link.” Our benchmark suggests otherwise.

When Gemini 3.8 Flash was released, we added it to the citation benchmark we had previously run against Gemini 3.7 Flash. We used the exact same prompt to keep the comparison fair.

The result surprised us: Gemini 3.8 performed considerably worse than Gemini 3.7. That made us wonder whether the problem was not the model, but the prompt.

Our original prompt asked for a “standard Markdown link.” We ran all nine models with a second prompt that showed the syntax and one worked example, then repeated the original prompt across all nine.

The short version:

• Among the nine models tested, DeepSeek V4 Pro 0813 was the cheapest to pass the syntax-plus-example round. At prices recorded on September 3, 2026, it cost $0.43 per million input tokens and $0.87 per million output tokens, about 4.8 times DeepSeek Flash.
• With the original prompt, Gemini 3.8 covered 247 of 305 citation opportunities. Gemini 3.7 had covered 293.
• Showing the syntax and an example raised Gemini 3.8 to 305 of 305.
• The syntax-plus-example prompt performed best in aggregate. Luna got worse, while DeepSeek Flash and Qwen produced mixed results across the three rounds.
• We repeated the original prompt to check whether the results held. That run performed better than the first, but the syntax-plus-example prompt still produced the strongest aggregate result.

The unchanged rerun mostly preserved the ranking, but it changed which models cleared the zero-violation bar. Qwen joined Terra in the passing set, showing that one run can still change a strict pass-or-fail qualification.

Prompt engineering still matters, even for standard Markdown. Showing the expected syntax and an example produced the strongest aggregate result. Even when retrieval returns the right evidence, citation handling can still fail elsewhere in the pipeline. Reliability therefore belongs to the exact model, prompt, and surrounding citation pipeline, not to the model alone.

The two prompts

Both prompts asked the model to copy a complete source address returned by the tool. The second condition changed three things: it asked for inline citations, supplied the [label](citation_url) syntax template, and included a filled example. This benchmark compares those two prompt conditions as a whole; it does not isolate the example as the sole cause.

The mention-only prompt:

Cite supported claims with a standard Markdown link whose destination is a returned citation_url. Copy that URL exactly. Do not alter it, add ./ or ../, construct another citation URL from logical_path or node_id, or cite a result without citation_url.

The syntax-plus-example prompt:

Cite supported claims inline as [label](citation_url), for example: "Transformers drop recurrence entirely [paper p.3](papers/attention.pdf?cite=n_abc)." The destination must be a returned citation_url. Copy that URL exactly. Do not alter it, add ./ or ../, construct another citation URL from logical_path or node_id, or cite a result without citation_url.

“Standard Markdown link” felt precise to us. In practice, models often returned bare bracketed URLs instead. The syntax-plus-example prompt made the expected shape explicit. This comparison cannot isolate whether the syntax template, the word “inline,” or the worked example drove the difference.

What counted as a pass

A model needed:

• at least 300 of the 305 required citation opportunities covered
• at least 95% required-citation coverage
• zero fabricated or mutated parsed-link destinations
• zero malformed citations
• zero citations on uncitable evidence
• deterministic disclosure when evidence was stale

Those rules are deliberately stricter than a reader merely recognizing what the model meant. GLM’s syntax-plus-example run covered every required citation opportunity with a parsed exact link but used different freshness wording once, so it did not pass the full contract.

What changed

Each model had 305 required citation opportunities and produced 45 answers in each round. Every result cell shows covered opportunities · fully clean answers. Coverage counts each required opportunity at most once; duplicate exact links do not increase it. “Fully clean” means every required citation in an answer was present, parsed as a link, copied exactly, and accompanied by the required freshness disclosure.

Coverage alone does not show how a model failed. Here is the complete violation breakdown for all three rounds. “Fabricated or mutated” applies only to destinations that parsed as Markdown links; malformed strings were counted separately without an allowlist check.

The syntax-plus-example prompt outperformed both mention-only rounds in aggregate. Terra was the sole passer in mention-only round one. Terra and Qwen passed round two. With the syntax-plus-example prompt, DeepSeek Pro, Gemini 3.8, and Terra passed.

Gemini 3.8 produced the clearest same-model result: its two mention-only rounds covered 247 and 269 of 305 required opportunities, while the syntax-plus-example round covered all 305. GLM moved from 253 and 274 under mention-only to 305 with the other prompt, although one answer still failed the separate freshness-disclosure rule.

The failure mix changed too. Mention-only round one produced 49 omissions, 9 fabricated or mutated parsed-link destinations, 185 malformed citation-shaped strings, and 2 freshness failures. The repeat produced 34 omissions, no fabricated or mutated parsed-link destinations, 165 malformed strings, and 1 freshness failure. The syntax-plus-example round produced no omissions, no fabricated or mutated parsed-link destinations, 132 malformed strings, and 1 freshness failure.

What the violations mean

The categories represent different product failures, so we do not collapse them into one error rate.

Omitted

The tool returned a citable source, but the model did not cite it. The claim remains unsupported, but the model does not send the reader to false evidence. In mention-only round one, Terra omitted 4 required citations, Luna omitted 12, and GLM omitted 33. In the repeat, GLM omitted 31 and Ling omitted 3. The syntax-plus-example round produced no omissions.

Malformed

The model returned a citation-shaped string that was not a valid Markdown link. The most common malformed form was a bare bracketed address such as:

[fact.pdf?cite=n_123]

instead of:

[source](fact.pdf?cite=n_123)

The first form renders as plain text. Mempipe cannot turn it into a citation chip, include it in a Sources list, or rewrite it for portable export. Malformed Markdown remained the dominant failure under both prompts. The scorer counts these strings but does not allowlist-check their embedded destinations.

Fabricated or mutated

The destination of a parsed Markdown link did not exactly match a citation_url returned by the tool. It may have gained ./, changed its encoding, altered its path, or changed its node ID. That is a corrupted citation even when it looks plausible. GLM produced nine such destinations in mention-only round one. Neither the repeat nor the syntax-plus-example round produced any parsed-link violations in this category.

Freshness

When a result was marked stale, the benchmark required the exact sentence Evidence freshness: stale. A different phrase or missing punctuation failed this rule so the scorer could remain deterministic. DeepSeek Pro missed it in both mention-only rounds. GLM missed it in mention-only round one and the syntax-plus-example round. These are strict contract failures, but they are not equivalent in severity to a mutated source address.

Citation on uncitable evidence

Some frozen results intentionally lacked a citation_url. Citing one would mean that the model constructed an address from other fields. No model did that under either prompt.

Luna is the important exception

The result is not “examples always make models better.” Luna moved in the opposite direction.

In the first mention-only round, Luna covered 293 of 305 citation opportunities and its 12 failures were omissions. In the repeat, it covered 299 and produced 6 malformed citations instead. Its failure type moved, but both rounds were better than its syntax-plus-example result.

With the syntax-plus-example prompt, Luna covered only 269 of 305. In four of five repetitions of the hardest encoded-path case, the scorer found nine malformed citation-shaped strings instead of valid links. The changed prompt traded a smaller number of omissions for a larger, highly repeated formatting failure.

To check whether that regression held, we ran Luna through the syntax-plus-example prompt once more. It covered 281 of 305 opportunities and produced 42 of 45 fully clean answers. The regression repeated, but less severely: 24 malformed citations instead of 36. The same bare-bracket failure appeared in two of five encoded-path repetitions and one root-and-nested-path answer. This supplemental run is not included in the three-round tables or aggregates above.

DeepSeek V4 Flash and Qwen were mixed. DeepSeek Flash tied the syntax-plus-example result at 296 in the first mention-only round, then rose to 301 in the repeat. Qwen moved from 290 to 305 across the mention-only rounds, compared with 296 under the syntax-plus-example prompt.

These exceptions matter because they rule out a universal prompting claim. The syntax-plus-example condition was high leverage for some models, but its effect was model-specific. The model and the exact prompt must be qualified together.

What we tested

The benchmark isolates citation transport from retrieval quality. Every model received the same frozen tool results. The model did not construct a path or node ID; it copied a complete citation_url exactly.

We ran:

• 9 pinned models through Vercel AI Gateway
• 2 prompt conditions, with the mention-only condition repeated
• temperature 0
• 9 cases per repetition
• 5 repetitions per round
• 45 answers and 305 required citations per model and round
• 1,215 model calls across the three nine-model rounds reported here

The corpus included prose, lists, tables, duplicate filenames, nested paths, Unicode, spaces, unmatched parentheses, long paths, stale evidence, uncitable results, distracting URLs, and paths that would be easy for a model to reproduce almost correctly.

The scorer accepted a citation only when it parsed as a real Markdown link and its destination exactly matched a URL returned by the tool. Code examples and escaped Markdown did not count. Browser rendering and source navigation remained separate release checks.

This is a citation transport benchmark. It does not test retrieval recall, tool selection, factual correctness, or whether every claim deserves its cited source.

What this changes for us

The prompt is part of the model interface, not incidental wording around it. A model recommendation is incomplete unless it pins the model, provider route, exact prompt, scorer, and repetition count.

For the syntax-plus-example prompt, DeepSeek Pro, Gemini 3.8, and Terra passed this run. The other six models remain best-effort under the benchmark’s strict requirements.

That roughly 4.8× price gap leaves us with a product decision: switch to the qualified model, or make the cheaper default explicitly best-effort.

The broader writing lesson is practical but narrow: if output must conform to a specific shape, test a prompt that shows that shape. Naming a standard may be technically sufficient for a human reader and still leave meaningful ambiguity for a model. Here, the syntax-plus-example prompt outperformed both mention-only rounds in aggregate.

But Luna is the guardrail against turning that lesson into folklore. Re-run the benchmark after a prompt change, for every model you intend to support. Do not assume an instruction that helps one model will help all of them.

Limitations

This was a narrow, point-in-time benchmark.

• It tested citation copying and formatting, not research quality.
• Model prices were recorded on September 3, 2026, and may change.
• Tool results were frozen, so it did not test whether models choose the right tools.
• It scored frozen output, not browser rendering or source navigation.
• Exact model behavior can vary by provider route even behind one gateway.
• Temperature 0 reduces variation but does not make hosted inference deterministic.
• Mention-only round one used the August 31 results for eight models and a September 3 Gemini 3.8 run. The all-nine mention-only repeat and the syntax-plus-example round were both collected on September 3. This provides a same-day comparison, but it is still sequential rather than randomized.
• Five repetitions expose repeated failure modes, but they do not establish a permanent pass rate.

We saw that run-to-run caveat directly. The two mention-only rounds changed the passing set from Terra alone to Terra and Qwen. An earlier syntax-plus-example run also produced 305/305 for DeepSeek Flash and 298/305 for Terra; the reported syntax-plus-example round reversed those outcomes.

What comes next

If you are shipping grounded answers, test citation transport separately from retrieval. Pin the model, provider route, prompt, scorer, and sample count. Reliability belongs to that exact system, not to the model name alone.