How to verify an LLM API serves the real model

“Which model are you?” is the first thing everyone sends and the last thing to trust. The answer is text the model was trained or prompted to produce: a relay can set it in a system prompt, a distilled model repeats its teacher's name, and most models decline to be specific about versions. Treat a wrong answer as a flag and a right answer as nothing.

Pick three facts dated inside the claimed model's training window and one from after it. A model from the right family and generation knows the three and declines or guesses on the fourth. An older substitute misses the later of the three; a newer one answers the fourth. Vary the facts between runs so they cannot be prompted around.

Build a prompt at about 80% of the context window on the model's page, bury one made-up fact deep inside it, and ask for the fact. A substitute with a shorter window, or a relay that silently cuts the prompt, cannot answer. The cost is the prompt itself at the input rate, so run it once per id, not daily.

A fixed prompt that mixes scripts, a date, arithmetic and a code fragment, sent at temperature 0, with one word asked for in reply. The one answer means little; the diff between today's and yesterday's means a lot, because nothing in the prompt changed. The canary the daily check uses today:

Paste it against any key and keep the answer per day:

Every model family tokenizes differently. Count the canary with the developer's own tool, then read usage.input_tokens in the gateway's answer. A difference of a few tokens is normal: Anthropic's token-counting reference (read 2026-10-31) calls its count an estimate, says the endpoint is free to use, and notes that models from Claude Opus 4.7 on tokenize about 30 percent higher than earlier ones, so count against the id you are testing. A count off by half or double is a different tokenizer, which is a different model family. OpenAI's tiktoken (read 2026-10-31) is “a fast BPE tokeniser for use with OpenAI's models”; encoding_for_model picks the encoding for an id.

Keep the canary answers. A provider that changes the route behind an id changes the answer to a prompt that did not change, and the day it happened is in your files. Pair that with the receipt: a signed record per request naming the model and the billed tokens. It does not inspect weights, but it binds the provider to a name, and the signature can be checked on the Verify page.

Once a day every model in the catalog gets the canary above at temperature 0, and the answer is compared with the earlier runs. Three verdicts: stable, the answer matched; changed, it moved, so what serves the id is not what served it before; not reached, no route answered the latest run, and the last verdict is history. The feed is public JSON at https://api.relayswitch.io/v1/proof, no key, cached five minutes; the Proof page renders one row per model with the time of the run.

A changed verdict is a routing change to go and look at. The status page carries the hourly probes per route beside it.