Checking AI Docs for Hallucinations Without an LLM Judge

Documentation written by a language model has a particular failure. A hallucination in AI-generated documentation is a reference that does not match the source: a function that does not exist, a file path that was never in the repository, or a citation to a line that says something else. The prose around it reads well, and a reader who trusts the guide pays for that later.

The common answer is to ask a second model to grade the first. We chose not to do that in AELSHA™ NEHEMVIX, our free tool that turns a Git repository into guides for beginners, senior engineers and architects (you bring your own AI model key, and your provider bills model use). This note explains what we check instead, how we test the checker itself, and what it still cannot catch.

Check in code whatever code can check

Many claims in a guide about code are not matters of opinion. A file either exists at a path or it does not. A function either is defined in the repository or it is not. A citation either resolves to the line that defines a name or it does not. Those questions can be answered deterministically, in milliseconds and at no cost, and the answer cannot be argued with by the model being checked.

Nehemvix handles invented content in five layers, each catching what the one before lets through:

1. Prevent. The instructions tell the model to mention only files, functions and behaviour present in the material, and to say when it is unsure.
2. Constrain. Facts that code can extract, such as the file reference, dependency inventory and diagram structure, are built in code and never left to the model. Repository text reaches the model marked as untrusted data, and secrets are redacted first.
3. Resolve. Citations are resolved in code against an index of the repository's symbols. Line numbers from the model are never trusted, and a citation that cannot be resolved stays plain text. It never becomes a precise-looking link.
4. Detect. Every path and code name in the narrative is looked up in the repository. When a section contains names that cannot be found, the model gets one chance to repair it, and the repair is kept only if it has fewer unverified names.
5. Disclose. Whatever remains is listed at the end of the guide by name, together with a score that states what it measures.

A score that says what it measures

Every guide carries a grounding score from 0 to 100, shown to the reader with its parts:

The label matters as much as the number. The score measures structure and grounding, not factual correctness, and the guide says so. A score that implied more would be a claim the checker cannot support.

Test the checker, not only the model

A checker that flags nothing has perfect precision and no value. So the checker is tested the way any safety control should be. The results below are Aelsha Innovations' own evaluation, not an independent audit, and they are published on the Nehemvix site in How we check every guide:

• Golden guides. Twelve real guides across five repositories and four languages, with facts verified by hand against the source. All twelve reproduce their original score exactly, with no false alarms.
• Planted mistakes. Invented names are planted into real guides and must be reported and lower the score.
• Blind rounds. In the first round, a separate tester who had not read the checker wrote a list of plausible invented names. The checker caught 80.2% of them on the first run, and 100 of 106 (94.3%) after two rounds of fixes made against that list, so the first list no longer counts as held out. A second, harder round was written by a new agent that was not allowed to read the checker, its documentation or the first round's misses. Across all five repositories, 189 of 227 were caught (83.3%), and none of 376 real names was flagged as invented. That second round is the held-out figure.
• Prompt injection. Test repositories containing instructions aimed at AI tools, such as a hidden comment asking every section to end with a fixed sentence, were documented by a real model. In our live runs the model obeyed none of them, and the guide now lists that text for the reader.

The figure to rely on is the second round's 83.3%: it was measured once, on a list nobody had tuned the checker against, while the first list's 94.3% came after fixes made against that list. A list written blind by a new author measures what the next model's mistakes will meet, not what the checker was tuned against.

What it cannot catch

Stating the limits is part of the design:

• A false claim made only of real names. "This class encrypts every payload" uses a real class name, so the score does not move. Only hand-verified facts, or a careful reader, catch this.
• Names that are not marked as code are not checked.
• In languages other than Python, names are found by pattern rather than by parsing, so an invented name that happens to appear elsewhere in the repository passes.
• Claims about behaviour, such as order of operations or default values, are not verified.

These limits are pinned by tests, so the list cannot quietly become untrue.

What carries over to other AI systems

The same approach applies well beyond documentation:

• Check in code before reaching for a judge. Anything with a deterministic answer should get one. It is cheaper, faster and reproducible.
• Never launder uncertainty. An unverified claim should look unverified. Turning it into a confident link or number hides the risk from the person who carries it.
• Test the guard with data it has not seen. Precision on familiar cases says little about recall on new ones.
• Publish the limits next to the results. People trust a system more when they know where to look harder.

If you want to see the checks on your own code, AELSHA™ NEHEMVIX is free to use with your own AI model key; your provider bills model use. If you are building agents that must be measured the same way, our AELSHA™ ASTRA framework covers evaluation and observability across the agent lifecycle.

About this research

• Question: Can a checker written in code, with no second AI model as judge, catch invented function and file names in AI-written documentation?
• Method: Names marked as code in a guide are checked against the repository. The checker was tested on hand-verified guides, planted mistakes, two blind rounds of invented names, and test repositories containing prompt injection.
• Data: Twelve hand-verified guides across five repositories and four languages; blind rounds of 106 and 227 invented names; 376 real names.
• Results: All twelve guides reproduce their original score with no false alarms. First list: 80.2% caught on the first run, 100 of 106 (94.3%) after two rounds of fixes against it. Held-out second round: 189 of 227 (83.3%) caught. None of the 376 real names flagged.
• Limits: The four listed under "What it cannot catch" above. They are pinned by tests.
• Reproducing it: The full evaluation is published at nehemvix.aelsha.com/evals. The test harness itself is not yet public.
• Who and when: Aelsha Innovations' own evaluation, not an independent audit. Author: Gladys Esmeralda. Published 28 October 2026. Corrected 12 November 2026: the first list's 94.3% was measured after two rounds of fixes against it; the first run caught 80.2%.

Further reading

• How we check every guide, AELSHA™ NEHEMVIX: The full evaluation behind the figures above, including the golden guides, both blind rounds and the prompt-injection tests.
• How to catch hallucinated function and file names in AI-generated documentation, AELSHA™ NEHEMVIX: The score weights and the four limits above, worked through on real guides.
• Build an eval gate for an LLM application, AELSHA™ NEHEMVIX: How the golden guides and blind rounds become a release gate, with who wrote each test set.

The checker and its test sets change as Nehemvix develops, so check the evaluation page for the current results before you rely on these numbers.