I was researching how search changes the answers an AI assistant gives. An assistant answering from company documents has two jobs: find relevant information, then decide whether that information actually answers the question. I wanted to understand which search methods help, and what rules an assistant can learn for when to search again, answer, or say the evidence is insufficient.

Consider a customer asking how long they have to request a refund. A search might return a document about warranty repairs. Both concern the same product, and both may contain a number of days. The assistant needs to recognize that a warranty deadline does not establish a refund deadline.

In June 2026, I attempted to let AI design an entire small experiment around that problem. AI coding agents implemented and ran it under my direction. We compared a simple search-and-answer process with a process that could keep checking the evidence before deciding what to return.[1][2]

The simple process scored zero out of eight. Yet when we inspected the model's actual responses, it had correctly recognized the lack of evidence in every case. The surrounding software had replaced those responses with wrong answers before grading them.

That discovery changed what the run could tell us. It exposed defects in how we produced and measured answers, and gave us a concrete way to inspect an AI-designed experiment: follow a few answers all the way from the documents to the final score.

What we wanted to learn

Our broader research involved two approaches to finding passages. Lexical search ranks text using word matches and their weights. Vector search represents queries and passages as learned numerical patterns, called embeddings, and ranks them by similarity. For example, a question about a “refund deadline” might need a passage using the phrase “return period.” Research such as Dense Passage Retrieval compares lexical and learned dense retrieval approaches.[3]

Finding a relevant passage is only part of the task. The assistant also needs a policy: rules for deciding what to do next. It might search again, read another passage, check a possible answer, answer the question, or decline to answer. Learning those decisions is another research direction; Self-RAG is one published example involving retrieval and reflection.[4]

This particular experiment focused on a narrower question: could a checking policy stop without giving an answer when the documents did not support one? An earlier synthetic test had appeared to improve results on answerable questions, which motivated the follow-up. That earlier verifier also had access to accepted answers, a limitation that matters when interpreting its apparent improvement.[2]

The June follow-up used only questions whose answers were absent from the supplied documents. This made it a diagnostic test of whether the system would withhold unsupported answers. Evaluating a useful assistant more broadly also requires checking whether it answers questions that the documents do support, a balance examined in published work on unanswerable retrieval questions.[5]

How the comparison worked

We used eight fictional questions and a collection of sixteen documents. Each question was tested once under each of two rule-based policies, giving sixteen recorded runs in total. The documents included tempting but inapplicable information, as well as explicit notes that the requested information was missing.[6]

We deliberately kept this exploratory run small to look for a basic signal before deciding whether to invest in a larger evaluation.[13] Starting small lets you catch problems early. Here, the failures we found show why those first cases should check the experiment itself as well as the system being evaluated.

The two policies took different routes:

  • The simple policy searched once. Before calling the model, its software extracted a possible answer from the search results. It then asked the model to answer from the collected evidence.

  • The checking policy used software rules to inspect possible answers and decide whether to continue or stop. Its verifier, the component that checked those answers, could consult the test's answer key.[8]

Both could use hybrid search, which combined lexical and vector results. The checking policy also had additional tools for searching by words, vectors, and text patterns. Because the tools and decision rules changed together, this run compared two complete processes. Isolating the effect of vector search would require holding the other parts fixed.[2]

The policies faced the same questions and configured resource limits, but used those resources differently. The simple policy made one model call per question, eight calls altogether. The checking policy made no model-generation calls on these cases; its eight outcomes came from rules.[6]

The saved results showed zero correct answers out of eight for the simple policy and eight out of eight for the checking policy. Those scores described what each process submitted to the grader. To understand the result, we had to inspect how those submissions were produced.

How a correct refusal became a wrong answer

One question asked for the fictional Nocturne product's purchase-refund window. Search found a warranty-service window of 31 days. The text explicitly said this was not the refund policy.

The model responded:

“The corpus does not support a Nocturne purchase-refund window; the only Nocturne-related evidence is a warranty-service window, which explicitly is not the refund policy.”

Here, “corpus” means the supplied document collection. The model had correctly distinguished a warranty deadline from a refund deadline. But the answer saved for grading was “31 days.”[7]

The substitution happened in the parser, the software that selected the final answer. It gave the earlier extracted candidate priority over the model's response. The wrong candidate survived; the model's correct refusal did not reach the grader. The same priority rule caused all eight substitutions.

There was a second defect in the grading contract. The grader recognized a refusal only when the submitted answer was the standalone word ABSTAIN, after normalizing capitalization and punctuation. The model's instructions did not require that word. Even if the software had preserved the natural-language refusals, this grading rule would still have marked them wrong. That is a conclusion from inspecting the code. The failures recorded in the run came from the candidate substitutions.[7]

These are separate repairs. The answer-selection code needs to preserve the intended final response, and the response instructions need to agree with what the grader accepts. Fixing either one alone would leave the other problem in place.

Why the perfect score also needed inspection

The checking policy's verifier could read the answer key. For every question in this test, the accepted answer was ABSTAIN. The verifier used that information to reject misleading candidates. When the policy reached its limit on the number of actions it could take, the software managing those steps returned ABSTAIN, which the grader counted as correct.[8]

That behavior explains the perfect score under these test conditions. To find out how well checking would work in deployment, the policy would need to make its decisions without seeing grading answers. Answer keys belong in the evaluator, or in diagnostic components whose access is explicitly part of the test.

The question set imposed another limit. Because all eight questions were unanswerable, a rule that refused every question would also have scored eight out of eight. A test containing both supported and unsupported questions could measure the tradeoff: how often the system avoids an unsupported answer, and how often it unnecessarily withholds a useful one.

The eight cases also used four repeated patterns, and the documents were unusually explicit about missing evidence. The observed correct model responses therefore establish a narrow result for this small synthetic test set. Broader reliability requires a wider evaluation. Work on sufficient context distinguishes whether retrieved material supports an answer from whether a model uses that material correctly. This case shows why the software that handles the model's response also needs inspection.[10]

What made the failed comparison useful

The study retained the questions, documents, settings, costs, and raw records of each run. Those records made it possible to reconstruct the defect without another paid run. Preserving outputs and examining small batches qualitatively are among the practices recommended by Biderman and colleagues for language-model evaluation.[9]

An automated integrity audit had checked counts and saved results. It did not check whether the scored answer agreed with the model's actual response.[11] The missing check mattered: a set of consistently saved results can still measure the wrong thing.

The diagnosis points to specific requirements for a future comparison:

  • Test the full path from response to score with known examples. Include a wrong preliminary answer followed by a correct refusal, so answer selection and grading can be checked together.

  • Keep accepted answers outside the policy being evaluated. Let the evaluator use them to grade the result.

  • Include both answerable and unanswerable questions, so refusing everything cannot look like a successful assistant.

  • Change one research variable at a time. To test vector search, hold the other components fixed and compare with and without it. To test a learned policy, keep its learning cases separate from its evaluation cases.

For a rerun of the document-answering task, I’d expect a newer flagship model at maximum reasoning to outperform the model used here. A corrected comparison of both model configurations on a mixed set of answerable and unanswerable questions would test that expectation. The tested model already refused all eight questions correctly, so this run does not demonstrate an answering advantage for a newer model. A rerun must also correct the response-selection and grading defects and remove the checking policy’s answer-key access.[14]

These are proposed changes that still need implementation and evaluation. The available record documents one attempt to let AI design an experiment; responsibility for proposing and approving individual rules remains unresolved.[1]

The run's useful outcome was the ability to locate the failures: a parser that overrode correct responses, a refusal format that the prompt never requested, and a checking policy that could see the answer key. Before using a benchmark to choose a model, search method, or policy, inspect a few complete traces. Check what each component could see, what it produced, and exactly which output the score describes.

Appendix: What We Ran and What the Scores Mean

The June 13 study records and the author's accounts are private. Readers can't inspect those sources through this article, while the published research linked below gives context for judging the design.

What we ran

On June 13, 2026, we tested eight invented questions against sixteen documents. None had a supported answer. Each of the two policies, sets of software rules for searching, checking evidence and deciding what to return, ran once per question. The questions used four repeated patterns.[6]

Sixteen runs in total. That gave us eight question-by-question comparisons between the policies, with saved responses and step-by-step records for each run.[6]

We kept the run small to look for early problems before investing in a larger evaluation. The record doesn't explain the exact choice of eight cases.[13]

The simple policy called OpenAI's gpt-5.4-mini once per question at low reasoning. The saved settings and responses identify the requested configuration, but don't independently confirm the exact backend version. The models that designed, implemented or reviewed the experiment remain unidentified.[1][6]

The checking policy's eight answers came from software rules, with no model calls.[6]

What we counted Simple policy Checking policy
Runs 8 8
Model calls 8 0
Answers the grading software accepted 0 of 8 8 of 8

Each policy could take five information-gathering actions per question. Estimated model cost was about one cent, all from the simple policy's eight calls. Human work and other system costs were excluded. The policies used different tools and actions, so the shared limits didn't make their work or spending equal.[6]

Why the scores were misleading

All eight model responses correctly refused to give an unsupported answer. The software changed them. It submitted a possible answer extracted before the model call instead, so the grading software received eight wrong answers. In the refund example, the model rejected a warranty deadline as evidence of a refund deadline, but the submitted answer was "31 days."[7]

The grading software accepted a refusal only as the single word ABSTAIN, ignoring capitalization and punctuation, although the model's instructions never asked for that word. Answer substitution caused the recorded failures. Reading the code showed that preserving the model's original refusals would still have left them failing the grading rule.[7]

The checking policy could see the answer key. Every accepted answer was ABSTAIN. Its code could also read the lists of supporting evidence used for grading, but those lists were empty in all eight cases. After rejecting misleading candidates and reaching its action limit, the policy returned ABSTAIN and received full credit.[8]

A note saying the requested information was absent appeared among the first two search results in every case, making the missing evidence unusually obvious. An earlier test on answerable questions also gave the checking code answer-key access. That apparent improvement remains unvalidated.[2][8]

What the result can support

Every question was unanswerable. Refusing everything would score eight out of eight, so a wider test needs supported questions too, to measure useful answers alongside refusals. The repeated patterns limit how widely this result applies. Peng and colleagues evaluate this balance using six categories of unanswerable questions, a framework our small test didn't reproduce.[5]

Because the policies changed their tools and decision rules together, this run can't separate the effect of vector search from those other changes.[2]

The saved scores yielded a two-sided sign-test p-value of 0.008, comparing which policy won on each question. Because those scores came from a flawed measurement process, they cannot establish broader model reliability or a deployment advantage for the checking policy.[6]

The saved records let us find the defects without another paid run. An automated audit missed the substitutions. It checked counts, costs, statistics and saved files without comparing model responses with submitted answers, and the checked records don't establish a successful paid rerun fixing these defects.[6][7][11]

The private files contain the exact prompts, search settings, answer-extraction rules and question patterns needed to reproduce the experiment.[6]

The research sources below predate the run. The June planning notes explicitly connect Peng and colleagues' paper with this follow-up. We use the other sources here to review the design after the fact, without claiming that the study followed their recommendations.[2]

Biderman and colleagues connect evaluation quality to scoring rules and the code that applies them. They also recommend reading sample outputs. Our saved records made this review possible, but don't establish that anyone read a sample before the run or make the study publicly reproducible.[9]

Sufficient-context research separates evidence supporting an answer from a model's use of it, while our records show a further failure in the software handling the response.[10]

NIST's voluntary guidance calls for documented evaluation methods that are checked for validity under conditions relevant to deployment. It also calls for independent expertise. Human review of the experiment's methods before this run remains unestablished. Answer-key access and the narrow sample prevent a claim about the checking policy's deployment performance without that access. These are review judgments using the guidance, and don't certify the study or establish that a redesigned policy would fail.[12]

Sources

[1] Private author accounts, October 7-8, 2026. They describe the research goal, the attempt to let AI design the experiment, and work by AI coding agents under the author's direction. They don't identify every model or establish who proposed or approved each rule.

[2] Private study protocol, June 13 planning notes and author-review notes. They distinguish the wider research plans from this run and record the earlier checking code's answer-key access.

[3] Karpukhin et al., Dense Passage Retrieval, November 2020, abstract. Background on word-based and learned search. It doesn't establish an effect in this study. Read source

[4] Asai et al., Self-RAG, October 17, 2023, version 1, abstract. Related retrieval and reflection research. We didn't use or train its method. Read source

[5] Peng et al., Unanswerability Evaluation for Retrieval Augmented Generation, July 2025, abstract and evaluation framework. We didn't replicate its evaluation. Read source

[6] Private June 13, 2026 run notes, settings, sixteen results, eight questions and saved model responses. They support the recorded setup, calls, costs and scores. They don't independently identify an exact backend model version.

[7] Private model responses, submitted answers and answer-selection and grading code. They show all eight substitutions and the refusal-format mismatch. The quoted refusal in the article is a recorded response. "31 days" is the submitted answer.

[8] Private checking and stopping-rule code, questions and run records. Every accepted answer was ABSTAIN. The grading-evidence lists were empty.

[9] Biderman et al., Lessons from the Trenches on Reproducible Evaluation of Language Models, May 23, 2024, version 1, sections 2.1, 2.3.1 and 3. This guidance was available before the run. Read source

[10] Joren et al., Sufficient Context, 2024, revised 2025, abstract. It separates evidence supporting an answer from model behavior. It doesn't validate this experiment. Read source

[11] Private June 13, 2026 integrity-audit record. The audit checked saved files and figures without comparing model responses with submitted answers.

[12] NIST, AI Risk Management Framework 1.0, January 2023, printed pages 29-30. Relevant guidance is MEASURE 1.3, 2.1, 2.3, 2.5 and 2.13. Read source

[13] Private author clarification, October 10, 2026. It explains why the run started small. The exact choice of eight cases remains open.

[14] Private author clarification, October 10, 2026. He expects a newer flagship answering model at maximum reasoning to perform better. No such comparison has been run in the checked record, and the expectation doesn't establish a repair of the software or evaluation defects.