# How do you choose the evidence an AI answer receives?

By Sean Everett · October 8, 2026

Inspect document preparation, candidate retrieval, re-ranking and answer support separately, then test the full evidence path against the questions the product must answer.

Topics: Retrieval · AI evaluation · Knowledge products

## Preserve the information needed to interpret a passage

Start with representative questions and identify the source material needed to answer each one. Inspect what survives parsing: headings, table labels, qualifications and links between sections. A sentence can look relevant while losing the condition that makes it applicable.

Choose passage boundaries by testing those dependencies. For a hypothetical equipment guide, a maintenance instruction may need its model identifier and an exception on the next page. Compare candidate chunking approaches against that complete information need, within the context and response-time limits of the product.

Keep source identity and version attached to the prepared passages. Include questions whose answers are absent, superseded or spread across documents. Those cases reveal whether the evidence preparation preserves distinctions the later model needs, before prompt changes obscure the underlying problem.

## Check what each selection stage contributes

Sentence Transformers documents a retrieve-and-re-rank pipeline: an efficient first stage finds candidates, then a cross-encoder scores query-passage pairs. Evaluate the handoff. Check whether the initial set contains the required evidence before judging whether re-ranking places it high enough to be used.

Use relevance judgments tied to the actual question. Include passages about the right topic that answer a different question, apply to another version or omit a necessary exception. Treat automated labels as proposed judgments for review, especially near these boundaries.

The original BEIR benchmark compared retrieval methods across varied tasks and domains, finding meaningful differences in generalization and computational cost. Its lesson for selection is to retain a credible baseline and test on your own workload. A strong result on another collection leaves that comparison open.

## Test the answer with the selected context

Liu and colleagues’ 2024 Lost in the Middle study found that answer performance could change when relevant information moved within a long context. The finding concerns the tested models and tasks. Use it to motivate checks of context length and ordering in the configuration you intend to release.

Compare a few bounded context policies while holding the questions and scoring criteria stable. Inspect missing evidence, irrelevant passages, repeated material and the time required to produce the answer. Adding more passages should earn its place through better completed answers under those constraints.

Finally, inspect whether each consequential claim follows from its cited passage, including qualifications and disagreements. An answer can cite a real document while overstating it. Keep evidence-coverage failures, selection errors and unsupported conclusions separate so the next improvement addresses the stage that failed.

## Related reading

- [Sentence Transformers: Retrieve and re-rank](https://www.sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html)
- [BEIR: Cross-domain retrieval evaluation](https://arxiv.org/abs/2104.08663)
- [Liu and colleagues: Long-context evaluation](https://aclanthology.org/2024.tacl-1.9/)
- [Designing a useful knowledge assistant](https://evergences.com/insights/knowledge-assistant-work-context/)
- [Related owned product: Jetpack Intel — custom internal AI workflows](https://jetpackintel.com/)

Source: https://evergences.com/insights/retrieval-evidence-selection/