Put the evidence boundary on the server.
A closer look at organization scope, retrieved passages, and citation validation in AgentOps AI.
Problem
A model can produce a plausible answer containing a document reference that was never retrieved. Giving the model access to a search tool does not, by itself, make its final citations trustworthy.
Context
AgentOps AI combines conversation history, tool calling, and organization-scoped document retrieval. This implementation note focuses on its server-side answer boundary. It does not claim a measured retrieval accuracy or a complete enterprise product.
Approach
Treat retrieved passages as the evidence available to a particular run. Let the model propose an answer and document identifiers, then validate those identifiers on the server. Supply displayed excerpts from stored tool output rather than accepting new quotations written by the model.
Implementation
The FastAPI service derives organization scope from the signed-in user. A knowledge-search tool retrieves chunks from that organization's documents in PostgreSQL with pgvector.
After knowledge search, the final response is expected to contain an answer and document identifiers. The server checks that every cited identifier belongs to the search result. It copies supporting excerpts from those retrieved chunks into the response.
An empty citation list after knowledge search produces an explicit insufficient-information answer. Other tools have different contracts: a calculator produces an arithmetic result, while analytics uses a closed set of parameterized operations. Neither needs invented document citations.
The agent loop is bounded. Runs and tool calls are persisted so an administrator can inspect outcomes, including failures. This is a useful inspection trail; it is not evidence that every generated answer is correct.
Results
The repository documents mocked model tests and separate API, service, agent, and tool checks. The answer path restricts citation membership and controls excerpt provenance. It does not prove that the cited passage supports every sentence of the answer, or that retrieval found the best passage.
There are no answer-quality scores or production success rates in this note.
Lessons learned
- Retrieval scope, citation membership, and semantic correctness are different checks.
- Organization boundaries belong in server-side data access, not in model instructions alone.
- An explicit refusal is more useful than presenting an answer without the evidence its contract requires.
- Evaluation should include missing evidence, contradictory documents, cross-organization requests, and a model that proposes an unknown document identifier.
Sources
Continue with the AgentOps AI case study.