A RAG agent does a lot of work before it answers a question. It searches documents, decides which passages matter, chooses whether to search again, and checks whether its answer matches the sources. Each step can involve a small judgment call.
I used to rely on language models for many of those calls. They worked, but even a cheap model could make the system slower and more expensive when I asked it to assess dozens of passages or citations. I’ve now integrated Jev, a decision model from TypeSafe, into my agentic RAG app for three jobs: model routing, steerable reranking, and citation checking.
The change isn’t about replacing the model that writes the final answer. It’s about giving the smaller decisions to a tool built for decisions. That distinction matters because an agent might make many such calls on its way from one question to one response.
What a Decision Model Does Differently
Most AI chat models produce an answer by writing one token at a time. That’s useful when I need an explanation, a summary, or a response to a question. It’s less useful when I need a simple judgment such as “Does this passage answer the query?”
Jev takes a different approach. I give it the situation to assess and define the decisions it can make. It returns a selected answer or score, along with information about how strongly it favors that answer. It doesn’t write a paragraph explaining itself.
That’s why I think of it as a System One decision model. I’m using that term for a fast, direct judgment rather than the extended written reasoning I’d ask of a chat model. A passage might go in, and a yes-or-no decision with a score comes back.
The output also has a defined structure. Jev won’t invent a fifth category if I’ve given it four citation categories. That doesn’t mean its judgment is guaranteed to be correct. It means my application receives an answer in the shape I specified, rather than having to interpret a free-form reply.
This becomes valuable when an application repeats the same kind of judgment at scale. RAG systems are full of those opportunities. They handle many document chunks, search results, claims, and conversation turns, each of which can need a quick assessment.
The Three Jobs I Gave Jev in My RAG App
My app lets an agent search a knowledge base and use the results to answer questions. I’ve placed Jev at three points where a decision affects what happens next.
- Model routing: Decide how much work a question requires before choosing the model that will handle it.
- Steerable reranking: Judge which retrieved document chunks actually help answer the search query.
- Citation checking: Assess whether a cited source passage supports a claim in the generated answer.
These aren’t interchangeable tasks. Routing happens near the start of a request. Reranking happens after search. Citation checking happens once there’s an answer to inspect. Together, they show how many decisions can sit around the model that eventually writes the response.
Route simple questions without always using the strongest model
My router looks at a question and judges how much work a correct, complete answer would require. A greeting belongs in a different category from a request that needs document search. A question calling for deep analysis and careful synthesis needs a different level of attention again.
Previously, I used the stronger model for everything. That meant a simple “hello” could take the same route as a question about a legal brief in the knowledge base. Jev lets me classify the request first, then choose a model that fits the job.
I give the router criteria in plain English. They distinguish simple acknowledgments and small talk from questions requiring documents or tools, and from requests involving several steps of analysis. Jev scores the available options, and my application uses that decision to route the request.
The important point isn’t that every simple request must use a particular model. It’s that the routing decision no longer needs the strongest model to make it. A small decision can help avoid a much larger expense downstream.
Filter search results before the agent reads them
When the agent searches the knowledge base, it can retrieve a broad set of candidate passages. In the example I built, it retrieves 60 chunks. Jev assesses those chunks against the search query, and I pass the most relevant ones onward.
That stops weak matches from filling the agent’s context. A passage can share words with the question without containing an answer. If I send every search result to the agent, I make it spend time sorting useful evidence from noise. I also give irrelevant text a chance to influence the final response.
Check whether citations support the claims beside them
A citation can point to a real document and still fail to support the sentence attached to it. I’ve built a citation checker that compares generated claims with their cited passages. Jev can classify a passage as fully supporting a claim, partially supporting it, saying nothing relevant, or contradicting it.
In the app, I can inspect the claim alongside its source passage and see the checker’s assessment. One example showed an 82% score for a supported claim, while another showed 57% for a partially supported claim. These are useful signals, especially when a response contains several citations that would otherwise take time to check individually.
I previously handled this job with a cheap language model. Checking many cited paragraphs that way was slow, and the cost added up. A decision model gives me a more practical way to run those checks as part of the app’s normal workflow. It still needs sensible thresholds and occasional human inspection, particularly where the distinction between full and partial support matters.
Why These Small Calls Affect Speed and Cost
Jev’s pricing, as shown during my build, was just over four US cents per million input tokens, with no charge for its short output. Since it returns decisions rather than written explanations, most of what I’m paying for is the material I send in.
The calls in my app give a useful sense of scale:
- Model routing used just over 400 input tokens for the example request.
- Reranking assessed 60 chunks containing almost 30,000 input tokens. The estimated cost was around one tenth of a US cent.
- Citation checking used around 11,000 input tokens, also costing a fraction of a cent in that example.
Those are measurements from the calls I inspected, not a promise that every application will have the same costs or timings. Document size, the number of candidates, and the decisions requested will all affect usage. The broader lesson is that a RAG system can make many small judgments, and using a writing model for each one may be an expensive way to get a label or score.
Speed matters for the same reason. A single extra check might seem minor. Repeat it across dozens of chunks, multiple citations, and every request, and it becomes part of the experience of using the app.
Where Jev Fits in an Agentic RAG Pipeline
RAG stands for retrieval augmented generation. Rather than relying solely on what a model already knows, a RAG app retrieves information from a knowledge base and uses it to help produce an answer. That lets an agent work with the documents available to the organization.
The process starts with ingestion. I import documents and divide them into smaller pieces called chunks. Chunking matters because a question usually needs a particular section, not an entire document. Loading whole documents into an agent’s context can consume space without improving the answer.
Next comes retrieval. The agent searches the indexed knowledge base for chunks related to the question. It might search again using different terms, locate a file, or assign a sub-agent to examine a full document. Its next action depends on what it finds.
That ability to work through a loop is what makes this app agentic. The agent doesn’t have to follow one fixed search and then answer. It can change its path as it gathers evidence.
There are decisions throughout both sides of this pipeline. At ingestion time, I might want to identify a table of contents or sensitive passage before indexing it. At retrieval time, I need to decide which search results deserve space in the agent’s context. A decision model fits particularly well where the answer is a category, rating, or yes-or-no judgment rather than a written explanation.
What the Langfuse Trace Revealed
It’s easy to describe an agent as if it simply receives a question, searches once, and responds. The actual path through my app is more interesting. I use Langfuse to inspect traces of the calls made along the way.
For one example, I added a legal brief to the knowledge base and asked the agent to tell me about it. The trace showed an initial document search, a pattern match against available files, another document search, and then a sub-agent assigned to analyze and summarize the document.
Each step created another chance to make a small decision. The trace also let me inspect the Jev calls rather than treating reranking and routing as invisible features. I could see the question supplied to the router, the instructions used to judge its difficulty, the chunks passed to the reranker, and the scores returned.
This visibility matters during development. If the agent gives a poor answer, I want to know whether search retrieved the wrong passages, reranking discarded a useful one, or the agent had good evidence but used it badly. A trace helps me separate those problems instead of guessing.
Inside the routing call
The routing call includes the user’s question and a defined set of criteria. For the legal brief request, the criteria ask Jev to distinguish between simple conversation, work requiring the user’s documents or tools, and deeper multi-step analysis.
Jev returns scores for those options. My application can then make a routing choice based on its output. The model isn’t being asked to write a plan for researching the brief. It’s assessing the amount and type of work the request appears to require.
Inside the reranking call
The reranking call includes the agent’s search query and the 60 chunks returned by search. For each candidate, Jev assesses whether the passage helps answer the query. The output gives me scores I can use to sort and filter the results.
I can inspect those scores in Langfuse, which is especially helpful when a document looks related but doesn’t contain the requested information. It lets me see whether the reranker recognized that difference.
Inside the citation-checking call
For citation checking, I provide the generated claim and its cited passage. One question asks Jev to choose the type of support the passage provides. The options include full support, partial support, no relevant support, and contradiction.
That’s a more useful result than a single “good citation” label. Partial support tells me something different from contradiction. It may mean the passage establishes part of a sentence but leaves another part unproven. My app can use those categories and scores to decide which citations need closer attention.
How I Ask a Decision Model a Question
The easiest way to picture a Jev request is to imagine giving an assessor a situation and a form to fill out. The situation might be an email, a passage from a document, or a piece of conversation. Jev calls that situation the state.
Then I define what belongs on the form. Jev supports three useful kinds of questions:
- Null: A yes-or-no assessment. For example, “Does this passage help answer the query?”
- Choice: One selection from a list I provide, such as the four citation-support categories.
- Score: A rating against levels I define, such as how much work a question requires.
I can put several questions on the same form and have them answered in one call. For each one, I describe the criteria in plain English. In a yes-or-no reranking check, for instance, “yes” means the passage directly helps answer the query or states the requested fact. “No” includes passages on a related subject that don’t answer it.
Behind the scenes, the request and response use a defined JSON structure. That structure restricts the output to the options I’ve supplied. It doesn’t remove the possibility of a bad judgment, but it does remove the need to parse a paragraph and hope the model used one of my expected labels.
The quality of my criteria still matters. If I define “relevant” too loosely, a table of contents might pass a reranking check because it names the right topic. If I clearly ask whether a passage answers the question, I’ve given the model a better decision to make.
Why Search Similarity Isn’t Enough
Search engines are good at finding text that resembles a query. Depending on the search method, a result might share exact terms, partial terms, or a similar meaning. None of that proves it answers the question.
Imagine asking how much notice a lease requires before termination. A table of contents might list “Termination and notice,” making it look like a strong match. But the heading doesn’t tell me the notice period. I need the actual clause.
That’s the difference between similarity and relevance to the answer. Search helps me find candidates. A second assessment asks whether each candidate provides useful evidence.
Without that second step, the agent may receive a mixture of useful passages, vague matches, and outright noise. It then has to spend part of its context and effort sorting them out. Worse, it may base an answer on material that sounds related but doesn’t establish the fact being asked for.
Reranking puts candidate results back in order based on how well they serve the query. In my app, the process has two stages:
- Search wide and fast. Retrieve a broad pool of candidate chunks. My example starts with 60.
- Assess the candidates again. Compare each chunk with the question, score whether it helps answer it, and pass the strongest results to the agent.
I can set a threshold in the application, too. If a chunk falls below that threshold, I drop it rather than send it onward. The agent could retrieve 60 candidates yet receive only 10 passages worth reading. The threshold is an application choice, not a universal number that works for every knowledge base.
What Makes the Reranking Steerable
A conventional reranker can compare a query with passages and produce relevance scores. Jev gives me another useful control: I can state how I want the results judged.
For the legal brief example, I supplied an instruction to prioritize documents explicitly titled or described as a legal brief. That instruction influences the scores alongside the search query and the candidate chunks.
There’s an important distinction here. The words used to retrieve candidates don’t have to be identical to the criteria used to rank them. An agent may search broadly to avoid missing a document, then apply more specific judgment to decide which results deserve attention.
Because the agent supplies the reranking instruction, it can adapt that judgment to the task at hand. If it needs a legal brief, explicit identification as a legal brief matters. For a different question, another criterion might matter more. I still need to define the call carefully, but I’m no longer limited to a fixed notion of similarity.
This is where I see a practical advantage for agentic retrieval. Search casts the net. The decision model helps decide what the catch actually contains. The agent gets a cleaner set of passages before choosing whether to search again, inspect a document, or answer.
Other Decisions I’d Consider Across the Pipeline
Model routing, reranking, and citation checking are the three jobs I’ve integrated into this app. They aren’t the only places a decision model could be useful. Once I look at RAG as a series of small assessments, I can see opportunities before retrieval, during an agent’s work, and after an answer is produced.
Classify document chunks during ingestion
As documents are imported and chunked, I could assess each chunk before it enters the index. A quick set of questions might identify whether it contains a footer, a table of contents, or a signature block. Other decisions could label its language, document type, or topic.
Those labels could become metadata I use later to filter search results. A table of contents, for example, might help an agent locate a section while being a poor passage to cite as evidence. Identifying it early gives the retrieval pipeline more information to work with.
I could also ask whether a chunk contains personally identifiable or other sensitive information. Another check might identify an older passage that a newer document has superseded. These are possible additions to the pipeline, not features I’m claiming to have implemented in this demonstration.
Check entities and relationships in Graph RAG
Graph RAG introduces another set of decisions. A language model can extract entities and relationships from documents. A decision model could then help check whether two extracted names refer to the same entity, classify a relationship from a defined list, or assess whether a passage supports that relationship.
Consider two references to the same person that use slightly different names. I need a judgment before merging their records. Likewise, an extracted relationship is less useful if the source passage doesn’t establish it. TypeSafe’s documentation includes a cookbook on knowledge graph entity alignment for this kind of pair-by-pair assessment.
These checks can repeat hundreds of times across a collection. That repetition is precisely where the cost and speed of each individual decision start to matter.
Keep an agent’s working context lean
An agent can accumulate a lot of material over several turns. Some earlier information remains useful; some doesn’t. I could ask a decision model whether a piece of context is still relevant and remove what the agent no longer needs.
I wouldn’t treat that as a reason to delete context indiscriminately. The question needs clear criteria, and the application needs to account for the risk of discarding important information. Still, it’s another example of a recurring yes-or-no judgment that doesn’t necessarily require a written essay.
Grade conversations after they happen
Langfuse traces also create an opportunity to assess completed conversations. I could review whether retrieved passages were relevant, whether an answer stayed grounded in its sources, and whether it cited those sources appropriately.
Without a cheap way to run those checks, I might inspect only a sample of conversations because there’s too much activity to assess individually. A decision model makes broader coverage more feasible. I’d still inspect the results and refine the criteria, but I could use those assessments to find patterns across many threads instead of relying on a handful of examples.
That’s the pattern behind my switch to Jev. I’m keeping language models for the work that calls for writing and synthesis. For the many smaller judgments around that work, I can specify the decision I need, inspect its score, and use it to guide the next step of the RAG pipeline.
