Introducing jev-reranker: Reranking and Relevance Filtering for RAG
Retrieval often finds passages that match a query without providing anything useful for the answer. Sending all of them to an LLM means more input to process, more tokens to pay for, and potentially more distracting material for the model to work through.
Filtering also gives the application a decision to make before generation. If no document looks useful, it can stop there or try another search query. In that setting, deciding what to leave out can matter more than getting every document into the right order. On NanoHotpotQA, the relevance filter removed about 92% of the candidate documents while reaching an nDCG@10 of 0.975. I will get to the setup and results below.
The figure below combines the NanoHotpotQA performance comparison with an example of how the relevance filter decides which documents to keep.
The charts compare hybrid search, reranking, and relevance filtering by nDCG@10 and the average number of documents retained per query. The example below them illustrates the filtering step: two passages answer the online-return question and are kept, while three passages about other policies or another shop fall below the 0.2 threshold and are removed.
Getting started with relevance filtering
Install the library:
uv add jev-reranker
Then pass a query and the documents you retrieved:
from jev_reranker import JevReranker
query = "How long do I have to return an online order to ACME Shop?"
documents = [
"ACME Shop accepts online returns within 30 days of delivery.",
"For ACME Shop online orders, submit your return request within 30 days of receiving the item.",
"ACME Shop in-store purchases can be returned within 14 days of purchase.",
"ACME Shop products come with a one-year repair warranty.",
"FooBar Shop accepts online returns within 60 days of delivery.",
]
reranker = JevReranker(api_key="YOUR-TYPESAFE-API-KEY...")
response = reranker.relevance_rerank(query, documents, threshold=0.2)
for item in response["results"]:
print(f"{item['score']:.2f} {item['text']}")
Example output:
0.98 ACME Shop accepts online returns within 30 days of delivery.
0.97 For ACME Shop online orders, submit your return request within 30 days of receiving the item.
The last three passages look superficially relevant, but they concern an in-store purchase, a repair warranty, and a different shop. None answers the question about returning an online order to ACME Shop. Their scores fall below the threshold, so they are left out.
What changes between relevance filtering and reranking?
The part I find interesting about Jev is that the scoring behavior changes through the instructions.
Ask how relevant a passage is to the query, and the scores can be used to sort search results. Ask for very low scores when a passage contributes little to answering the question, and those scores become useful for filtering retrieved context before it reaches an LLM.
The model stays the same. The instructions describe the judgment you need. That flexibility is a large part of what interests me about Jev.
Results on NanoHotpotQA
I evaluated the library on NanoHotpotQA, a compact retrieval dataset sampled from the multi-hop question-answering dataset HotpotQA.
The candidate data is built using hybrid retrieval: BM25 and dense retrieval with harrier-oss-v1-270m, taking the top 100 results. The dataset construction can append a positive as a 101st candidate when none appears in the top 100. Reranking quality is measured with nDCG@10.
This run used all 50 queries in NanoBEIR-en's NanoHotpotQA split and the full candidate pools, with 100 documents per query. Both Jev methods used listwise scoring:
| Method | Threshold | nDCG@10 | Documents retained per query, mean | Documents removed |
|---|---|---|---|---|
| Hybrid search | — | 0.833 | 100 | 0% |
rerank |
0.0 | 0.969 | 100 | 0% |
relevance_rerank |
0.2 | 0.975 | 7.62 | 92.38% |
The evaluation script guide describes the method and how to reproduce it.
The relevance filter retained all 97 labeled positive documents present in the candidate pools. Three other positives were already missing before reranking. The relevance scores produced the same nDCG@10 before and after threshold filtering, so the smaller retained set came without a loss on that metric in this run.
For a RAG pipeline that would otherwise pass the full candidate pool to an LLM, this substantially reduces the number of documents to include while preserving the labeled evidence available in the pool. My expectation is that giving a smaller LLM less distracting material could also help it avoid hallucinations; that is an expectation about downstream use, rather than something this retrieval evaluation measures.
The threshold is configurable: a document is retained when score >= threshold. For example, threshold=0.05 retains more documents, while threshold=0.5 filters more aggressively. Evaluate the cutoff on your own data, especially where missing evidence would be costly. The instructions can also be changed to describe what counts as useful evidence in your own application.
These results came from jev-1.13.0. A model update may change the scores and the behavior of a chosen threshold.
In my local tests, I have also seen reranking quality comparable to bge-reranker-v2-m3, a CrossEncoder built specifically for reranking. If you already use it, the evaluation script gives you a way to compare them on your own setup.
Why use a library around the Jev API?
Calling Jev directly is enough to implement the basic idea. I built jev-reranker to include the practical pieces that come up when using it in a retrieval pipeline.
Listwise and pointwise scoring
There are two main ways to arrange the input:
- Listwise: put the query and the candidate documents into one context, roughly
[query, doc1, doc2, ...]. - Pointwise: send the query with each document separately, roughly
[[query, doc1], [query, doc2], ...].
The library supports both. In the comparisons I tried, listwise was generally faster and gave better results, so it is the default. Other tasks may favor pointwise.
Long inputs need some care, particularly when listwise scoring puts many documents into one request. Jev's 32k budget for the state plus the longest question is relevant here. The library uses character counts or token counts to split requests into groups that fit its configured budgets.
A public Jev tokenizer would make it easier to check the exact input length in advance. As far as I have found, there is not yet a tokenizer or API for doing that exact preflight count, so the library works with estimates.
Concurrency and retries
Sending every request one at a time can be slow. The library supports bounded concurrency, with a configurable number of requests in flight. It also handles retries and waits when transient failures or rate limits occur.
That is the reason for the wrapper: the reranking and relevance instructions are included, together with input splitting, concurrency controls, and error handling. If those are pieces you would otherwise build around the API yourself, I hope the library saves you some work.
What interests me about Jev
There is a useful space between asking an LLM to make every decision and training a separate model for each task. An LLM may be slower than the application needs. A specialized model takes data, training effort, and people who can build and maintain it. Jev's approach to structured decisions fits into that space.
The ability to steer the judgment through instructions is particularly appealing for retrieval. What should rank first depends on the context: a passage that is broadly related to a topic may still be useless for the answer the application needs to produce. Being able to express that distinction in the request is useful.
The reranking and relevance-filtering results are encouraging to me. More broadly, I would like to see more models explore this direction: general models for structured decisions that are practical to place inside an application. Jev makes that direction feel concrete, and I am looking forward to seeing where it goes.
