Kicaulah/rag-faithfulness-evaluator

RAG Faithfulness & Hallucination Evaluator (v1.0.0)

A production-grade, lightweight, and calibrated evaluator for verifying whether an LLM/RAG generated answer is factually grounded in retrieved passages.

Key Distinctions

Unlike crude binary classifiers or uncalibrated prompt judges, this system explicitly distinguishes:

1. Contextual Faithfulness (strict grounding in supplied passages) vs. Real-world Factual Correctness (world truth). An answer can be true in reality but unfaithful to context.
2. Entailment vs. Direct Contradiction vs. Unsupported Extrapolation.
3. Atomic Claim Breakdown: Decomposes answers into testable propositions with exact evidence span attribution.
4. Adversarial Failure Modes: Stress-tested against numerical perturbations, date swaps, entity substitution, negation flips, and missing qualifiers.

Benchmark Results (Test Split)

Evaluator Model	Accuracy	Macro F1	Weighted F1	MAE	Spearman ρ	Latency (ms)
Lexical Baseline	0.395	0.267	0.253	0.364	0.250	0.3
TF-IDF Baseline	0.158	0.061	0.068	0.298	0.242	6.0
LLM Judge Baseline	0.447	0.409	0.413	0.302	0.394	0.1
Hybrid Cross-Encoder (Ours)	0.553	0.595	0.459	0.271	0.644	0.5

Quickstart

from src.inference.pipeline import FaithfulnessPipeline

pipeline = FaithfulnessPipeline()
result = pipeline.evaluate(
    context="Alphabet Inc. reported Q4 2023 consolidated revenues of $86.31 billion, up 13% year-over-year.",
    question="What were Alphabet's Q4 2023 revenues?",
    answer="Alphabet's Q4 2023 consolidated revenues reached $86.31 billion, representing a 13% increase."
)

print("Faithfulness Score:", result.overall_faithfulness_score)
print("Category:", result.predicted_category)
print("Grounded:", result.is_contextually_faithful)
for claim in result.claim_evaluations:
    print(f" - [{claim.status}] {claim.claim_text} (Evidence: {claim.evidence_span})")

Intended Use & Scope

• In-Scope: Automated CI/CD guardrail for RAG applications, LLM evaluation pipelines, synthetic data filtering, hallucination monitoring.
• Out-of-Scope: Legal liability determination without human review; medical clinical decisions without licensed clinician oversight.