AI Engineering Case Study

FACTR: an evidence-grounded AI verification engine

By Luca Viscomi

An end-to-end AI system engineered to transform long-form speech into structured claims, retrieve supporting evidence from curated knowledge bases, and produce traceable, evidence-grounded assessments.

What I built

FACTR is a multi-stage AI engineering pipeline that takes unstructured, long-form conversation — a recorded debate — and turns it into structured, retrievable, and assessable knowledge. Rather than asking an LLM to simply “decide what’s true,” FACTR separates the problem into specialised stages: transcription, claim extraction, semantic representation, vector retrieval, evidence matching, and structured verification — an explainable, measurable AI workflow instead of a single opaque prompt.

I designed and engineered the complete workflow, end to end: the ingestion and transcription integration, the claim-extraction prompting and JSON schema, the embedding and retrieval layer, the evidence-matching and verification logic, the evaluation methodology, the Streamlit interface, and the Docker packaging for reproducibility.

Engineering capabilities demonstrated

Architecture

The original project vision

Before the code, there was a design: how the pipeline was originally pitched, illustrated for a non-technical audience.

From the original project presentation materials — illustrative, not a literal spec. One example: this graphic describes “speaker labels” at the transcription stage; the verified architecture above reflects what’s actually shipped, where diarisation was scoped for a later iteration.

Engineering the pipeline

1. Long-form speech → structured transcript

Challenge. AI can’t reliably analyse a debate until spoken language has been transformed into structured, machine-readable content.

Engineering approach. FACTR integrates faster-whisper transcription directly into the pipeline, converting raw audio into timestamped, structured text ready for downstream processing.

Output. A structured transcript suitable for claim extraction.

Capability demonstrated. Speech AI integration · NLP preprocessing · pipeline engineering.

2. Unstructured text → atomic, checkable claims

Challenge. Long-form argument is full of compound, hedged, and context-dependent statements — not the kind of thing you can fact-check as a single blob.

Engineering approach. I designed a prompting and JSON-schema strategy that has an LLM (GPT-4.1 mini) extract individual, independently-assessable claims from transcript chunks, rather than reasoning over the whole transcript at once.

Output. A structured set of atomic claims, each ready to be independently retrieved against and verified.

Capability demonstrated. LLM prompt engineering · structured output design · working within model context limits.

3. Claims → retrieved, ranked evidence

Challenge. A claim is only as good as the evidence you can find for or against it — and “find” has to mean something more precise than keyword search across 162,000+ passages of scripture, hadith, tafsir, and commentary.

Engineering approach. Each claim is embedded with sentence-transformers into a 384-dimensional vector and matched against the knowledge base by cosine similarity, returning ranked, provenance-tagged evidence per tradition.

Output. Top-K evidence passages per claim, each carrying its source, tradition, and similarity score.

Capability demonstrated. RAG architecture · embeddings · semantic retrieval.

4. Evidence → structured, confidence-scored verdict

Challenge. Retrieval finds relevant evidence; it doesn’t tell you whether that evidence actually supports or contradicts the claim. Similarity is not entailment.

Engineering approach. Retrieved evidence is formatted into a verification prompt and passed to GPT-4.1 mini for LLM-based NLI-style reasoning, returning a structured verdict object — not free text.

Output. A per-tradition and overall verdict, with a confidence score and explanation, grounded in the specific evidence retrieved.

Capability demonstrated. Explainable AI design · structured LLM output · evidence-grounded reasoning.

Building the retrieval engine

FACTR’s retrieval layer is directly relevant to modern RAG engineering: claims are converted into vector representations and matched against curated, domain-specific source material with full provenance carried through to the final output.

Engineering beyond the prompt

The engineering value here sits in the system built around the LLM calls, not in the calls themselves. FACTR deliberately separates claim extraction, retrieval, evidence selection, and verification into distinct stages, each producing a typed, structured object rather than free-form text. Here’s the actual verdict schema a claim resolves to:

{
  "claim_id": "c_0043",
  "claim_text": "...",
  "side": "affirmative",
  "verdict_overall": "agreement",
  "verdict_islam": "agrees",
  "verdict_christian": "insufficient",
  "confidence": 0.88,
  "explanation": "...",
  "evidence_islam": ["..."],
  "evidence_christian": ["..."]
}

Every field above is a real field from the shipped VerificationRecord schema — nothing here is illustrative.

Evaluated, not just demonstrated

FACTR was built with quantitative evaluation as part of the engineering process, not bolted on afterward — measured across 109 claims using retrieval metrics, evidence-coverage analysis, and pilot feedback.

MSc Artificial Intelligence — Distinction

FACTR formed the dissertation component of my MSc Artificial Intelligence, awarded with Distinction (dissertation mark: 83%). The work combined applied NLP, information retrieval, semantic search, LLM integration, evaluation methodology, and responsible AI design into a functioning end-to-end system.

My engineering contribution

FACTR was designed as an end-to-end engineering project. I developed the architecture, integrated the AI components, implemented the retrieval workflow, constructed the evaluation pipeline, and built the user-facing system — from the first line of ingestion code through to the Streamlit interface a pilot user actually clicked through.

Selected code

Retrieval — semantic search over the knowledge base:

def kb_search(
    query: str,
    cfg: Optional[FactrConfig] = None,
    top_k: int = 5,
    tradition: Optional[str] = None,
) -> List[KBHit]:
    cfg = cfg or FactrConfig()
    _load_kb(cfg)
    q = _MODEL.encode(
        [query],
        convert_to_numpy=True,
        normalize_embeddings=True,
    )[0].astype("float32")
    # Cosine similarity via dot product (embeddings are normalized)
    scores = emb @ q
    top_idx = np.argsort(-scores)[: top_k]

Why this matters: this layer converts a claim into a semantic vector and returns ranked evidence from independently curated source material — the retrieval core of a working RAG system.

Evaluation — the actual Precision@k / Recall@k / mAP implementation used to measure retrieval quality:

def evaluate_query(q, k=K):
    df = search(q, k=k)
    preds = df["faiss_id"].astype(int).tolist()
    gold  = set(labels[(labels["query"]==q) & (labels["relevant"]==1)]["faiss_id"].astype(int))
    hits  = [1 if p in gold else 0 for p in preds]
    p = sum(hits) / float(k)
    r = (sum(hits) / float(len(gold))) if len(gold) > 0 else None
    ap_num = 0.0; tp = 0
    for i, h in enumerate(hits, start=1):
        if h:
            tp += 1
            ap_num += tp / float(i)
    ap = (ap_num / max(1, len(gold))) if len(gold) > 0 else None

Why this matters: proves the metrics in the results section above were actually computed from a working evaluation harness, not asserted.

Reproducibility — pinning the embedding model at Docker build time to eliminate a whole class of silent dimension-mismatch bugs:

ENV FACTR_EMBEDDING_MODEL="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
RUN python3 -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')"

Why this matters: a small detail that signals production instinct — pre-baking the model into the image avoids first-run latency and, more importantly, avoids the KB and query embeddings ever silently drifting out of sync.

Engineering decisions & lessons

Similarity ≠ entailment

Semantic similarity is excellent for candidate evidence retrieval, but a highly similar passage doesn’t automatically prove or disprove a claim. Engineering response: FACTR is architected to keep retrieval and verification as separate stages, so a retrieval failure and a verification disagreement are two distinguishable, debuggable things — not one opaque failure mode.

Model consolidation over per-stage specialisation

The original design specified different models per stage (a smaller model for extraction, a larger one for verification). In practice, the shipped pipeline consolidated both stages onto GPT-4.1 mini. Engineering response: a reasonable simplification for a research-scope system — and exactly the kind of per-stage cost/quality tuning I’d revisit first in a production pass.

Retrieval floor as a tunable, not yet a gate

The retrieval layer has a configured similarity floor and a UI control for it, but production hardening means actually enforcing it in the retrieval call, not just exposing it as a tunable. Engineering response: stated plainly here as the next concrete step, not glossed over.

Diarisation, scoped for v1

Speaker-level diarisation was part of the original architecture design. The v1 build prioritised getting the full claim-extraction-to-verification path working end to end over speaker attribution. Engineering response: a scoping decision, not a gap discovered too late — and the first thing on the list below.

FACTR 2026 — how I’d engineer it for production today

FACTR proved the underlying architecture and research approach. Since completing the original system, my engineering work has expanded substantially into agentic orchestration, cloud AI platforms, API engineering, observability, and production AI patterns. If I were taking FACTR into an enterprise production environment today, here’s how I’d evolve the architecture:

From FACTR to agentic AI

FACTR established practical foundations in orchestrated AI pipelines, model/tool composition, retrieval, structured state, evidence grounding, and evaluation. Those foundations now extend directly into the agentic and cloud AI platform work covered elsewhere in this series — LangGraph, Google ADK, Vertex AI, Azure AI Foundry, AWS Bedrock, and enterprise AI platform design. FACTR is where that thinking started; it isn’t where it stopped.

What FACTR demonstrates

✓ End-to-end AI system architecture ✓ Python AI engineering ✓ NLP and speech integration ✓ LLM integration ✓ RAG architecture ✓ Vector embeddings and semantic retrieval ✓ Evidence grounding and provenance ✓ Structured outputs ✓ AI evaluation methodology ✓ Explainability ✓ Responsible AI thinking ✓ Production architecture capability

Building AI systems that go beyond the demo

My focus is engineering AI systems that combine models, data, retrieval, orchestration, and production discipline into solutions that can be understood, evaluated, and operated — not just demoed once and shelved.

Explore Agentic AI Builds · Get in Touch

Build agents. Ship impact. Stay an AgentJunky.