The problem
A private-equity operations team runs a recurring exercise: take a spreadsheet of extracted contract terms — governing law, renewal windows, termination rights, pricing clauses, and dozens more — and confirm each value actually matches what the signed PDF says. Done by hand, an analyst opens a contract, hunts for the relevant clause, reads it against the spreadsheet cell, and marks it verified or wrong. Across hundreds of contracts and dozens of fields per contract, this is thousands of manual lookups per deal, slow and easy to get wrong.
Two things make it hard to automate naively. First, the spreadsheet is messy — the same field shows up a dozen ways ("NY", "New York", "New York State"), so a literal comparison fails before it starts. Second, contracts are long, unstructured PDFs — some scanned, some multi-column — so finding the clause that answers a given field is itself a retrieval problem. Throwing every cell-and-contract pair at an LLM would work, but it's slow, expensive, and non-deterministic at that volume.
This tool is a pipeline that solves both: it normalizes the spreadsheet into canonical values, then verifies each value against the contract corpus through a staged retrieval-and-adjudication funnel that only spends an LLM call when cheaper methods can't decide.
Stage 0 — Normalizing the input
Before anything can be matched, the raw spreadsheet has to be cleaned. The normalizer treats this as a layered process where each layer makes the next one cheaper — and the rule set gets smarter over time.
- Bootstrap — generate the rules. When a column has no rules file yet, the pipeline collects its unique values and sends them to a cheaper model whose only job is to author the regexes: patterns and keyword matches mapped to the column's canonical labels, with numeric range filters for bucketed values. The generated rules are written to a per-column rules file before any normalization runs, so the up-front model cost is paid once per new column shape — not once per value.
- Pass one — pattern rules. With a rules file in hand, each value is matched against its regexes and keyword rules (plus generic fallbacks for dates and currency). This pass is deterministic, cheap, and handles the bulk of values instantly.
- Pass two — LLM classification. Whatever pass one can't confidently resolve gets batched — the unique unmapped values plus the target canonical labels — and sent to an LLM, which returns a value-to-category mapping applied back across every matching row. (This pass also translates foreign-language values into the canonical set.)
- Learn and cache. Every mapping the LLM resolves gets written back into that column's rules file as a new pattern. The next run — even on a completely different file — catches those values in pass one, for free.
Every normalized cell is tagged with how it was resolved — regex, basic, or llm —
so QA can see at a glance how much of a dataset was handled deterministically versus
classified, and spot-check the LLM-touched rows first.
Live demo · fabricated sample data
Messy input → canonical output
| column | raw value | normalized | method |
|---|---|---|---|
| state | ny | — | … |
| state | California (CA) | — | … |
| state | TX - Lone Star State | — | … |
| state | FL / Florida Keys | — | … |
| contract_duration | 180 days | — | … |
| contract_duration | 2 yrs | — | … |
| contract_duration | 5-year contract | — | … |
| contract_duration | 24 months | — | … |
Hybrid resolution: fast pattern rules (regex / basic type parsing) run first and handle the majority of values for near-zero cost. Whatever is left unmapped is sent to an LLM along with the target canonical labels — its answer is applied to the dataset and cached back into the rule set so the same value never needs another LLM call.
For large jobs, rule generation can run through a batch API path (roughly half the cost, async) instead of live per-column calls. A separate rule-testing tool replays saved rules against real data and reports the regex / would-classify / unmapped breakdown without spending a single token — so rules can be tuned before a full run.
The matching funnel
With clean canonical values in hand, each one is verified against the contract corpus through three stages arranged as a funnel. The point of the ordering is cost: every stage is more expensive than the last, so a value is only escalated when the current stage can't decide it. Most values never reach the LLM.
- Stage 1 — heuristics. Fast fuzzy matching (token-set ratio) against the contract text, accelerated by a character-trigram index that pre-filters candidate documents. Score ≥ 0.92 → match; ≤ 0.75 → insufficient; anything in between escalates. Very short values skip straight to stage two, where they fare better.
- Stage 2 — hybrid retrieval. The value is embedded and searched two ways in parallel: dense vector similarity (FAISS, cosine over normalized embeddings) for semantic matches, and SQLite FTS5 full-text search for lexical / keyword matches. The two result sets are fused into a single ranked list. If the normalized value appears verbatim in the top-ranked chunk, that's a deterministic match with no LLM call. If nothing retrieves, it's insufficient. Otherwise the top chunks become context for stage three.
- Stage 3 — LLM adjudication. The retrieved evidence, the field, and the normalized value go to an LLM prompted to return strict JSON: a decision (match / mismatch / insufficient), a confidence score, a plain-English reason, and the exact evidence snippet with its source document and page range. The prompt is hashed and the full result cached, so re-running a deal never re-pays for a decision already made.
Because every decision carries its evidence and page citation, the output isn't just a verdict — it's an auditable trail an analyst can click through, which matters when the answer feeds a real deal.
Live demo · fabricated sample data
Four values, one trip down the funnel
governing_law = “New York”
queuedrenewal_notice_days = “60 days”
queuedtermination_for_convenience = “Not permitted”
queuedliability_cap = “$5,000,000”
queued
Getting contracts into the corpus
The retrieval stages are only as good as the text they search, so ingestion is its own subsystem. PDFs are parsed with PyMuPDF, with multi-column layouts detected and re-flowed into reading order. Scanned or image-only pages fall through a three-tier OCR chain — native text first, then an OCR pipeline, then a vision-model fallback with image preprocessing — so even a faxed contract becomes searchable. Parsed text is chunked with heading-aware logic that respects clause boundaries and tags each chunk by topic (governing law, term, termination, liability, and so on) so retrieval can filter by clause type. Everything is content-addressed by file hash and cached, so re-ingesting an unchanged corpus is a no-op.
The tech
The pipeline is Python on Polars for fast spreadsheet handling, FAISS and SQLite FTS5 for hybrid retrieval, and PyMuPDF for PDF parsing. LLM work — classification, adjudication, and embeddings alike — runs through a unified provider layer over the Anthropic and OpenAI SDKs, with prompt caching on the shared instruction context and a built-in cost tracker that logs every call's tokens and dollars by operation and model.
Storage and indexing sit behind pluggable backends. In development everything lives on the
local filesystem; the deployed version runs as a containerized serverless API on Azure,
with the corpus, indexes, and rules in blob storage, secrets in a managed vault, and
infrastructure defined in Terraform. Moving to that environment drove a couple of
deliberate stack choices: embeddings are API-based (OpenAI text-embedding-3-small)
rather than a self-hosted model, so there's no multi-gigabyte ML runtime or GPU to ship in
the image; and the OCR toolchain is baked into the container (Tesseract, Ghostscript)
rather than assumed present on the host. Because the FAISS + SQLite index is mutated on
local disk and synced back to blob storage under a per-workspace lease, the service runs as
a single warm instance rather than scaling out — a pragmatic trade for a low-volume,
batch-oriented internal tool.
The system is exposed three ways depending on who (or what) is driving:
- a React + TypeScript web app with real-time progress, live LLM-conversation tracking for transparency, cost analytics, and isolated profiles so each client/deal keeps its own corpus, indexes, and rules with no cross-contamination;
- an HTTP API (function endpoints with poll-for-status jobs on long-running ingest, normalize, and match operations) for programmatic and frontend use; and
- an MCP server so an AI agent can invoke ingestion, retrieval, and adjudication directly as tools.
There's also a prompt-recommender that analyzes a run's mismatches and proposes improved extraction prompts, closing the loop back toward the upstream step that produced the spreadsheet in the first place.
Why it matters
Manual contract verification doesn't scale, and a pure LLM-per-cell pipeline doesn't stay cheap or reproducible. This design gets both. The normalizer's rules handle routine cleanup for free and improve every run; the matching funnel reserves LLM adjudication for the genuinely ambiguous cases and caches every decision it makes. Over repeated runs the ratio of deterministic to LLM-based resolutions climbs — accuracy stays high, every verdict is backed by a citation, and cost trends toward zero for the data shapes and contracts the system has already seen.