Building an LLM-as-judge pipeline for semantic matching
How a four-stage pipeline — intent extraction, embeddings, vector search, and an LLM judge — turned a fuzzy 'who should see this' problem into something testable.
Matching people on intent — who wants to grab dinner tonight, who's up for a specific kind of night out — sounds like a recommendation problem until you actually sit down to build it. Rule-based tagging is cheap but brittle; sending every candidate through a full LLM call is accurate but slow and expensive at scale. The system I ended up with splits the difference across four stages: intent extraction, embedding, vector search, and a final LLM-as-judge pass.
Cutting the LLM's workload before it ever sees a prompt
Before anything reaches a model, candidate pairs go through a deterministic pre-scoring layer built on a taxonomy of 2,922 terms grouped into 172 clusters. Overlap between two users' terms and clusters produces a cheap similarity score with zero inference cost. That score alone is enough to discard obviously-bad candidates before they ever reach the expensive part of the pipeline.
The next stage swapped HuggingFace embeddings for OpenAI's text-embedding-3-small — a change driven by wanting tighter semantic clustering for short, casual text (the kind people write in a bio or an activity note, not a paragraph), and it paid off in the vector search step that follows.
Making the judge accountable
The last stage is an LLM-as-judge call that scores the surviving candidates and explains why. The part that actually made this reliable wasn't the first prompt — it was the iteration loop. I built a 101-case evaluation suite covering the edge cases that kept breaking in production (near-misses, sarcasm in bios, conflicting signals), and ran every prompt change against it before shipping. That eval suite is the reason the judge prompt went through more than 35 versions: each one was a measurable change against a fixed yardstick, not a guess.
- Deterministic taxonomy pre-scoring filters out the easy no's for free
- Embeddings only need to be 'good enough' to feed a vector search, not perfect on their own
- An eval suite turns prompt engineering from guesswork into something you can actually iterate on
None of the four stages is doing anything exotic on its own — the value is in how cheaply the first two stages narrow the field before the expensive judge call ever runs.