Search looks simple. Until it becomes the core of the product.
When search is "just" a text box at the top of the page, plenty of off-the-shelf answers will do the job. But in products where search is how users actually find things, rather than browsing categories or menus, it stops being just another component. It becomes the central nervous system.
That's usually the point where the conversation changes. It's no longer, "How do we implement search?" It's, "Which decisions (about search) will we still be able to defend a year from now, but accommodate the budget, the timeline and the constraints we have today?"
Honestly, we could spend days unpacking each of those decisions. Because every decision here carries a rationale that rarely lives in the code alone.
But before we get into the concrete case we want to tell you about, let us pause on one point. To us, it's the most important thing in this whole piece:
There's no recipe. There's no silver bullet.
What you actually have is a toolbox and a body of knowledge, and you build the solution according to the need at hand: the user's, the project's, the moment's.
We'll come back to this idea more than once.
The board before the first commit
Before a single line of code, a whole board of constraints is already on the table.
Sometimes the client has already picked their cloud provider, and that alone rules out some of the possible architectures.
There's budget, which decides whether you're reaching for a frontier-class language model (one of the largest and most capable ones) or a leaner, cost-optimised alternative that gets the job done just as well.
There's data protection and data residency, which sometimes force you to process information in a specific region, even when the "ideal" region is technically somewhere else.
There's how much latency the user will tolerate, which changes everything depending on whether they'll accept one second or ten.
There's the nature of the problem you're solving: finding a house isn't finding a pair of trainers, which isn't finding a clause in a contract.
And there's the reality of the skills your team has today, not the ones you wish you had available.
Each of these pulls the architecture in a different direction, and the job is balancing that equation, no single variable decides anything on its own. But together, they close off the space of possible solutions before you've even opened your editor.
It was within that board of constraints that we arrived, incrementally, at a solution for a property search system.
That's the context behind the architecture we're about to look at.
The obvious technology we didn't use
If we said "AI applied to search", what's the first thing that would come to mind?
We'd bet embeddings. Vector search, semantic matching, the promise of understanding any phrase through similarity.
Funnily enough, we don't use them. ¯\_(ツ)_/¯
We'll admit this surprises quite a few people when we explain it. At a time when "vector search" has become nearly synonymous with modern search, it sounds like a step backwards.
In our case, it wasn't.
Custom fine-tuning was off the table, too. Fine-tuning is brilliant, but it's easily overkill here. It brings a lot of moving-parts, dataset curation, training pipelines, and long-term maintenance overhead, that just didn't make sense for what we needed. There was no point taking on that extra complexity when off-the-shelf models do intent extraction so well with a tight schema, clear validation, and a few good examples in the prompt.
A "traditional" search engine, such as Elasticsearch in our case, is still excellent for a huge range of use cases. Fast. Predictable. Cheap to run. And really good at anything that's a structured filter: price range, number of bedrooms, location, property type.
Coming later doesn't make a technology the right answer. It's just another tool in the box. Not the tool.
So where does AI come in, if the search engine already handles the structured part so well?
It comes in exactly where the search engine can't help on its own: understanding what the user meant when they wrote it the way people actually think and speak.
What about Elasticsearch's native AI?
Yep, Elasticsearch has its own built-in AI search capabilities, like ELSER. Why not just use those?
In short, it came down to what we actually needed to solve:
- Intent extraction vs. Semantic retrieval: ELSER is great at finding documents by meaning, for example, "somewhere quiet". But we needed exact, structured data extraction ("4 bedrooms, under $500k") validated against a strict schema — something ELSER isn't built to output. It's a retrieval model: what comes out is a set of weighted terms for ranking, not a JSON object we could hand to the next layer.
- Cost: Running a retrieval model as part of the search tier puts the cost in capacity you provision, rather than in calls you actually make. And every change to the model means re-embedding the whole catalogue. Our on-demand model plus caching means we pay per call, with no index to rebuild when the model changes.
- Determinism: Approximate vector search gives you a ranked similarity score, and by design it doesn't guarantee the true nearest neighbours. That's exactly the right trade-off when you're ranking by relevance. It just isn't the rigid, auditable output our business logic needed.
It's also fair to say Elasticsearch's AI story isn't only ELSER any more. You can point an inference endpoint at an external language model, and there's even an ES|QL COMPLETION command to call one from inside a query. But that moves where the model call happens without changing anything that actually mattered to us. We'd still need the schema contract, the canonicalisation and the gate, because none of that comes for free just because the call now lives inside the search engine.
ELSER is great if we need to match vague meanings, but we just needed to turn user text into clean, strict filters. It's a great tool, just not for what we were building.
The heart of the system: extracting intent, not searching
The user types something like "4-bed farmhouse near a certain area, under a certain price, with land".
That's human language: full of nuance, synonyms, and implied context. The search engine doesn't interpret it as a set of filters, actually it sees it as loose text. The language model bridges that gap by extracting the user's intent and returning a structured, canonical JSON with the fields the search engine already knows how to interpret.
On paper, it sounds simple enough. Except this is exactly where the most interesting problem in the whole system shows up. And it's the one we really want to unpack with you now.
The paradox: asking for determinism from something that isn't deterministic
A language model, by nature, isn't deterministic. Ask it the same thing twice and the answers can differ, different formats, a field too many or a field too few. The business, on the other hand, expects exactly the opposite, a stable, canonical JSON that the next layer can turn into a deterministic query against the search engine.
How do you reconcile a probabilistic component sitting in the middle of a path that needs to behave as if it were deterministic?
The answer is less glamorous than the question, it comes down to engineering discipline.
The layers that guarantee determinism
This isn't a "chat freely and we'll try to make sense of it afterwards" situation. It's a proper schema contract, checked at every step before it's trusted:
The four layers, in order:
- Structured output — the model responds within a schema contract, with explicit fields and categories
- Canonicalisation — every field is checked against closed, controlled vocabularies (property type, style, land type, energy-efficiency band, and so on), so if a field doesn't match anything on that list, it simply doesn't get through
- Grounding and cross-field checks — every field is held back against the user's actual words, so a status nobody mentioned, or an exclusion with no "except" or "outside" behind it, gets dropped rather than trusted, along with the handful of cases that only make sense in relation to each other (a place both searched for and excluded, a price range whose two ends have collapsed into one)
- Schema gate — either what survived is canonical and still adds up to a query we can run, or the system degrades to plain full-text search.
There's no middle ground where a value we couldn't verify slips through.
Each layer has a different job. Some guarantee consistency. Others guarantee security. Only once an intent has passed through all of them does the system accept it as valid.
Controlled vocabularies as a security boundary
Restricting model outputs to a closed set of known values isn't just about data validation, it's a core security defense. Have a look at the OWASP Top 10 for LLM Applications and you'll see entire categories of risk, like prompt injection or output that escapes its expected format and turns into unwanted behaviour further down the line, lose a good chunk of their attack surface once the model's output can only ever be something from a known vocabulary.
That isn't accidental. It's a deliberate design decision, where constraining the response space of a probabilistic system is, at once, reliability engineering and security engineering.
Why all this rigour
"Why go to all this trouble?", you might ask.
Because the business context here is unforgiving: the project already had a search system running before we arrived. AI is coming in as a new capability layered on top of what already exists, not as a rewrite. Nobody wants to trade predictable behaviour for a suggestion that they "trust the model, it'll be fine".
Determinism, in this case, isn't technical purism. It's an explicit business requirement.
But this isn't set in stone
That said, it's worth being clear so this doesn't come across the wrong way: this controlled vocabulary isn't gospel forever. It solves today's problem very well, given the vocabulary and the granularity the product needs right now. If that vocabulary ever needs to grow substantially, or if the product ends up needing matching by similarity rather than exact matches against a fixed list, that's when embeddings come back onto the table as something worth considering.
Not because "that's what everyone's doing these days," but because the need has changed. It comes back to the same idea: use the tool that fits the problem you're solving today, and the solution that fits the moment.
For now, though, the list stays closed. And a decision like that only stays defensible if you keep proving that it's actually holding up.
Evaluation as a compass, not a checklist
All this engineering for determinism is worth nothing if you can't measure how well it's actually working.
So we built an eval harness: a set of fixtures pairing real natural-language queries with the canonical JSON we expect back. Touch the prompt layer or the controlled vocabularies, and the whole set gets replayed, so we can see what came out against what should have come out.
The same fixtures run against two models. A lighter, local one scores close to 93%. The stronger one, which is what serves staging and production, generally clears 97%.
Why not use the stronger model everywhere? Every call to it carries a real cost, and during development what we need most is to find out quickly whether the change we just made broke something. The local model gives us that in seconds, cheaply.
But it isn't the check we ship on. Before anything reaches staging or production, the stronger model has to hold up against the same fixtures. The local model keeps us moving while we build. The stronger one is what we actually trust.
Which leaves the remaining 3%. And 97% sounds great until you go and look at what's in there.
Some of it the system can spot on its own. The rest looks perfectly fine from the inside.
The first group is the easy one. Output that doesn't parse, or a value the schema gate rejects outright, takes the whole intent down with it, and search degrades to plain full-text. A value that simply isn't in our vocabularies is a smaller matter: that one field gets dropped, and the search carries on with the filters that survive. We planned for both.
The awkward case is a JSON that's perfectly canonical and still wrong: it misreads what the user meant, so the filters come back a bit too narrow or too broad. The grounding and cross-field checks trim back what they can prove the query never asked for, a price bound the wording doesn't support, a status nobody asked for, an exclusion with no negation behind it. What's left after that is the genuinely invisible part: nothing about it is structurally invalid, so it sails through.
Those are the ones we log. Every drop and every extracted intent leaves a trace, and reading them back tells us whether we're looking at a model issue, a genuinely ambiguous query, or a gap in our own validation. That's what feeds the next round of prompt and vocabulary tweaks, measured again against the same fixtures.
The same cost discipline applies once the system is live. If we've already interpreted exactly the same query in the same context, there's little value in asking the model to do it again. A cache can return the previous result immediately, avoiding another model call altogether. That skips an entire model call per refinement. Less processing, less cost, less latency, and as a bonus, a better experience, because the result comes back instantly.
The rule of thumb is simple: the language model only gets involved when there's natural language to interpret. Where there's a structured filter, the search engine is already excellent at that on its own.
When the AI fails, the system can't go down with it
There's one final design decision worth mentioning. What happens when the model simply doesn't respond? Timeout, unavailability, any failure in the call. The system doesn't sit there waiting. It degrades.
Drop the intent extraction, and search becomes a plain full-text search against the search engine, using the same query the user typed. They never even notice a whole layer went missing. They just get results, maybe a little less refined, but they get results.
Resilience here wasn't a plan B bolted on after someone found a bug in production. It was designed alongside everything else.
The same discipline that guarantees determinism on the happy path is what guarantees that the AI's absence never turns into an error screen.
The alchemy, in the end
Step back far enough and the same handful of forces keep showing up: latency, cost, measured accuracy, and that opening board of constraints, the business context, the budget, data protection, the nature of the problem, what the team actually knows how to do today.
None of these variables decides anything on its own.
And the weight each one carries shifts depending on where the product sits in time. What makes sense at launch isn't what makes sense once user volume takes off. What's safe today might need revisiting tomorrow, once the need has changed.
That's why we call this the alchemy of modern times.
You're not following a fixed formula. You're combining ingredients (predictability, uncertainty, protection, cost, resilience) in the right proportion for the problem in front of you right now.
Perhaps that's the biggest lesson we've taken away.
Models evolve, providers change, and new techniques appear all the time. A good architecture should expect that, that's why we built around clear contracts and well-defined boundaries, making it much easier to replace individual components instead of rethinking the whole architecture.
But the ability to see the constraints before writing the first line of code remains one of the most valuable skills an engineer can develop.
What's your alchemy?
We'd love to hear whether you've taken a different approach, or if there are trade-offs we haven't considered.