How intelligent systems find what you're looking for

Imagine you're searching for the best books about Machine Learning. You type your question into a search engine, and within seconds, you get a perfectly curated list of results. But have you ever wondered what's really happening behind the scenes?

That's what we'll explore here. In this post, we'll dive into Information Retrieval (IR), the science that powers modern search engines and intelligent systems, and we'll take a first look at the three main types of retrieval systems that make it all work.

At ustwo, we've been exploring how these same retrieval principles power real-world products. It's not just theory, we're putting these concepts into practice with hands-on projects.

A great example is our work with the Cancer Platform, where we used Retrieval-Augmented Generation (RAG) to connect people with trusted medical information. Our goal was to make complex data feel accessible and meaningful: using semantic search to find relevant articles and studies from a carefully curated knowledge base, then generating clear, human-friendly answers while preserving the original citations. It's the same IR logic in action, but applied with purpose, helping users find what truly matters to them.

By the end of this article, you'll understand the logic that connects data, meaning, and relevance, and how machines decide what deserves to appear first.

My goal is to open this world for you, to show that these techniques aren’t magic reserved for data scientists, but skills anyone curious enough can learn.

Let's open the black box of search together. 🚀

What is information retrieval?

At its core, Information retrieval is about finding relevant information within a large amount of data, based on what a user asks for. Think of it like a massive library. You walk in and ask, "Do you have books about Artificial Intelligence?" The librarian, powered by IR, quickly checks the catalogue and hands you the most relevant ones first.

It's the same logic behind:

• Google search results;
• movie and product recommendation systems;
• and AI models like ChatGPT that retrieve knowledge before generating answers.

Everything starts with the same goal: finding the right piece of information, quickly and accurately.

How it works: the building blocks

An IR system can be broken down into four main steps. Let's walk through each one with practical examples.

Step 1: Indexing, building the catalogue

Imagine you have three small documents:

Doc1: I love coffee and plants.
Doc2: Coffee is essential for developers.
Doc3: Plants make people happy.

The system breaks them into words (tokens) and builds an index, a map of which words appear where:

• coffee → documents 1, 2
• plants → documents 1, 3
• developers → document 2
• happy → document 3

So when someone searches for "coffee", the system instantly knows that Doc1 and Doc2 are relevant. That's indexing in action. Once the index is built, the system is ready to handle queries.

Step 2: Querying, finding matches

When a user types something like:

coffee AND plants

The system looks up both words and returns only the documents that contain both:

Result: Doc1

This is how a Boolean retrieval model works: logical, fast, and exact. But sometimes we need systems that can understand meaning, not just matching words. That's where ranking comes in, helping the system decide which results matter most.

Step 3: Ranking, choosing what matters most

Now imagine two documents both mention "coffee", but one mentions it once, and the other mentions it ten times.

We can assign scores to measure relevance:

scores = {
  "Doc1": 0.3,
  "Doc2": 0.9
}

sorted_docs = sorted(scores.items(), key=lambda x: x[1], reverse=True)
print(sorted_docs)

Output:

[('Doc2', 0.9), ('Doc1', 0.3)]

That's ranking, showing the most relevant results first. In real systems, this can involve techniques like TF-IDF, BM25, or even vector similarity between text embeddings. But before any of this can happen, there's a crucial step: preparing the data.

Step 4: Pre-processing, cleaning and preparing text

Before indexing, systems need to clean up data: remove punctuation, convert to lowercase, and split into tokens.

import re

text = "Coffee is Essential for Developers!"
tokens = re.findall(r'\b\w+\b', text.lower())

print(tokens)

Output:

['coffee', 'is', 'essential', 'for', 'developers']

Now the text is standardised and ready to be indexed, because "Coffee" and "coffee" should obviously be treated as the same word.

Why this matters

Every search you do, whether it's finding a product, a song, or an answer from an AI, relies on these exact steps: clean → index → query → rank.

That's the foundation of Information Retrieval, and it's also the heart of Retrieval-Augmented Generation (RAG), the technique that powers intelligent systems capable of both searching and reasoning.

The three main retrieval systems

Now that we know what IR is, there's one big question: Why do we need different types of retrieval systems?

Well, not every search is the same. Sometimes you know exactly what you're looking for, like a specific book or product. Other times, you're just exploring, hoping to discover something interesting or similar.

Different retrieval systems are designed for different goals. Let's look at the three main ones.

1. Boolean retrieval model

Imagine you're on an e-commerce site with very strict filters. You know exactly what you want: a Samsung smartphone under $500.

This is how the Boolean model works: it follows precise logic. You use operators like AND, OR, and NOT to filter results:

• smartphone AND samsung AND under 500
• iphone OR samsung
• laptop NOT gaming

✅ Pros:

• You get exactly what you asked for.
• You can exclude things you don't want.
• Simple and predictable.

❌ Cons:

• It's "all or nothing."
• Results aren't ranked by relevance, they either match or they don't.

This model works perfectly when you need strict, rule-based searches, like filtering inventory or technical data.

2. Vector space model

Now imagine you're still on that same e-commerce site, but this time you type:

"A smartphone with a great camera for travel photos."

Here, the system needs flexibility. It measures how close each product is to what you described, ranking them based on similarity.

That's the Vector space model. It represents documents and queries as vectors in a multi-dimensional space and calculates similarity (often with cosine similarity or TF-IDF scores).

✅ Pros:

• Finds results that are similar, not just exact matches.
• More human-like understanding of what's relevant.

❌ Cons:

• If ranking isn't well-calibrated, results can feel "off."
• May require query adjustments to get better matches.

It's perfect for search engines, recommendation systems, or any place where you want the best match, not just an exact one.

3. Probabilistic retrieval model

Finally, imagine a system that knows your taste, almost like a personal assistant. It has learnt from millions of users and can predict what you're likely to prefer based on your history and behaviour.

That's the Probabilistic retrieval model, the foundation of recommendation systems and modern ranking algorithms like BM25. It works by estimating the probability that a document (or product) is relevant to your query.

✅ Pros:

• Feels personalised and context-aware.
• Improves as more data is collected.

❌ Cons:

• Requires large amounts of data to perform well.
• May struggle with very unique or new queries.

This is what powers systems like Netflix recommendations, Amazon's "You may also like", and even Retrieval-Augmented Generation (RAG) in AI.

When to use each model

Boolean retrieval model

• Best for: when requirements are strict and binary
• Example: "Samsung phone under $500"

Vector space model

• Best for: when exploring or ranking by similarity
• Example: "Phones with great cameras"

Probabilistic retrieval model

• Best for: when personalisation matters
• Example: "Products similar to what I liked before"

Each model fits a different kind of search: Boolean is for precision, Vector Space for similarity, and Probabilistic for personalisation.

Wrapping up

Understanding these systems is more than just theory: it's the foundation of how intelligent systems connect people and information. Everything from search engines to chatbots, recommendation engines, and AI assistants relies on these same principles.

This was a high-level introduction, just enough to give you the big picture. In the next parts of the series, we'll explore each model in practice: Boolean, Vector Space, and Probabilistic, to see how they actually work under the hood.

See you in the next chapter. 🌍