Category: Web Search tool

When our agent needed to answer a complex question, it could not rely on search results alone. Sometimes the answer was buried deeper in a webpage. For eg: “Give me minute by minute breakdown of the world cup 2026 Final game” requires going deeper into the content of a related article as compared to “Who was the top scorer for Spain in the world cup 2026 Final?” which could be found in search results alone. 

We covered how our web search tool found relevant URLs in our previous blog.

In this blog, we are going to cover how our web crawl tool finds the relevant content inside a URL through semantic search.

When search results are not enough

A user would ask a question, and web_search would return five URLs that looked relevant. The agent would check the first result, get the description from the preview, and give an answer based on that.

Often the answer was right. A simple question like “how many goals did Ronaldo score against Croatia in World Cup 2026?” could be answered from the preview alone. The search result would have the number right there in the description or first line of the page. Done.

Sometimes it was not enough. A more complex question like “give me a minute by minute breakdown of the events in the Portugal vs Croatia game” could not be answered from the preview. The preview might just say “exciting match with 3 goals” or something generic. The real breakdown of what happened at each minute was deeper in the page, buried in a full match report or article. We had to crawl the entire URL to find the detailed information.

When that happened, we needed to go deeper. We needed a second tool called web_crawl. This tool takes a URL and reads the full page, looking for the specific information the user asked for.

That is when the second problem started.

The leftover problem

Our early version of web_crawl did what we asked it to do. It took a URL and returned the entire content of that page. All of it. Every word.

If a user asked “what is the return policy” and we had to crawl a product page, web_crawl would return not just the return policy, but the product description, customer reviews, navigation menu, footer, everything that was on the page.

Or if a user asked “how many goals did Ronaldo score against Croatia” and we crawled a sports news article, it would return the entire article, including player biographies, team history, match statistics for other games, and commentary. All of it, even though the answer was just one number.

This worked, technically. But it created two real problems.

Why that hurt

First, all that extra content wasted the model’s context tokens. If a user asked “how many goals did Ronaldo score against Croatia” and we crawled a full sports article with 5000 words of match details, player stats, and team history, we were burning tokens on 4999 words of garbage to answer a question that needed one number. With many queries running, those wasted tokens added up fast and made everything slower and more expensive.

Second, dumping unrelated information confused the model. When a model reads a page full of product reviews mixed with the return policy mixed with shipping information, it gets confused about what matters. It might pull information from the wrong part of the page, or make up an answer based on the noise. The real answer gets buried.

It is like asking a librarian for one specific fact and having them hand you the entire book instead of just the page you need. You have to read through all the noise to find the answer, and you might miss it or get lost in the details.

The shift in thinking

We realized that the problem was not with how we were getting the page. It was with what we were keeping from it.

The goal was not to return everything. The goal was to return only what the user asked for. So instead of giving back the whole page when web_crawl ran, we needed a way to figure out which parts of the page actually matched the meaning of the user’s query.

This was different from just searching for words. If a user asked “minute by minute breakdown of the Ronaldo vs Croatia game,” a simple word search would find pages with those words in them. But it would not know which parts of the page had the actual breakdown and which parts had other information like team stats or historical context. We needed web_crawl to understand meaning, not just match keywords.

 

The fix

We used a small AI model called sentence-transformers/all-MiniLM-L6-v2.

Here is how it works. The model takes the user’s query and converts it into a set of numbers that capture the meaning of that query. Then it does the same thing for each chunk of the page. It breaks the page into smaller pieces and converts each one into numbers that capture its meaning.

The magic is that these numbers put similar meanings close together. So “return policy” and “how to send items back” end up close to each other in this space, even though the words are different. “free shipping” and “no shipping cost” also end up close together. And “minute by minute breakdown” and “events that happened during the match” are also nearby in meaning, even though they use different words.

Once we have these numbers for the query and all the chunks of the page, we find which chunks are closest to the query. Those are the chunks that mean the same thing as what the user asked for. We keep only those chunks and return them to the model.

The function kept the same inputs. It still took a query and a URL. It just got smarter about what it gave back.

The result 

Now when a user asks about a return policy, web_crawl gives back only the return policy section. When they ask about a minute by minute breakdown of the Ronaldo game, it returns only the breakdown section from the article, not the player biographies or team history. When they ask about shipping, it returns the shipping information. No more entire pages full of unrelated content.

Lessons Learned

This fixed both problems. The context is clean and focused, so we do not waste tokens on garbage. The model reads only what it needs and does not get confused by unrelated text. There are fewer hallucinations because there is less noise to confuse the model. The results are faster and cheaper because we use fewer tokens.

The real lesson is this. Our web_search tool gets us started with quick answers. But when we need to go deeper into a page, web_crawl now does it smart. It does not just dump the whole page. It finds only what matters.

Matching by meaning beats matching by volume. Giving less, but more relevant, information is better than giving everything. In almost every case, thirty words on topic are better than a thousand words scattered all over the place.

At a glance: Our AI agent’s first web search tool used Crawl4AI, which opened a new headless Chromium browser for every search. Since each browser uses an estimated few hundred MB of memory, a load test with 10 concurrent searches exhausted the server’s memory and crashed it. We rebuilt web search for AI agents with SearXNG as the default, a plain HTTP metasearch call with no browser, and kept Crawl4AI only as a fallback that now runs one shared browser with capped concurrency. The lesson: the most powerful tool is not always the right default.

Our AI agent needed to search the web. At first, this sounded like a solved problem, so we reached for a popular, powerful tool: Crawl4AI, which reads pages by driving a real headless browser. In fact, it worked perfectly for a single search. However, when we load-tested it with 10 searches running at the same time, the server ran out of memory and crashed. In this post, we explain why that happened and why the tool itself was not the problem. Finally, we show how we rebuilt web search for AI agents around a much lighter default, SearXNG, while still keeping Crawl4AI for the pages that truly need a browser.

The first attempt

Our first version used a tool called Crawl4AI. Crawl4AI is an open-source Python library built for collecting web content for AI systems. The way it works is that it opens a full, real web browser in the background, a headless Chromium browser, and uses it to load and read a page the same way a normal browser would. It is a popular and widely used library, with more than 60,000 stars on GitHub, so it felt like a solid, well supported choice.

The crash

Why Crawl4AI handles hard pages so well

Because Crawl4AI drives a real browser, it can do things a plain request cannot. It can wait for the page to finish loading, scroll down to pull in content that only appears as you go, run the page’s own scripts, and then hand back a clean, readable version of the page. This is why it works so well on difficult websites, the ones that build their content with JavaScript after the page first loads, where a plain request would often return only an empty shell. That same power is also where our problem started.

How much memory does a headless browser use?

A real browser is heavy. Running headless Chromium is close to running a full copy of Chrome, with all of its moving parts loaded into memory at the same time. As a result, every browser instance that Crawl4AI opened used a large amount of memory. Public estimates put a headless Chromium instance at roughly 100–500 MB, with 300–500 MB a common planning figure once it renders a real page.

What happened under load testing

On top of that, our early script opened a brand new browser every single time it ran a search, instead of reusing one. So when many searches ran together, we were not opening one browser and sharing it. We were opening a separate browser for each search. Ten searches at the same time meant ten separate browsers. Each one carried its own few hundred MB, and none of that memory was shared between them, so the total added up fast with every extra search running at once.

Two other things made this worse. First, when you run Crawl4AI on your own server, you have to manage all of this yourself, including the browsers and how many of them run at once. Second, the memory cost does not go down as you do more work. Every extra page you want to read at the same time means another full browser, and another few hundred MB on top.

Reading one page at a time was fine. The trouble started the moment we needed to handle many searches at once, which is exactly what a real product has to do. We saw this clearly when we used JMeter to put more load on the application. When we reached 10 concurrent searches, the many browsers running at the same time filled up the server memory, and the server crashed.

The realisation

When we looked at what had happened, we saw the real mistake. We had reached for the most powerful tool and made it our default. But most of our searches were ordinary. They did not need a full browser to read and render an entire page. They only needed a quick and light way to get search results. The powerful tool was not wrong. It was just the wrong choice for the common case.

The fix

We changed our default to a tool called SearXNG.

SearXNG is a free and open-source metasearch engine. A metasearch engine does not keep its own index of the web. Instead, it sends the query to several other search engines, collects their results, and combines them into a single list. Because of this, it does not need to open a browser at all. It talks to the search engines directly and returns results, which is fast and light on memory. It is written in Python, it is released under the AGPL-3.0 open-source license, and it can be self hosted, which means we run it on our own server and stay in control of it. It also does not track or profile its users. It can pull results from many sources, including Google, Bing, Brave, DuckDuckGo, Qwant, Startpage, and Yahoo.

Right now our default setup uses SearXNG together with DuckDuckGo. SearXNG is the layer that runs the search and gathers the results, and DuckDuckGo is one of the search engines it draws those results from. This combination gives us clean search results without the heavy cost of a browser.

We did not throw Crawl4AI away. Some pages really do need a full browser to be read properly. So we kept Crawl4AI as a backup, ready to step in for those harder pages, instead of using it for every search.

How our web search works now

Our web search now runs as a small job pipeline instead of a single function call. First, the chat backend sends a request to our MCP API at /api/web-search. The API verifies the user’s JWT, checks that they have access to the web search tool, and then creates a job ID.

Next, the job goes into Redis. Its state is stored under its own key, and the work itself is pushed onto a Redis stream. At the same time, the backend subscribes to a Pub/Sub completion channel for that job before it starts waiting. As a result, the backend never polls for results; instead, it is notified the moment the job finishes.

A crawler worker then claims the job from the stream and decides what kind of work it is: a search, or a page to read. For searches, it uses SearXNG, which is a plain HTTP call that returns JSON, with no browser involved. Only when SearXNG fails, or a page needs JavaScript to render, does the worker fall back to Crawl4AI. Even then, it no longer opens a new browser per request. Instead, all fallback jobs share one headless Chromium browser with capped concurrency, which fixes the exact mistake that crashed our first version.

Finally, the results are stored back in Redis. Search results skip our semantic re-ranking step, because the search engines have already ranked the snippets. (We cover semantic re-ranking for full pages in our post on smarter web crawling with semantic search.) A response aggregator then builds the final response, the query plus its list of sources, marks the job as done, and publishes the completion event, which wakes the backend instantly.

The lesson

The main lesson for us was simple. The most powerful tool is not always the right default. It is better to match the tool to the job you do most often, and to keep the heavy tool only for the few cases that really need it.

Our first setup worked for a single search but broke when many searches ran at once, which is exactly what happens in a real product. The fix was not a bigger server or a clever trick. It was stepping back and asking what the job actually needed. Most of the time, the answer was less than what we first reached for.