Some engineering problems do not look like problems when they first arrive. Ours started with a simple request. Our product needed to search the web. That sounded easy. People search the web all the time, so it felt like something that was already solved. Once we started building it, we found that the simple request hid a real decision about which tool to use.
The first attempt
Our first version used a tool called Crawl4AI. Crawl4AI is an open-source Python library built for collecting web content for AI systems. The way it works is that it opens a full, real web browser in the background, a headless Chromium browser, and uses it to load and read a page the same way a normal browser would. It is a popular and widely used library, with more than 60,000 stars on GitHub, so it felt like a solid, well supported choice.
The crash
Because Crawl4AI drives a real browser, it can do things a plain request cannot. It can wait for the page to finish loading, scroll down to pull in content that only appears as you go, run the page’s own scripts, and then hand back a clean, readable version of the page. This is why it works so well on difficult websites, the ones that build their content with JavaScript after the page first loads, where a plain request would often return only an empty shell. That same power is also where our problem started.
A real browser is heavy. A headless Chromium browser is not a small program. It is close to running a full copy of Chrome, with all of its moving parts loaded into memory at the same time. Each browser instance that Crawl4AI opened used a large amount of memory, around 300 MB every time.
On top of that, our early script opened a brand new browser every single time it ran a search, instead of reusing one. So when many searches ran together, we were not opening one browser and sharing it. We were opening a separate browser for each search. Ten searches at the same time meant ten separate browsers. Each one carried its own 300 MB, and none of that memory was shared between them, so the total added up fast with every extra search running at once.
Two other things made this worse. First, when you run Crawl4AI on your own server, you have to manage all of this yourself, including the browsers and how many of them run at once. Second, the memory cost does not go down as you do more work. Every extra page you want to read at the same time means another full browser, and another 300 MB on top.
Reading one page at a time was fine. The trouble started the moment we needed to handle many searches at once, which is exactly what a real product has to do. We saw this clearly when we used JMeter to put more load on the application. As soon as we increased the number of concurrent runs, the many browsers running at the same time filled up the server memory, and the server crashed.
The realisation
When we looked at what had happened, we saw the real mistake. We had reached for the most powerful tool and made it our default. But most of our searches were ordinary. They did not need a full browser to read and render an entire page. They only needed a quick and light way to get search results. The powerful tool was not wrong. It was just the wrong choice for the common case.
The fix
We changed our default to a tool called SearXNG.
SearXNG is a free and open-source metasearch engine. A metasearch engine does not keep its own index of the web. Instead, it sends the query to several other search engines, collects their results, and combines them into a single list. Because of this, it does not need to open a browser at all. It talks to the search engines directly and returns results, which is fast and light on memory. It is written in Python, it is released under the AGPL-3.0 open-source license, and it can be self hosted, which means we run it on our own server and stay in control of it. It also does not track or profile its users. It can pull results from many sources, including Google, Bing, Brave, DuckDuckGo, Qwant, Startpage, and Yahoo.
Right now our default setup uses SearXNG together with DuckDuckGo. SearXNG is the layer that runs the search and gathers the results, and DuckDuckGo is one of the search engines it draws those results from. This combination gives us clean search results without the heavy cost of a browser.
We did not throw Crawl4AI away. Some pages really do need a full browser to be read properly. So we kept Crawl4AI as a backup, ready to step in for those harder pages, instead of using it for every search.
The lesson
The main lesson for us was simple. The most powerful tool is not always the right default. It is better to match the tool to the job you do most often, and to keep the heavy tool only for the few cases that really need it.
Our first setup worked for a single search but broke when many searches ran at once, which is exactly what happens in a real product. The fix was not a bigger server or a clever trick. It was stepping back and asking what the job actually needed. Most of the time, the answer was less than what we first reached for.