In Part 1, we built MindzEval to replace a feeling with a number. Just as in Part 1, the examples use a film catalogue as a stand-in for our real data.
The idea behind MindzEval was that instead of grading the agent’s SQL letter by letter, we grade whether it returned the right films. To do that, we keep a known-correct query, which we call a golden query, for every test question. We then run it fresh on every run and compare its answer with the agent’s.
However, that design rests on one assumption: that every golden query is actually correct. As it turned out, one of ours was not. In Part 1, we said the golden query for “films from Spain” returned 2,662. The correct answer is 2,661.
This post explains how we found that out, and what MindzEval now does so that it does not happen again. As before, the examples use a film catalogue as a stand-in for our real data.
How we found the wrong golden query
The question was the same one from Part 1: how many films come from Spain? The agent ran its query and answered 2,661. Our golden query, on the other hand, answered 2,662. By the rules of the suite, the agent was off by one, so the case was marked as a failure.
When we looked closer, though, the agent turned out to be right, and our golden query was the one that was wrong.
The cause was a leftover in the data:
- The catalogue links films to countries through a separate table. Each row in that table says “this film is from this country”.
- One of those rows pointed at a film that no longer existed. The film had been removed from the film table, but its link to Spain had been left behind after an import.
- Our golden query counted films through the link table, so it counted that missing film too.
- The agent counted films starting from the film table itself, so it only counted films that actually exist.
Since the difference was only one film, nobody would have spotted it by reading the numbers. Nobody recounts 2,662 films by hand. If we had trusted the golden query, we might have changed the correct agent to match a broken test. As a result, the agent would have got worse while its score went up. That is why a wrong golden query is so risky: it does not just miss real bugs, it reports bugs that are not there.
How MindzEval checks its golden queries
The lesson we took from this is that golden queries are not something you write once and trust forever. Like any other code, they need their own tests. So MindzEval now has a separate validator for the golden queries.
It is different from the self-checks in Part 1: those test the harness code, whereas this one tests the data we grade against. Moreover, every golden query has to pass it before we trust the scores it produces, and it runs three checks, starting with the cheapest.
Check 1: Flag golden queries that come back empty
Why it matters
When a golden query returns no results, it does not make the test harder. Instead, it means the test itself is broken, because there is nothing for the agent’s answer to be checked against.
For example, if a question about silent films quietly returns no rows, then any agent that also returns no rows will pass, including one that has simply given up. Consequently, an empty golden query looks like a test while testing nothing.
What we do
- The validator runs every golden query and checks whether it returned any rows.
- It flags every empty result unless the case says that an empty answer is expected.
- An unexpected empty result counts as a bug in the test, not as a pass for the agent.
Check 2: Compare the count with what we recorded
Why it matters
As Part 1 explained, we store the golden query rather than its rows, so that the answer stays current when the data refreshes. Even so, a live query can change its answer for two very different reasons. Either the data really changed, or the query no longer means what we intended. From the score alone, you cannot tell which one happened.
For example, suppose our golden query for sci-fi films used to return 215, and today it returns 225. That could simply mean ten new sci-fi films were added to the catalogue, in which case the new answer is correct.
On the other hand, suppose the catalogue renamed its ‘Sci-Fi & Fantasy’ genre to ‘Science Fiction’. In that case, our golden query would still be looking for the old name, so it might suddenly return far fewer films, or none at all. Both changes show up as a different number, but only the first one is a real change in the data. The second one is a broken test.
What we do
- When we first write a golden query, we note the count it returned, for example “this should return 215 sci-fi films”. The query still produces the answer; the note is only there for comparison.
- The validator re-runs the query and compares the live count with that note.
- If the two disagree, we look at the case before it reaches a report. That way, we find out whether the data changed or the query drifted, instead of spending a morning blaming the agent for a gap the test created.
Check 3: Make sure the golden counts real things
Why it matters
Checks 1 and 2 are good at spotting warning signs. Check 1 tells us when a golden query returns nothing, and Check 2 tells us when its count suddenly changes. However, neither of them can catch the problem we had with the Spain example.
Check 3 looks for orphan records. To see what an orphan is, it helps to know that the catalogue keeps two separate lists:
- The films list holds every film that actually exists, one row per film.
- The country list links each film to its country, with rows such as “film 500 is from Spain”.
Normally, every film in the country list also appears in the films list. An orphan appears when that is no longer true:
- A film is deleted from the films list, for example film 900.
- However, its row in the country list stays behind, so it still says “film 900 is from Spain”.
- That leftover row is the orphan, because it points to a film that no longer exists.
This is exactly what happened in the Spain example:
- Our golden query counted Spanish films using only the country list, so it counted the orphan too and returned 2,662.
- The agent only counted Spanish films that also appear in the films list, so it skipped the orphan and returned 2,661.
Check 3 is designed to catch this. It finds orphans like film 900 and works out whether any golden query is counting them as if they were real films.
What we do
- The validator first looks for orphan records in the link tables.
- For every golden query that uses one of those tables, it rewrites the query with the orphans removed.
- It then runs both versions and compares the answers.
-
- If they match, the orphans did not affect that question, so the golden query is fine as written.
- If they differ, the golden query depends on counting something that does not exist. In that case, the validator fails the query and asks for the count to start from the real record, not from the link table.
That is exactly the change that turns 2,662 into 2,661. More importantly, the fix does not come from the agent’s answer, and it does not come from our own judgement either. Instead, it comes from the database.
Why it is easy to trust golden queries too much
Writing golden queries already feels like the careful part of building an evaluation, so it is tempting to assume they are the one thing that does not need checking.
However, that is exactly where our own mistakes hide. Because every score is measured against the golden queries, one wrong golden query does not just affect one case. It affects every number that depends on it.
After all, an evaluation suite exists to catch confident wrong answers, and a golden query is a confident answer that we wrote ourselves. So it should meet the same standard we set for the agent.
The lesson
Part 1’s lesson was to grade the answer, not the query. Part 2 adds one step before that: check the golden queries before you trust the grade. In practice, that means three things:
- Flag golden queries that come back empty.
- Watch their counts for unexpected changes.
- Make sure they count records that actually exist.
Above all, when the agent and your test disagree, keep in mind that the test might be the one that is wrong. In our case, it was.
Want an evaluation you can trust?
MindzEval grades every stage of a text-to-SQL pipeline, scores refusals in both directions, and checks its own golden queries against the live database before it trusts a single result. To find out what it could reveal about your agent, call us on +91 85957 97775.
Leave a Reply