In our last post, we explained what MindzEval does differently. This post is the practical follow-up: it shows you how to set it up and run your first evaluation.
MindzEval is also now on PyPI, so you can install it with pip. If you would like to see the code, it lives in its main repo on GitHub, github.com/MindzKonnected/mindzeval. To show it working, though, we use a separate example repo, and once that runs, you swap in your own agent.
Before you begin
To run the example, you will need:
- Python 3.11 or newer.
- An API key for a chat model. The example uses OpenAI’s
GPT-4o-miniby default. However, you can switch to any LangChain chat model that supports structured output, including free ones, by changingagent/llm.pyand installing that provider’s LangChain package.
Working through the example repo
mindzeval-integration-examples holds ready-to-run examples, one per framework. The first one uses LangGraph, and more frameworks will follow. When you clone it, you get:
- A working SQL agent, built with LangGraph, that answers questions about a film catalogue, as in our earlier posts.
- Seven test cases covering single questions, a three-question follow-up conversation, a vague request, an off-topic question and a display limit.
How the repo is organised
Everything for this example sits in the langgraph folder:
langgraph/
├── agent/
│ ├── graph.py
│ └── llm.py
├── eval_adapter.py
├── mindzeval.yaml
├── cases/films.yaml
├── setup_db.py
└── requirements.txt
The key point is that the folder splits into two parts:
agent/is the agent itself. It plans, runs the SQL and writes the reply, yet it does not know MindzEval exists.- Three files connect it to MindzEval.
eval_adapter.pyis the bridge between MindzEval and the agent,mindzeval.yamlsays where the agent and the database are, andcases/films.yamlholds the questions along with their answer keys.
The remaining two files, setup_db.py and requirements.txt, only prepare the example, and you run both in the next section.

Setting up and running your first evaluation
In a terminal, run these commands in order:
git clone https://github.com/MindzKonnected/mindzeval-integration-examples.git
cd mindzeval-integration-examples/langgraph
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
export OPENAI_API_KEY="your-key" # Windows: $env:OPENAI_API_KEY="your-key"
python setup_db.py
mindzeval lint
mindzeval validate
mindzeval run
- Clone the repo and open the example folder.
- Create and activate a virtual environment.
- Install the requirements.
- Set your model’s API key.
- Build the sample database.
setup_db.pycreatesfilms.dbwith 200 made-up films, so no real data is involved. - Check the config and test cases.
mindzeval lintlooks for mistakes such as typos, missing fields or a password typed straight into a file, without connecting to anything. - Check the answer key.
mindzeval validateruns each reference query against the database, catching any that are broken, empty or unstable before the agent is graded. - Run the evaluation. Lastly,
mindzeval runasks the agent every question, compares its results with the reference queries, scores behaviour and safety, and writes the reports.
Understanding your first results
Every turn gets one of four verdicts:
- PASS: the agent got it right.
- FAIL: the agent got it wrong.
- NOT_SCORED: the turn could not be graded, as the agent did not send a field that the check needs.
- BLOCKED: the reference query itself failed its check against the database. Rather than give the agent an unfair fail, MindzEval leaves that turn ungraded.
On top of that, every failed turn gets exactly one cause, such as truncated, wrong_value, extra_rows, asked_instead_of_answering or safety_violation. That way, you always know where to look.

With that in mind, here is a typical run of the example:
- 7 of 9 turns pass. The follow-up conversation counts as three turns, which is why there are nine.
- Both failures are display limits, not wrong queries. The agent shows at most 50 rows, but two questions have more matching films (107 and 76). MindzEval therefore re-ran the agent’s SQL without the limit, found the full results correct, and gave the cause as
truncated. - Uncapped accuracy is still 100%. This score asks whether the query was right once the display limit is ignored, and here it was, even though two turns failed. The re-run only happens when that SQL ends in a
LIMIT. It is skipped when the answer key has its own limit, as in a “top 10” question, and for questions whose result is a single number or grouped totals. - Behaviour accuracy is 100% as well. This score checks whether the agent made the right call to answer, ask or decline. In this run, the agent asked for clarification on “Show me the data”, declined the weather question, and answered the clear question about Japanese films.
- The run exits with code 0, because both scores clear the thresholds set in
mindzeval.yaml.

Each run also saves three report files in the mindzeval-results folder:
- HTML, to share. It shows every turn, the SQL, and why each verdict was given.
- CSV, with one row per turn, for answering “why did this fail?”
- JSON, for CI and other tooling.
Connecting your own SQL agent
Moving from the example to your own agent comes down to a few changes:
- Install MindzEval in your own project with
pip install mindzeval. If your database is PostgreSQL, usepip install "mindzeval[postgres]"instead. - Replace the agent, but keep the adapter’s shape. In the example, the adapter is about 15 lines:
from agent.graph import build_graph graph = build_graph() # build once, reuse for every case async def answer(question, history, params): messages = [(m["role"], m["content"]) for m in history] + [("user", question)] state = await graph.ainvoke( {"messages": messages}, config={"configurable": {"model": params.get("model")}}, ) return { "answer": state["messages"][-1].content, "sql": state.get("sql"), "status": state["status"], # answer | clarify | decline "reason": state.get("reason"), } - Return the right fields. The adapter sends back a dictionary that uses MindzEval’s exact key names:
answer: the agent’s reply.sql: the query it ran, which MindzEval re-runs against the database and checks for safety.status: whether the agent answered, asked or declined, so that its behaviour can be scored.
A misspelt key, such as
queryinstead ofsql, is rejected with an error rather than ignored. For agents that are not in Python, or that you cannot call during the eval, the connecting agents guide covers HTTP services and saved outputs. - Point
database.urlinmindzeval.yamlat your database. - Write 10 to 20 test cases from real user questions. The case format guide shows every option.
- Run the same three commands:
mindzeval lint,mindzeval validateandmindzeval run.
Common mistakes
- Using a database login that can write. A read-only login means nothing can change by accident. Keep its password in an environment variable such as
${DB_PASSWORD}, rather than in the file. - Passing the history twice. If your agent already remembers the conversation, for example through a LangGraph checkpointer, send just the new question. Otherwise, the earlier turns arrive twice.
- Leaving out
status. MindzEval then cannot tell whether answering, asking or declining was correct, so those checks are skipped. - Cutting rows in code instead of the SQL. A correct query can look incomplete as a result. To avoid that, return
truncated: truewhenever your agent trims the list itself. - Testing only easy questions. A handful of simple cases gives false confidence. Include a few vague and off-topic questions too, plus clear ones that must not trigger a clarifying question.
- Pasting credentials or customer data into the config, the cases or a GitHub issue, where they stay in your git history and logs.
What else can you do
- Compare models. Add a
matrix:block tomindzeval.yaml, for examplemodel: [gpt-4o-mini, gpt-4o], and every combination runs side by side. - Run part of the suite. Use
mindzeval run --case <id>for a single case, ormindzeval run --listto see what would run before anything does. - Break the agent on purpose. Edit the prompt in
agent/graph.pyand watch which cases change. - Tell us what happened. Feedback and bug reports are welcome in the MindzEval issues.
In our next post, we will cover the upcoming features of MindzEval.
Want to see how your SQL agent scores?
MindzEval is built to give you a number you can trust for your SQL agent. To find out more, call us on +91 85957 97775.





























