THE BEST LOCAL AI MODEL

The wrong question

If you insist on an answer anyway, the internet offers plenty of leaderboards. Leaderboards are grade point averages. That is unfortunate when the average hides the fact that the candidate failed exactly the subject that matches its future job.

In early September 2026 I had to answer this question regardless. The project budget and the hardware derived from it were fixed: four RTX 5070 Ti with 16 GB each. The task: deliver in local knowledge management.

Which model won is revealed at the end, but it is the least interesting part. What is interesting is the approach. It will still work in a few months, while the model chosen today will probably be obsolete by then. Four steps, one week of effort.

Step 1: Fix the yardstick before the first number exists

This takes twenty minutes and is the most important step of all. If we do not know exactly what we are measuring and cannot express it in numbers, every comparison remains a matter of gut feeling.

But what actually is a "high-quality answer"? That, too, has several dimensions. Each one needs a weight, otherwise it cannot be factored in. And that weight is set beforehand, not afterwards.

On top of that come hard limits: values at which a candidate is out, no matter how good it looks otherwise. The next step shows how important they are.

Step 2: Filter smartly, expensive tests last

The order of the tests is not arbitrary. First come the questions that can be answered in minutes. First: does it even start? Then: does it serve several users? And only after that the expensive question of answer quality.

Nine of twelve configurations drop out before the first question is asked. For three quarters of the field, answer quality is not even the relevant variable. They simply do not fit the available hardware. Of course you could now experiment to get them running after all: a different quantization, a different split across the cards, higher memory utilization, a newer version of the serving software. But as long as enough candidates run out of the box and their results meet the requirements, that remains plan B.

Concurrency and speed

A model counts as usable at a given number of users if two conditions are met: the time to first token stays below three seconds at the 95th percentile, and every user gets at least 20 tokens per second. We deliberately did not set a fixed target number. Instead we run a ladder, 1, 2, 4, 8, 16 users and onwards, until the gate fails twice in a row. The highest level that still meets both conditions is the usable concurrency. For the three finalists that was eight simultaneous users for Nemotron and four each for the other two.

What matters is what is measured. Not the total latency of a request: it mixes the reaction time until the first token with the writing time. Nobody experiences the responsiveness of a system that way. In front of the screen these are two separate things. Above all: how long does nothing happen after my input? And only then: how smoothly does the text flow into the chat window?

Step 3: Build the questions from your own files

This is the real difference to any published leaderboard. And at the same time the biggest effort. We do not measure with publicly available question sets. The models know those by now. We measure with what the model will see every day in the future: an excerpt from your own documents and a question set built on top of it. In our case that was 300 questions from 145 entries of the internal knowledge platform, spread across various departments.

For answer quality we want the broadest possible picture of the model. A simple "can read off facts correctly" would fall short. Hence eight question types. The decisive one is "unanswerable". The trick: 60 of the 300 questions deliberately have no answer in the given context. "I know that I don't know" is one of the most important properties here. A model that supports employees in knowledge management will inevitably hit gaps. Not every question is covered by the documentation. An invented answer then hurts twice: it is wrong, and it looks like information from your own company.

Throw in something unexpected

Internally, AI models rely heavily on world knowledge, meaning what they picked up from their training data and what corresponds to their "most likely" understanding of the world. But what if a fact in your company does not match what is generally taught and described? There is a criterion for that, too. "counterfactual" measures the share of questions where the model answers correctly from the given source, against its own world knowledge.

Step 4: Evaluate, and why the effort pays off

The three remaining candidates got the same 300 questions, the same context, the same format. Grading was done by a separate, larger language model that checked each answer against the stored reference answer.

The overall scores of the three finalists are close together: 0.90, 0.82 and 0.91. This shows that all well-known open models now perform well within a clearly defined task area. A closer look at the measurements, however, confirms the statement from the beginning: the "best model" always depends on what you need it for later.

Broken down by question type, the same field looks different.

Nemotron answered about 31 percent of the unanswerable questions instead of saying that the information is not available. For Qwen it was 3 percent. The same model has the best fact score in the field and the only perfect format adherence. On a public leaderboard it would have looked brilliant.

For a system that gives employees information about company documents, it is the least suitable of the three. An assistant that makes something up in almost a third of the cases instead of naming the gap creates work instead of saving it. Every answer then has to be double-checked.

That is the whole point. Public leaderboards are overall scores and cannot know which sub-score is decisive in your case.

The last row is a planning hint: if your main use case is calculation, model selection is not your main problem. Then the model needs a harness, an environment in which it can hand off calculations and analyses to be executed externally. A language model on its own does not calculate reliably.

The winner

In this case Qwen 3.6 was chosen, not the points winner. Contrary to the weighting, the decisive reason was the refusal rate, which simply has to be right for a knowledge management system.

A points calculation orders a field. But it does not replace the decision.

The limits of the method

A hard threshold makes close cases look sharper than they are. The load gate says: under three seconds passed, above failed. One candidate came in at 2.91 seconds, and exactly that doubled its usable concurrency. A gate orders the field, but it is not a law of nature.

The whole thing is a snapshot. A specific version of the serving software, a specific driver, a specific model release. Two of the five candidates that did not even start failed due to gaps in the software that will probably be closed within six months. The approach remains valid. The ranking does not.

The recipe

1. Fix the weighting before the first number existsAnd write down at which weight the recommendation flips.
2. Sort tests by costDoes it start? Does it serve enough users? Only then the expensive quality measurement.
3. Build the question set from your own documentsWith about 20 percent unanswerable questions, each checked against the full text.
4. Evaluate by question type, not just as an overall scoreOtherwise the decisive sub-score disappears in the average.

The right question is not which model is the best, but which one starts on your hardware, serves your number of users and does well on the one sub-score that matters in your case.

That can be measured. It may take a while, but it is always worth the effort. And afterwards you know not only which model to use, but also which change in circumstances should make you take another look. That is the real return.