Local AI, in plain terms,
and when it is the wrong answer.
Most of what is written about local AI is written by people selling hardware or people who have never run it. This is what it means, what it is good at, and the cases where a hosted model is simply the better purchase.
What the words mean
"Local AI" means the model runs on a computer you control. Not on OpenAI's servers, not on Google's, not on ours. The model file is on the disk, the question goes in from your network, and the answer comes back. Nothing crosses a border.
The reason people care is almost never performance. It is that the document being read contains something they would rather not upload: a client contract, a personnel file, a drawing under NDA, or a price list that is the whole business.
That is a legitimate reason. It is also the only reason that survives contact with the alternatives, so it is worth being honest about the rest.
- Good at: reading a document and pulling the numbers out, sorting, summarising, drafting from your own material, answering questions about a fixed set of files.
- Weak at: open-ended reasoning, long multi-step work, and coding. The hosted frontier models are better and it is not close.
- Costs: hardware once, plus somebody's attention for updates and backups.
- Saves: the per-token bill, and the question of where your documents went.
A local model is not a frontier model.
It is very good at reading a document and pulling the numbers out. It is not a replacement for the largest hosted models on hard reasoning, and the gap is wider than the marketing suggests. If the job needs the best available answer rather than a private one, buy the hosted model.
When local is the right answer
One: the documents cannot leave. If the file is a client's, or contains personal information, or is covered by an agreement that says it stays on your premises, then the decision is made for you and the only question left is which model.
Two: the volume is steady. A machine is bought once. If you are reading a hundred documents a week, every week, the arithmetic works. If it is a handful a month, it does not.
Three: the connection is not dependable. Warehouses, job sites, and anywhere the internet is a convenience rather than a utility.
Four: somebody has to answer for it. If the output matters, you should be able to name the model, the version and the documents behind an answer. That is easier when you own the machine.
Five: the job is narrow. The narrower the task, the better a small model does. "Pull every quantity and unit price out of these tender documents" is a job a local model does well. "Work out why this project is running late" is not.
The pattern is that local wins on constraint, and hosted wins on capability. Decide which one your problem is before you buy anything.
Yes
- The volume is steady enough that a machine pays for itself
- Somebody there can look after a computer on the network
- The job is narrow: reading, pulling, sorting, answering from fixed files
No
- Buy the hosted model. It is better, and at low volume it is cheaper.
Questions people actually ask
It means the model runs on a computer you control, in your building, rather than on somebody else's servers. Your questions and your documents never leave the network. That is the only thing the word guarantees. It says nothing about quality.
No, and anyone telling you otherwise is selling something. A local model of the size a small business can run is very good at reading documents and pulling facts out of them. It is noticeably weaker at open-ended reasoning, long multi-step tasks and coding.
It depends entirely on the model, and the model should be chosen after the job rather than before it. A machine that comfortably runs a mid-size model costs about the same as a good workstation. There is no single answer and anybody who gives you one before knowing the job is guessing.
Yes, that is one of the real reasons to do it. Once the models are installed and the documents indexed, the machine does not need a connection. Job sites and warehouses are the obvious cases.
At steady volume, usually. At low volume, almost never: the hardware costs more than the subscription and it keeps costing attention. The crossover depends on how many documents a week actually get read, which is why the first question is always about volume and never about hardware.
Three things, in order of likelihood. It answers confidently and wrongly, which is true of every model and is why an evaluation set matters. It stops being maintained, because it is a computer on a network and nobody owns it. And it is quietly retrained or updated and starts giving different answers, which is why versions get pinned.
If it still sounds right
The way to start is not a plan. It is one document and one question you already know the answer to. Run that, and you find out in a morning whether the whole idea is worth pursuing for your business.
If the answer is right, the rest follows. If it is close but wrong, that tells you what to fix. If it is useless, you have saved yourself a hardware purchase.
- The local AI machine, which is the service this article feeds.
- AI agents that act, which is the hosted version of the same idea for work that needs a stronger model.
- Data extraction, the job local models do best.
Tell us what is not working.
Describe the problem in a few lines. You get a plain answer about whether it is worth fixing and what it would take, before any money changes hands.