Local LLMs for business: what they are, when they pay off, and how to run one
A local LLM is a large language model that runs on infrastructure you control: a server in your office, a rack in a Canadian data centre, or a private cloud tenancy with no shared endpoints. Your prompts, your documents and the model's answers never leave that boundary. This is the practical core of sovereign AI, and it is now within reach of mid-size companies, not only banks and governments.
Two things made that possible. Open-weight models such as Llama, Mistral, Qwen and Gemma now match the general-purpose cloud models of eighteen months ago on most business tasks. And small language models (SLMs) in the 3 to 14 billion parameter range run well on a single workstation-class GPU, which means a capable deployment no longer needs a data-centre budget.
When a local LLM pays off
Sensitivity. If the model needs to read client files, contracts, patient notes, engineering drawings, network configurations or anything covered by Québec's Law 25 or by a customer's own confidentiality terms, sending it to a public API is a conversation you would rather not have with legal. A local model removes the question.
Volume. Cloud models bill per token. At a few hundred requests a day the bill is trivial; at tens of thousands of requests a day, or when every request carries a long document, a local deployment can cost less within the first year, with a predictable monthly number instead of a usage curve.
Latency and availability. A model running next to your systems answers in tens of milliseconds and keeps working during a provider outage or a rate-limit incident. For voice agents and anything on a factory floor, that is the difference between a tool people trust and one they route around.
Specialisation. A small model fine-tuned on your terminology and retrieving from your approved documents often beats a much larger general model on your actual tasks, because it is judged against your examples, not against the internet.
When it doesn't
If your data is not sensitive, your volumes are modest and the work is generic (drafting, summarising, translating public material), a managed API is cheaper and simpler. The honest answer is that many companies should start there and move the sensitive, high-volume workflows to a local model once they exist.
What running one actually involves
Hardware. One GPU with 24 to 48 GB of memory serves a 7 to 14 billion parameter model to a team; two to four GPUs serve a 70 billion parameter model or heavier concurrency. Used enterprise cards and workstation GPUs make this a capital cost in the tens of thousands of dollars, not hundreds.
Serving. An inference server (vLLM, llama.cpp, Ollama or TensorRT-LLM) exposes the model behind an API that looks like the public ones, so applications do not care where the model lives. Retrieval sits beside it: your documents, indexed and permission-aware, so the model answers from approved knowledge rather than from memory.
Evaluation. Before anything goes live, a set of real questions with known good answers, scored every time the model, the prompts or the documents change. This is the step most projects skip and the one that decides whether people keep using the system after week two.
Operations. Model updates, access control, logging, backups and a named person who owns it. A local model is a system you run, not a subscription you forget.
The three questions to settle first
Which workflow, specifically, and what does success look like in that workflow? What data does it need to see, and who is allowed to see the answers? And what is the monthly cost of the way it is done today, so the project is judged against a real number? With those answered, a first local deployment is a six-to-ten-week project, and it starts with a 30-minute conversation.