The question usually arrives after the first month with an AI tool: could we not run this on our own server? Then the data stays in the house and the usage based bill goes away.
Technically it works. The weights of many models are public, the software to run them is free, a suitable machine rented within an hour. The question is when it pays off.
When is a local AI model worth it for a small business?
When a category of data must not leave the building, when the volume is steadily high, or when a workflow has to keep running without an internet connection. Otherwise a model behind an interface is cheaper and set up in a fraction of the time.
Data residency is the reason named most often for a server of your own, and the weakest: with a provider processing inside the EU and a processor contract under Article 28 GDPR, the legal construction is in place. Volume, by contrast, is arithmetic.
What is a local AI model?
A local AI model is a language model whose weights sit on hardware you own or rent and whose requests never leave that machine.
The difference from the cloud lies in location and billing. You pay for compute that stands ready even when nobody asks, and you work with the version you downloaded.
What hardware and what licence does it need?
Graphics memory sets the limit, because the model should fit into it completely. Four figures from vendor documentation.
- According to Google, Gemma 4 at 12 billion parameters needs roughly 6.7 GB of graphics memory in the Q4_0 quantisation and roughly 26.7 GB at full precision.
- A rented server with 24 GB of graphics memory costs 214 euro a month on the Hetzner price list, plus a 209 euro setup fee, excluding VAT. The 96 GB tier is listed there at 999 euro.
- The card in that rented server, an RTX PRO 4000 Blackwell SFF, is specified by NVIDIA at 24 GB of memory and 70 W maximum power draw. Electricity is not the item worth thinking about here.
- According to the Ollama documentation, the context memory required grows with the number of requests processed in parallel. What is enough for one person is not automatically enough for ten.
Open weights are not the same as an open licence. According to its model overview, Mistral publishes Mistral Small 4 and the Ministral 3 models under Apache 2.0. The Llama 4 Community License from Meta, by its own text, requires a visible Built with Llama notice and a separate licence from Meta above 700 million monthly active users. For Gemma, Google requires the use restrictions to be passed on with every redistribution.
What does your own server cost against the cloud?
At small volumes the cloud is markedly cheaper, and the gap is wider than people estimate. Your own server starts to pay at a volume a twenty person business rarely reaches.
A calculation with its assumptions on the table. One case, say sorting an incoming email and drafting a reply, uses about 1,000 tokens of input and 300 tokens of output. According to the Mistral pricing page, Mistral Large costs 0.50 dollars per million input tokens and 1.50 dollars per million output tokens. A thousand such cases a month come to about 0.95 dollars.
To reach the monthly rent of the 24 GB server, there would have to be more than 200,000 cases a month. What neither figure contains is the work on the server: installation, updates, monitoring and a named person for the morning when the model does not answer. As a guide, the first setup takes two to five days.
A local model rarely saves money, it shifts something: you trade a bill based on usage for fixed cost and a responsibility inside the house. You make that trade for the data or for the volume.
What does a local model change legally?
A local model removes one question and leaves the rest standing. If no data goes to a service provider, that step needs no processor contract: Article 28 GDPR requires one for processing carried out by a processor. Duties under the AI Act are untouched.
- The open source exemption in the AI Act does not carry far here. Article 2(12) exempts AI systems released under free and open source licences, but does not apply where they are put into service as high risk systems or as a system falling under Article 5 or Article 50.
- AI literacy attaches to the use, not to the location of the hard drive. Article 4 of the AI Act requires providers and deployers to take measures supporting the AI literacy of their staff and of others operating the system on their behalf.
- Offering a self hosted system under your own name can move you into the provider role under Article 25(1) of the AI Act. The provision concerns high risk systems; whether a given case falls under it is a legal question.
Austria has its own point of contact for the AI Act. The KI Servicestelle at the regulator RTR describes itself as a contact point and information hub, publishes material on the regulation and can be reached at ki@rtr.at. It is not legal advice. Which records belong kept when AI is in use is set out on our page about GDPR and the AI Act.
How do you decide this?
In this order. The first three steps need no hardware.
- 1.Count how many cases arrive per month. Do not estimate: keep a tally for two weeks and extrapolate.
- 2.Check which data goes into the case. A category that must not leave the building is the first solid reason for a server of your own.
- 3.Build the workflow for four weeks on a hosted model, then look at the invoice. Anything under ten euro a month needs no machine of its own.
- 4.Only once volume or data category argue for it, rent a machine and put the same workflow on it. Rented rather than bought: then the exit is a cancellation.
- 5.Write down who looks after the server, who updates the model and when the decision gets reviewed. Half a year is a workable interval.
One reason appears on no invoice: the version stays put. In its deprecation list, OpenAI names 14 January 2027 as the shutdown date for two older models and promises at least six months of notice. Running a tested workflow on a downloaded model means you pick the moment to switch. How we select and introduce tools is on our page on tool selection and rollout.