What is On-Premise AI Hosting?
An AI model is the program that reads a question and writes an answer, like the one behind a chatbot. Most businesses use one through a cloud service, which sends your text to the provider’s servers.
With on-premise hosting, the model runs on a server in your office or data centre instead. This is also called private or self-hosted AI.
This is possible thanks to open-weight models: AI models whose files (the “weights”) anyone can download and run. Each model has its own licence, and the terms differ.
What it needs
- A server with a GPU (a graphics card, which runs AI models much faster than a normal processor) with enough memory for the model you choose.
- Fast storage: model files are large, often several gigabytes each.
- Networking and access control, so only the right people and systems can reach it.
- Upkeep: updates, security patches and monitoring.
Why businesses use it
Your data stays with you: questions, documents and answers are not sent to an outside AI provider.
Contracts and the law: some contracts forbid sharing data with third parties, and POPIA (section 72) sets conditions for sending personal information outside South Africa. Local AI avoids that step.
No per-use fees: you pay for hardware, power and upkeep, not per request. Whether that works out cheaper depends on how much you use it.
Works offline: it keeps running without the internet, and only changes when you update it.
Common tools for running the model include Ollama (opens in a new window) (a simple way to download and run models), vLLM (opens in a new window) (built for fast serving on GPUs) and llama.cpp (opens in a new window) (lightweight; also runs on normal processors and Apple silicon). Each offers an OpenAI-compatible API, a common way for software to talk to AI models. Ollama also offers cloud models, so we set it to local models only.
What we do for you
Needs assessment: what you want AI to do, what data it will touch and which rules apply.
Hardware and model sizing: which model fits the job, and what server it needs. We can check what you already have.
Installation and setup: Linux server, GPU drivers, serving tool and model, usually in Docker containers.
Connecting it to your documents or software, so it answers from your own files.
Security hardening: access control, firewall rules and logs, with no outside traffic unless you allow it.
Updates and monitoring: patches, model upgrades and health checks.
Is it right for you?
A good fit if
- Your data must stay in your network, because of a contract, policy or law.
- Your team will use AI often, every day.
- You already run your own servers, or are happy to.
Maybe not if
- You have no data-control requirement and light use. A cloud AI service is simpler and quicker to start.
- You need the most capable models. Many leading cloud models can’t be downloaded, and large open models need costly hardware.
- No one will look after the server, and you don’t want ongoing support.
- You can’t budget for GPU hardware upfront.
On-premise vs cloud AI at a glance
| On-premise | Cloud AI service | |
|---|---|---|
| Where your data goes | Stays on your servers | Sent to the provider |
| Upfront cost | Hardware purchase | Usually none |
| Running cost | Power, upkeep and staff time | Pay per use or subscription |
| Model choice | Open-weight models | The provider’s models, including ones you can’t download |
| Upkeep | You, or us for you | The provider |
| Works offline | Yes | No |
| Growing | Add hardware | Usually immediate |
How we work
Talk
We learn what you want AI to do, and what data and rules are involved.
Plan
We choose a model, size the hardware and agree budget and security.
Build
We set up the server, model and connections, then test with your real tasks.
Support
We handle updates, monitoring and model upgrades.
Questions we get asked
Something else on your mind? Ask us.
Is a private AI model as good as the big cloud services?
It depends on the task. Open-weight models handle many everyday jobs well, such as summarising and drafting. The largest cloud models may do better on harder tasks. We test with your real work before you buy hardware.
What hardware do we need?
It depends on the model’s size and how many people use it at once. Larger models need more GPU memory. Compressed (“quantised”) versions need less, with some loss of quality.
Can we use open-weight models in our business?
Usually, but check each licence. Most of Mistral’s open models use Apache 2.0, which allows commercial use. Meta’s Llama licence adds conditions, such as an acceptable use policy. We check the terms for the model you choose.
Does it need the internet?
Only to download the model and updates. After that it can run fully offline.
Can it answer questions from our own documents?
Yes. A common method, called RAG, finds the relevant passages in your files and gives them to the model with the question.
Are you a partner of any AI company?
No. We are independent and recommend what fits your needs.
Related pages
Let's work together
Ready to talk?
Tell us what you need. We'll reply within 2 working days.
Ollama, vLLM, llama.cpp, Llama and Mistral are trademarks of their respective owners. Mammoth Digital is not affiliated with or endorsed by any of them.
Last updated