Hosted AI APIs vs self-hosted models: how to choose
This decision has to be made first, because it eliminates options rather than merely ranking them. Choosing a model and then discovering the data cannot leave your network is one of the few AI mistakes that forces a rebuild instead of a tuning pass.
What is the actual difference?
A hosted API means your data leaves your infrastructure, goes to a provider, and an answer comes back. You pay per call, get whatever the provider's latest model is, and carry none of the operational burden.
Self-hosting means running open-weight models on hardware you control, whether that is your own servers or GPU instances in your own cloud account. The data never leaves, you pay for the hardware by the hour, and you own the operations: model updates, scaling, monitoring, and the GPU capacity planning that goes with them.
The capability gap has narrowed considerably, and for many production tasks open-weight models are now genuinely sufficient. The gap that has not closed is operational effort, and that is usually what decides it for smaller teams.
| Hosted API | Self-hosted | |
|---|---|---|
| Data leaves your network | Yes | No |
| Cost shape | Per call, scales with usage | Per hour, paid whether idle or busy |
| Best volume profile | Spiky or low | Steady and high |
| Time to first working version | Days | Weeks |
| Model updates | The provider's problem | Yours |
| Capability at the top end | Currently ahead | Sufficient for most production tasks |
| Operational burden | Minimal | Real and ongoing |
| Licence obligations | Commercial terms you accept | Must be checked per model |
What decides it, in order?
Data residency first, because it is binary. If contract, regulation or client policy says the data cannot leave your infrastructure, self-hosting is the only option and the rest of the comparison is irrelevant. This is common in defence, healthcare, parts of financial services, and any engagement where a client's own data protection agreement forbids sub-processors.
Then volume profile, because it sets the economics. A per-hour cost only beats a per-call cost when the hardware is busy. Spiky traffic on dedicated GPUs means paying for idle capacity, and low steady traffic almost never justifies it.
Then latency, if it matters at all. A round trip to a hosted provider adds network time you cannot tune away. For a background job that is irrelevant; for something in a user's typing loop it may not be.
Only then capability, and by that point the field has usually narrowed to one option anyway.
What does self-hosting actually cost in effort?
More than teams expect, and the model itself is the easy part. You need GPU capacity planning, because a model that fits comfortably in memory at batch size one may not at ten. You need a plan for updates, since open-weight models improve quickly and staying on an old one is a slow-accumulating disadvantage. You need monitoring that catches quality drift rather than just uptime.
You also need someone who owns it. The most common failure is not technical: it is that the model was stood up by whoever was enthusiastic, and eighteen months later nobody knows how to update it or what it was evaluated against.
Where this is worth it, it is clearly worth it. Where residency does not force it and volume does not justify it, it is a large ongoing commitment bought for very little.
What about licences on open-weight models?
This is the part that gets skipped, and it is the one that creates obligations for you rather than for a vendor. Open weights are not the same as open licences, and the terms vary a great deal.
Some carry permissive terms that allow commercial use freely. Some are copyleft, and where a model or its surrounding library is licensed under a strong copyleft such as AGPL, delivering the software over a network can oblige you to offer the source of your whole application. Some are explicitly non-commercial, which means they cannot be used in a product you sell regardless of how the code is licensed. And the training data can carry its own restrictions, separate from the weights, with several widely used research datasets permitting research use only.
The practical rule: check the licence on the weights, the licence on the inference library, and the terms on any dataset used for fine-tuning, before anything reaches a customer. Enterprise buyers increasingly ask for a software bill of materials, and this is where an awkward answer surfaces.
Can you do both?
Yes, and it is often the right answer. A common pattern is a hosted API for the small share of genuinely hard cases, and a self-hosted small model handling the bulk. That keeps the majority of traffic and the majority of the data inside your infrastructure while retaining a capability ceiling for the exceptions.
It also hedges the decision. Building behind an internal interface rather than calling a provider's SDK directly from your application means swapping the backing model is a configuration change instead of a project. Given how quickly this market moves, that abstraction earns its keep within a year.
If you are unsure, start hosted behind your own interface. It ships sooner, tells you whether the feature is worth having, and leaves the door open.
Common questions
- Can AI be used if our data cannot leave our infrastructure?
- Yes, by running open-weight models on hardware you control, either your own servers or GPU instances in your own cloud account. The data never reaches a third party. This is the standard approach where contract, regulation or client policy forbids sub-processors, and it should be decided before a model is chosen because it eliminates the hosted options entirely.
- Are open-weight models good enough for production?
- For most production tasks, yes. Classification, extraction, summarisation and routing are well served by open-weight models today. The remaining gap is at the hardest end of reasoning, and the more relevant gap for most teams is operational: someone has to own capacity planning, updates and drift monitoring, indefinitely.
- Which is cheaper, a hosted API or self-hosting?
- It depends on your traffic shape rather than on the technology. Hosted APIs charge per call, so cost scales with usage and idle time is free. Self-hosting charges per hour whether the GPU is busy or not, so it wins only at steady high utilisation. Spiky or low-volume workloads are almost always cheaper hosted.
- Do we need to check licences on open-weight models?
- Yes, and on three things separately: the model weights, the inference library, and any dataset used for fine-tuning. Strong copyleft licences such as AGPL can oblige you to offer your application's source when you deliver it over a network, some weights are explicitly non-commercial, and several widely used training datasets permit research use only.
