Reading time: about 8 minutes
Build vs buy for on-prem AI is really a three-way choice, not two: build your own on-prem stack (your hardware and your team), buy a productized private platform deployed at your site, or rent cloud or a managed API. On pure cost at moderate volume, a per-token API is usually cheapest. On-prem, whether built or bought, only starts to win on price at high, sustained utilization. And the bill is not decided by the GPUs but by two lines that most calculations drop: people and compliance. So an honest decision splits into two questions: do the data require on-prem, and if so, build or buy, so you do not pay with your own team for something you could get as a product.
Start with the mistake that wrecks most of these calculations. People line up the GPU price on the on-prem side against an apparent zero on the cloud side, because cloud has no visible cost at the start. Cloud looks cheaper, until you sum eighteen months of invoices and add what the price list does not show. Build vs buy is not a question about hardware, it is a question about the full cost of ownership over a three-year horizon.
Three options, not two
"Build vs buy" sounds like a binary, but with AI it hides three different models, each with its own bill.
- Build, that is DIY on-prem. You buy your own hardware, stand up an open-weight model yourself and maintain the whole stack: updates, security, integrations. Full control, but full operational responsibility too, realistically one to two engineers.
- Buy, that is a productized private platform. You get a product deployed on-prem or in a dedicated, isolated instance, but the vendor maintains the stack, versions and support. The same control over data as build, without building a platform team from scratch.
- Rent, that is cloud or a managed API. You stand up nothing and pay for usage. The lowest barrier to entry, but the data leaves your perimeter and the regulatory and per-token cost grows with volume.
Mixing these three is the commonest reason a calculation does not survive a board meeting. Before you count anything, establish which of these paths is your real alternative, because comparing the CAPEX of your own hardware with a per-token rate measures two different things: capital versus usage.
The five cost categories that must enter TCO
An honest TCO covers five categories on each side, over the same three-year horizon. Compute, the raw cost of running the model, is the easiest to count and the least differentiating. The real difference sits elsewhere.
- Compute. A GPU, or a per-token rate multiplied by time. The smallest differentiating line, even though most calculators focus on it.
- Infrastructure and energy. Power, cooling, networking, colocation. Real on-prem, hidden in the price on cloud.
- People. Standing up and running the model, security, updates. This is where the biggest gap between build and buy lives.
- Compliance. On cloud, annual vendor due diligence, transfer mapping, DPA clauses, re-assessment on every subprocessor change. Real work with no convenient unit rate.
- Project and exit costs. Integration with ERP and MES, training, the first six months of iteration, and the cost of any exit from a vendor.
The scale of this error can be large. In the three-year on-prem cost breakdown cited by Spheron's analysis, staff cost can exceed hardware cost, and power and maintenance add tens of thousands more that the "GPU price" never shows. Rule of thumb: add 20 to 40 percent to the plain compute sum for lines that have no convenient unit rate.
When on-prem genuinely wins on cost
One variable flips the result: utilization. On-prem is largely a fixed cost, so the higher and steadier the load, the lower the cost per answer. Cloud is the opposite: you pay for what you use, so at low or variable traffic it is cheaper.
Two credible 2026 sources show how much the threshold depends on assumptions. The Lenovo Press analysis of multi-GPU configurations argues that for high, sustained load owned hardware breaks even within months against on-demand cloud, and that "the era of cloud-first for all AI workloads is over" for continuous inference. On the other side, Spheron's analysis notes that at competitive GPU cloud rates the break-even case for on-prem essentially disappears, and that most production teams run at 40 to 65 percent utilization due to traffic variability, which makes on-prem hard to justify on cost alone. The practical line that emerges from both: somewhere around 70 to 80 percent sustained utilization, and only against expensive on-demand cloud, not against every offer.
The conclusion is neither "always on-prem" nor "always cloud". At moderate volume you justify on-prem by data control and compliance, not by price, and you do it with eyes open, knowing what that control costs. Which workload genuinely needs on-prem and which can stay in the cloud is what we unpack in the piece on how only part of your processes need a sovereign approach.
Decision table: build, buy or rent
The quickest way to find the direction is to line up your real situation against the model that fits it and the reason behind it.
| Your situation | Model | Why |
|---|---|---|
| Data could be public, moderate volume, a pilot | Rent (cloud / API) | Lowest barrier to entry, isolation adds nothing here |
| Sensitive data, but no ML or platform team | Buy (a ready platform) | Data control without the cost of building and running a stack |
| High, steady volume and your own engineering team | Build (DIY on-prem) | Lowest marginal cost when the hardware is truly loaded |
| NIS2-covered entity, audit asks about the data boundary | Buy or Build | The boundary defends itself in one sentence, the team decides which |
| Variable traffic, load below half the day | Rent, possibly Buy | Owned hardware pays for the hours when no one asks |
The pattern is clear: the choice between cloud and on-prem turns on data sensitivity and volume, and the choice between build and buy turns on whether you want to, and can, run your own ML stack.
Three paths, two questions: do the data need on-prem, and if so, build or buy.
Build vs buy: what really separates the two
Once the data settle that you are staying on-prem, the real build vs buy question remains. Both paths give the same control over data, they differ in who bears the cost of maintaining it.
Build hands you full control over every part of the stack, but moves the whole operational load onto you: standing up the model, updates, security patching, hardware sizing, integrations. That is realistically one to two engineering roles and the longest time to first result. It makes sense when AI is a core of what you do, not a tool, and when you have a team you want to have anyway.
Buy leaves you control over the data and takes away maintaining the stack. The vendor owns versions, security and sizing, you get a working tool faster and without building the competence from scratch. It is usually the right path for a manufacturer for whom AI is an important tool but not the business itself. The point is not to let "buy" become a new form of dependence, which is why criteria such as rights to the model and a real exit plan have to be checked up front, as we unpack in the piece on single-tenant versus shared cloud.
In other words: build buys maximum flexibility at the cost of your own team, buy buys time and predictability at the cost of some flexibility. The full picture of when on-prem makes sense at all is in the guide to on-prem AI in manufacturing: when it fits and when it doesn't, and when private AI beats public in the piece on private AI for manufacturing.
How to compute your own TCO step by step
An order that turns a rough estimate into a decision you can defend to the board.
- Establish real volume and mode. How many queries per month, and whether the model must be available around the clock or only during plant hours. That number decides which side of the threshold you are on.
- Pick the cloud reference point. If you would stand up the model yourself anyway, compare against GPU rental. If a managed API would do, compare per token. These are two different calculations.
- Count five categories, not just compute. Add people, infrastructure, compliance and project costs, over the same three-year horizon on both sides.
- Separate the cost decision from the risk decision. If volume is moderate, you choose on-prem for data control, not for price, and you need to know what that control costs.
- Only then build vs buy. Once you know you need on-prem, ask whether you want to and can run the stack yourself, or buy it as a product faster and cheaper.
To plug in your own numbers rather than rough orders of magnitude, start with our value calculator. And if you want to check first which of your processes need on-prem at all, the readiness mini-audit takes 10 minutes and leaves no data behind.
Frequently asked questions
What is cheaper, on-prem or cloud for AI?
At moderate volume, on pure cost, a managed per-token API usually wins. On-prem only starts to beat cloud on price at high, sustained utilization. Below that threshold you justify on-prem by data control and compliance, not by price.
At what utilization does on-prem AI pay off?
It depends which cloud you compare against. 2026 analyses point to a threshold around 70 to 80 percent sustained utilization against expensive on-demand cloud, but at competitive GPU rates that threshold can disappear. Most teams run at 40 to 65 percent, which is hard to justify on cost alone.
Build or buy for on-prem AI?
Build gives maximum flexibility, but at the cost of one to two engineering roles and the longest rollout. Buy keeps control over data and removes maintaining the stack, so it is usually right for a company for which AI is a tool, not the business.
What most often drops out of a TCO calculation?
On the on-prem side: people, energy and first-half-year project costs. On the cloud side: the vendor's regulatory cost, egress, storage and GPU hours paid while no one asks. Add 20 to 40 percent to the plain compute sum.
Does cheaper cloud mean on-prem makes no sense?
No. It means on-prem has to be justified by data control and compliance, not price, while volume is moderate. At high, sustained load the cost case and the regulatory case point the same way.
Fryderyk, CortexMine. We write about private AI for NIS2-covered manufacturers, based on our own deployments and tests.
Prefer to talk it through? Book a 30-minute call with the founder, no pitch, just your case.