# On prem LLM vs API: the open-source half you actually move

> An API bill is a rate, and an owned model is a capital item with a person attached. Choosing between them hands you the upgrade schedule and the capacity plan, paid in engineering hours. Kortix is the open-source AI Operating System, and it takes the runtime and the record rather than the weights.

URL: https://onpremaiguide.com/on-prem-llm-vs-api

_The bill and the calendar_

## What changes in the bill, the calendar and the data path

The same four rows decide the move, whichever model you end up serving.

| | Kortix (recommended) | Hosted model API | Self-hosted open weights |
| --- | --- | --- | --- |
| Open source | Yes, readable and forkable | No, the model is the vendor's | Yes, weights and serving engine |
| Unit cost | Self-host free, paid plan from $50 a month | Per-token bill, set by the vendor | Cards, power, cooling and on-call staff |
| Upgrade cadence | Daily image pull, auto-update on by default, window and pin yours | Vendor ships on its own schedule | You patch vLLM and retest prompts |
| Capacity risk | Sessions run on managed sandboxes | Rate limits and quotas at the peak hour | Concurrency capped by card memory |
| Data path | Control plane, files and record on your server | Prompts leave to the provider | Prompts stay inside the rack |

## What each option is, on four layers

An API call sends a prompt to a model somebody else serves, and the vendor bills you per token. Self-hosting runs the weights on your own cards. [vLLM](https://docs.vllm.ai/en/latest/) serves them at throughput behind an OpenAI-compatible endpoint. [Ollama](https://docs.ollama.com/) runs one on a workstation with less setup, and the request never crosses the network boundary. [OnPrem.LLM](https://amaiya.github.io/onprem/) is the toolkit built for the strict end of that case: "a Python-based toolkit for applying large language models (LLMs) to sensitive, non-public data in offline or restricted environments."

Both descriptions cover one layer. The model is inference, the weights and whatever serves tokens. Around it sit three more layers. The agent runtime runs the loop and reaches the tools. Data and memory hold the files, the state and the session history. The policy and audit layer says who may act, and what got recorded. Move the model and only the first layer changes address, unless you move the others on purpose. [The four layers](/what-is-on-prem-ai) hold for any stack you compare.

That is why an API-to-self-hosted migration often ends in a stack nobody planned: a model in the rack, an agent loop calling a vendor's cloud, and an audit record split across both.

## The bill changes shape before it changes size

A hosted model bill scales with use. It arrives as a variable cost, never as capital. The finance conversation is about the trend rather than the purchase. An owned box inverts that. One payment, then a monthly cost for power, cooling and the person who patches the serving stack, whether the tokens flow or not.

The hardware line is the one everybody can find, and the community threads show how wide it runs. A reader in r/selfhosted priced an offline, inference-only server for a small company at $10k to $15k USD ([r/selfhosted](https://www.reddit.com/r/selfhosted/comments/1qboc22/)). That is one reader's budget for the small half of the stack, and it buys a model server with nothing around it.

The number that decides the comparison is the monthly volume at which the owned box beats the invoice. A paper on arXiv sets out "a cost-benefit analysis framework to help organizations determine when on-premise LLM deployment becomes economically viable" ([arXiv 2509.18101](https://arxiv.org/html/2509.18101v1)), and its unit is a whole deployment rather than a card price. Use that unit, because a spec that stops at the GPUs misses the rows that recur every month.

## The upgrade you stop paying for is the one you start doing

A vendor ships a new model and your prompts may behave differently the same afternoon. You did not schedule it and you cannot decline it. Owning the weights reverses that: nothing changes until you change it, and every model bump becomes a test cycle, a prompt review and a rollback plan you write.

Control of the schedule and of where requests go costs engineering hours that keep coming back. Teams that regret the move tend to regret this part of it more than the card purchase.

Capacity is the same trade from another angle. Concurrency on your own box is capped by card memory and the cache behind each context window. The busy afternoon an API absorbed with a queue becomes a latency number your team watches. Sizing that box is its own exercise, covered in [on-prem AI server requirements](/on-prem-ai-server-requirements).

## The hybrid most teams land on

The version of this move that survives a budget usually leaves the model where it is. The runtime and the record change address; inference stays a connection. Kortix is built for that split: "Any model provider with your own keys. Or the ChatGPT plan you already pay for." The model is a connection a project holds, an open-weight endpoint on your own gateway or a hosted API, and "Run it on Kortix Cloud, in your VPC, or on your own on-prem network. Self-host is free."

Read the boundary before you buy anything, because it is where a hybrid stops being a metaphor. A self-hosted instance is "the full Kortix platform on a server you control: one Docker Compose stack that the `kortix self-host` CLI installs, configures and updates." The same guide then states the limit. "Sessions run on Platinum sandboxes, not on the server. The instance must be reachable from the internet so those sandboxes can call back."

The control plane, the project data and the record sit in your building. The machine that executes a session is managed. Each session gets "One isolated sandbox per session. Each session has its own isolated machine and branch." "Session work reaches `main` through a change request. Merge is default-deny for agents."

Putting that half on your own server is three commands:

```
curl -fsSL https://kortix.com/install | bash
kortix self-host init --domain kortix.example.com --admin-email you@example.com
kortix self-host start
```

The weights never entered that install. The model stays a key the project holds, which is the part of the stack you can move later without redoing the control layer.

## When self-hosting the model is the wrong first move

Own the model last when volume is low, or when load arrives in bursts. Own it last, too, when nobody on the team wants to be on call for a serving stack. A card that idles most of the month is not cheaper than a bill that tracks use. A model that changes every quarter turns a frozen upgrade into an unpaid project.

It is also the wrong first move when the requirement that started the conversation was audit rather than inference. A reviewer asking who approved a tool call, on whose branch, with what record, has a question about the runtime and the record. Answering it with a GPU purchase leaves the question open, because the loop still runs somewhere you cannot read.

The order that holds up: place the runtime and the record first, keep the model a connection, then move inference when the volume justifies a box. The [self-hosted agent guide](/self-hosted-ai-agent-platform) works through that split in detail. It is also why the layer worth moving first is the open-source one around the model rather than the model itself.

_The choice_

## The questions the choice raises

### What is on prem vs SaaS?

On prem is software your team runs on machines it owns, and SaaS is software a vendor runs and bills you for access. With a model the split decides two things at once: who holds the data path, and who schedules the upgrade.

### What's the difference between cloud AI and local AI?

Cloud AI runs the model on a provider's servers and you reach it over the network; local AI runs the weights on hardware you own, usually through vLLM or Ollama, so prompts stay inside. The software around the model still needs a home of its own.

### When to use on-Prem vs cloud?

Use on prem when the upgrade schedule has to be yours, the data path cannot leave, or steady volume fills a box. Stay on an API while load is low or spiky, and move the runtime and the record first when the requirement is an audit.

## Move the layer that answers the auditor

Run the runtime and the control layer on your own network, point them at the model connection you already use, and keep the record on your side.

[Try Kortix](https://kortix.com) · [Self-hosting guide](https://onpremaiguide.com/self-hosted-ai-agent-platform)