On prem LLM vs API: the open-source half you actually move

What each option is, on four layers

An API call sends a prompt to a model somebody else serves, and the vendor bills you per token. Self-hosting runs the weights on your own cards. vLLM serves them at throughput behind an OpenAI-compatible endpoint. Ollama runs one on a workstation with less setup, and the request never crosses the network boundary. OnPrem.LLM is the toolkit built for the strict end of that case: "a Python-based toolkit for applying large language models (LLMs) to sensitive, non-public data in offline or restricted environments."

Both descriptions cover one layer. The model is inference, the weights and whatever serves tokens. Around it sit three more layers. The agent runtime runs the loop and reaches the tools. Data and memory hold the files, the state and the session history. The policy and audit layer says who may act, and what got recorded. Move the model and only the first layer changes address, unless you move the others on purpose. The four layers hold for any stack you compare.

That is why an API-to-self-hosted migration often ends in a stack nobody planned: a model in the rack, an agent loop calling a vendor's cloud, and an audit record split across both.

The bill changes shape before it changes size

A hosted model bill scales with use. It arrives as a variable cost, never as capital. The finance conversation is about the trend rather than the purchase. An owned box inverts that. One payment, then a monthly cost for power, cooling and the person who patches the serving stack, whether the tokens flow or not.

The hardware line is the one everybody can find, and the community threads show how wide it runs. A reader in r/selfhosted priced an offline, inference-only server for a small company at $10k to $15k USD (r/selfhosted). That is one reader's budget for the small half of the stack, and it buys a model server with nothing around it.

The number that decides the comparison is the monthly volume at which the owned box beats the invoice. A paper on arXiv sets out "a cost-benefit analysis framework to help organizations determine when on-premise LLM deployment becomes economically viable" (arXiv 2509.18101), and its unit is a whole deployment rather than a card price. Use that unit, because a spec that stops at the GPUs misses the rows that recur every month.

The upgrade you stop paying for is the one you start doing

A vendor ships a new model and your prompts may behave differently the same afternoon. You did not schedule it and you cannot decline it. Owning the weights reverses that: nothing changes until you change it, and every model bump becomes a test cycle, a prompt review and a rollback plan you write.

Control of the schedule and of where requests go costs engineering hours that keep coming back. Teams that regret the move tend to regret this part of it more than the card purchase.

Capacity is the same trade from another angle. Concurrency on your own box is capped by card memory and the cache behind each context window. The busy afternoon an API absorbed with a queue becomes a latency number your team watches. Sizing that box is its own exercise, covered in on-prem AI server requirements.

The hybrid most teams land on

The version of this move that survives a budget usually leaves the model where it is. The runtime and the record change address; inference stays a connection. Kortix is built for that split: "Any model provider with your own keys. Or the ChatGPT plan you already pay for." The model is a connection a project holds, an open-weight endpoint on your own gateway or a hosted API, and "Run it on Kortix Cloud, in your VPC, or on your own on-prem network. Self-host is free."

Read the boundary before you buy anything, because it is where a hybrid stops being a metaphor. A self-hosted instance is "the full Kortix platform on a server you control: one Docker Compose stack that the kortix self-host CLI installs, configures and updates." The same guide then states the limit. "Sessions run on Platinum sandboxes, not on the server. The instance must be reachable from the internet so those sandboxes can call back."

The control plane, the project data and the record sit in your building. The machine that executes a session is managed. Each session gets "One isolated sandbox per session. Each session has its own isolated machine and branch." "Session work reaches main through a change request. Merge is default-deny for agents."

Putting that half on your own server is three commands:

curl -fsSL https://kortix.com/install | bash
kortix self-host init --domain kortix.example.com --admin-email you@example.com
kortix self-host start

The weights never entered that install. The model stays a key the project holds, which is the part of the stack you can move later without redoing the control layer.

When self-hosting the model is the wrong first move

Own the model last when volume is low, or when load arrives in bursts. Own it last, too, when nobody on the team wants to be on call for a serving stack. A card that idles most of the month is not cheaper than a bill that tracks use. A model that changes every quarter turns a frozen upgrade into an unpaid project.

It is also the wrong first move when the requirement that started the conversation was audit rather than inference. A reviewer asking who approved a tool call, on whose branch, with what record, has a question about the runtime and the record. Answering it with a GPU purchase leaves the question open, because the loop still runs somewhere you cannot read.

The order that holds up: place the runtime and the record first, keep the model a connection, then move inference when the volume justifies a box. The self-hosted agent guide works through that split in detail. It is also why the layer worth moving first is the open-source one around the model rather than the model itself.