How on premise AI splits into four layers, open source at each

What does "on premise AI" mean?

On premise AI describes where a system runs before it describes what that system is. TDWI puts it as AI on infrastructure you own: "servers, networking equipment, and software that your organization owns and maintains" (TDWI). iguazio's glossary says an on-premise platform "runs AI services and applications within the organization's physical environment" rather than on the cloud (iguazio). Both sentences put the hardware and the software on your side of the wall, run by your own staff.

TDWI's next sentence carries more weight than the definition. On-premises AI "gives you complete control over your AI environment but also complete responsibility for managing it". The responsibility it then lists is hardware procurement, software installation, infrastructure management, and staff. None of those items is about the model. The definition assumes the AI system moves as one object.

Advice on the choice keeps that assumption. Microsoft's guidance for developers weighing cloud against local models compares privacy, cost, maintenance, latency, and scaling for one model, on one machine or one endpoint. That is the right frame for a desktop app. A company running agents over its finance system is placing something larger, and the parts of it do not share an answer.

So the definition worth carrying into a budget splits the system four ways and asks the location question at each. The four names hold for the rest of this decision: inference, agent runtime, data and memory, and policy and audit.

Four layers, four separate answers

Inference holds the weights and serves tokens to whatever calls it. vLLM and Ollama run open-weight models on hardware you own. onprem.ai sells that layer as a complete stack, hardware included, or as software for servers you already run. A hosted API leaves the layer in someone else's building. Putting it on-prem buys control of the weights and the upgrade schedule, and it caps throughput at the cards in the rack.

The agent runtime is the loop that reads a task, decides which tool to call, keeps the session's state, and hands back the result. Open-source options differ in what they hand you. LangGraph is a library for building the loop yourself. ibl.ai ships a sandboxed server per agent. Kortix ships the runtime, with the control layer that sits around it. The decision is where that loop executes, and who can read what it did.

Data and memory covers files, state, and the record of work: repos, buckets, databases, transcripts. On most stacks this layer is already on-prem and nobody had to decide it, so it goes missing from the plan. It also decides the sharpest question in the exercise: what leaves when a session ends.

Policy and audit is who may act, under what approval, and what got recorded. No hardware quote contains it. It is the layer a reviewer opens first, and the one where open source is worth most, because your reviewer reads the permission check and the log writer instead of sending you a questionnaire. Kortix states the layer as "Per-resource permissions for people and agents. Roles, groups, and an audit trail", and "Every action is recorded. Reading, exporting and streaming the audit log depend on the plan."

What on-prem moves into the budget

Hardware is the line every quote gets right. TDWI lists the rest on the same page: ongoing costs for "power, cooling, and maintenance", plus staff costs for management and support, on top of the upfront spend on hardware and software. Add the agent runtime, the integration work, and the audit record. The shape of the purchase changes: one large capital item sits under three recurring ones, each with a different owner and review cycle.

Whether that trade pays is a real question with a published method. A paper on arXiv sets out "a cost-benefit analysis framework to help organizations determine when on-premise LLM deployment becomes economically viable" (arXiv 2509.18101), and its unit of analysis is a whole deployment. Read it before the hardware spec. The model server is one line inside a larger calculation.

A budget shows the per-token bill. It rarely shows the capacity you buy and cannot shrink, or the person who is now on call for the model server. Sizing the box is its own exercise, covered in on-prem AI server requirements.

What on-prem changes about control

Control gets easier to argue about once it is broken into three questions. Whose keys does the model call carry? Whose record can an auditor read without asking a vendor? What happens to the work when the contract ends?

Kortix answers the first two for the layers it runs. "Run it on Kortix Cloud, in your VPC, or on your own on-prem network. Self-host is free", and the model is "Any model provider with your own keys", so the keys can stay yours without a GPU in your building. The record is a layer your reviewer reads directly, in the permission check and the log writer, rather than through a vendor's answers.

Where the boundary falls on a real install

A self-hosted Kortix is "the full Kortix platform on a server you control: one Docker Compose stack that the kortix self-host CLI installs, configures and updates". The same guide sets the limit of the word on-prem. Sessions run on Platinum sandboxes, not on the server, and the instance must be reachable from the internet so those sandboxes can call back. What stays in your building is the control plane, the data, and the record. The machine that executes agent work is managed.

If the requirement is that nothing leaves at all, not even a sandbox calling home, Kortix does not meet it today, and two products document that they do. onprem.ai promises "reliable operation, even in air-gapped environments" (onprem.ai), and ibl.ai's runtime "deploys inside your perimeter, on your cloud, on-premise or air-gapped" (ibl.ai). The self-hosted agent guide takes the same five questions further.

In a budget, each of the four arrives as its own line, with its own owner, cost, and exit. A stack where all four sit inside your building is a different purchase, and a different set of risks, from one where only the control plane does. Deciding the runtime before the model server is usually the cheaper order, because the runtime decision tells you what the inference layer has to serve.