# On prem AI server requirements: one half GPU, one half open-source > A model small enough for one 24 GB card runs on a server you can buy this week. The requirements list that survives a budget has two halves: the box, and the open-source software that runs agents on it. Kortix is the open-source AI Operating System, and it fills the runtime and the record. URL: https://onpremaiguide.com/on-prem-ai-server-requirements _The install_ ## The server's own requirements, as the guide writes them DNS and ports first, then the install, then the check the guide runs last. ### DNS `DNS records` ``` Create A or AAAA records for kortix.example.com and api.kortix.example.com, plus a wildcard *.p.kortix.example.com for previews. Open ports 80 and 443. ``` Source: https://kortix.com/docs/host ### Install `terminal` ```sh curl -fsSL https://kortix.com/install | bash kortix self-host init --domain kortix.example.com --admin-email you@example.com kortix self-host start ``` Source: https://kortix.com/docs/host ### Check `terminal` ```sh kortix self-host doctor ``` Source: https://kortix.com/docs/host ## GPU memory is the first number, and the only line you pay once On-prem model sizing is arithmetic, and the arithmetic is public. Weights take about two bytes per parameter at 16-bit precision. A 70B model is roughly 140 GB of files before it serves anything. The serving engine holds a KV cache sized by your context length and your concurrency. Add a 1.3 to 2.0 multiplier on top for everything else, as one vendor sizing guide does ([iternal.ai](https://iternal.ai/how-to-deploy-llm-on-premise), read October 2026). That guide puts a 7B or 8B model on one 24 GB card, and a 32B to 34B model on 48 to 80 GB. A 70B model takes two to four 80 GB cards at FP8. Quantization is where the card count stops halving. Four-bit weights cut a model to about a quarter of its 16-bit size. Mixture-of-experts models keep less of that saving than the parameter count suggests. Every expert stays resident in memory, and a token touches only the few it routes to. A 671B model with 37B active still wants around 700 GB at FP8. Size for total parameters, since active ones decide only compute. A budget built on the active count shows up as an out-of-memory crash in week two. Three more numbers belong beside the card count. System RAM holds the loader, the tokenizer, request queues, and on a CPU-only build the weights themselves. Storage carries about 140 GB per 70B model at 16-bit. Add a quantized copy if you keep one, plus logs and a cache that spills under load. Memory bandwidth sets how fast tokens leave the box. A card with enough capacity and a narrow bus reads as slow next to a data-center part. Network depends on which half you buy. One box serving one model needs a path to the applications that call it, and little else. Two or more nodes serving one model add an interconnect between them, sized to move activations rather than files. Whether the model should sit on your box at all is the earlier question, and [renting tokens or owning them](/on-prem-llm-vs-api) settles it. ## The software rows a hardware quote never carries A box that answers with tokens is half a system. The other half decides whether the model reaches your systems, who is allowed to ask, and what record survives the quarter. Five rows belong in the requirements document, each with a name beside it, because somebody has to patch them. - Inference server: vLLM and Ollama are the two open-source defaults. [vLLM](https://docs.vllm.ai/en/latest/) serves one model to many callers behind an OpenAI-compatible endpoint, and Ollama runs a model on a workstation with less setup. onprem.ai sells this layer as a turnkey server with the hardware included. - Agent runtime: Kortix runs the agent loop and the tools it reaches. "Run it on Kortix Cloud, in your VPC, or on your own on-prem network. Self-host is free." Each session gets "One isolated sandbox per session. Each session has its own isolated machine and branch." - Identity and keys: the directory your company already runs, pointed at the runtime, plus the model keys. Most teams own this row and forget to list it. A self-hosted stack still wants its own accounts, and [Keycloak](https://www.keycloak.org/) covers the open-source case. "Secrets are encrypted at rest with a key per project." - Approval: the rule that says which tool calls run without a person. Write it down, keep it in the repo beside the agent config, and name who may change it. - Audit: "Every action is recorded. Reading, exporting and streaming the audit log depend on the plan." Read that sentence before the first session runs; [what an audit needs](/on-prem-ai-security-and-audit) is the longer version. None of the five appears in a demo, and four of them recur every year. ## Inference-only keeps the box small An inference-only server serves tokens to applications that live somewhere else, usually on the same network. The requirements stay in the hardware half: cards, RAM, storage, a path for requests. There is no DNS to create, no certificate to renew, and no accounts to manage on the box. That is why a small company's build fits in a modest budget. The full stack pulls the rest of the rows onto the same server. The spec changes with them: ports 80 and 443 open, DNS records pointed at the machine, a box that can reach the internet. A self-hosted instance is "the full Kortix platform on a server you control: one Docker Compose stack that the `kortix self-host` CLI installs, configures and updates", and its guide states the limit plainly. "Sessions run on Platinum sandboxes, not on the server. The instance must be reachable from the internet so those sandboxes can call back." What sits in your building is the control plane, the data and the record. The machine that executes a session is managed. If the requirement is that no packet leaves, the self-hosted runtime does not meet it today. onprem.ai documents "reliable operation, even in air-gapped environments" ([onprem.ai](https://www.onprem.ai/en/platform/)). ibl.ai says its runtime "deploys inside your perimeter, on your cloud, on-premise or air-gapped, with local models if you want them" ([ibl.ai](https://ibl.ai)). Settle that constraint before the hardware, because it decides whether you are buying one box or two. ## What it costs, honestly The hardware number is the one everybody can find, and the community threads show how wide it runs. A reader in r/selfhosted priced an offline, inference-only server for a small company at "$10k–$15k USD" ([r/selfhosted](https://www.reddit.com/r/selfhosted/comments/1qboc22/)). That is one reader's budget, and it buys the small half. A thread in r/mlops takes the same question one size up, to an on-premise cluster ([r/mlops](https://www.reddit.com/r/mlops/comments/1fe5i82/)). The recurring side has a published method. A paper on arXiv sets out "a cost-benefit analysis framework to help organizations determine when on-premise LLM deployment becomes economically viable compared to commercial subscription services" ([arXiv 2509.18101](https://arxiv.org/abs/2509.18101)). Its inputs are hardware requirements, operational expenses and performance benchmarks, weighed against what the same work costs on a subscription. Put the runtime, the integration hours and the audit owner into that same calculation. A quote that stops at the cards gets reopened at the first renewal. The person reopening it is the one now on call for the serving stack. The GPU is the only line on the list you pay for once. Everything after it is a rate: power and cooling, the person who patches vLLM, the upgrade that changes behavior mid-quarter, the audit record. ## What goes in the requirements document Four sections, in this order. The box: cards, system RAM, storage, and the interconnect if the model spans nodes. The software rows, each with a named owner. The boundary: what may leave the network, whether a sandbox may call back, and which design that rules out. The break-even: the monthly token volume at which an owned box beats the current bill, read off the framework instead of the invoice. Kortix fills the runtime and the record on that list, and it does not sell GPUs. Take the model server from whoever supports the cards you can buy, then point a runtime you can read at it. [What on premise AI buys at each layer](/) sets the purchase beside this spec, and the [four layers behind it](/what-is-on-prem-ai) are worth one reading before the hardware arrives. _The spec sheet_ ## Three questions the spec sheet still has to answer ### Can I run my own AI server? Yes. A 7B or 8B open-weight model runs on one 24 GB card, and a 70B model runs on two to four 80 GB cards at FP8. What decides the build is the software around the model: the serving engine, the runtime, and the record. ### What is an on-premise AI platform? The software layer that runs AI services inside your own environment, on servers and networks your organization owns and maintains. In practice that means a model server, the runtime that runs agents, and the permissions and record around them. ### Can AI be deployed on premise? Yes, at whatever size your hardware sets. Open-weight models, a serving engine and an agent runtime all run on a server in your rack. The trade is that the model, its updates and the on-call rotation become yours. ## Start with the half you can run yourself Run the runtime on your own network, point it at the model server you already have, and keep the record on your side. [Try Kortix](https://kortix.com) · [Self-hosting guide](https://onpremaiguide.com/self-hosted-ai-agent-platform)