The self-hosted AI stack is open source up to its rules

What goes into a self-hosted AI stack

A self-hosted AI stack is four layers, each open source, on hardware you run: inference for the model, an agent runtime for the loop and its tools, data and memory for state, and policy and audit for who may act and what got recorded. The fourth is what turns a stack that answers questions into one that does work.

Most guides, and the assistants that answer these questions, reach for the same short list: Ollama or vLLM for models, Open WebUI for the chat, Dify or n8n for workflows, OpenClaw or Hermes Agent for a personal assistant. Each one is open source, and each owns a layer. Deciding which agent may touch which system, and keeping the record when it does, is left to the company running the stack.

That gap is easier to see in a file than in a paragraph. Kortix's project template declares each agent and what it may reach in one kortix.yaml, and the file sets its own default: "Grants are deny-by-default: an omitted skills, subagents, commands, extensions, connectors, secrets or kortix_permissions grant is none. Write all, none, or a list of names." The front-door agent starts with everything, and the file says so:

    prompt_file: agents/kortix.md
    # The project's own front-door agent gets everything, so a fresh project
    # is capable out of the box. Narrow it once you know what this project
    # needs.
    skills: all
    subagents: all
    commands: all
    extensions: all
    connectors: all
    secrets: all
    kortix_permissions: all

Narrowing an agent is one line in that file, in the repository the agents live in. A reviewer reads it as a diff beside the prompt it applies to. A stack assembled from four separate products has no such file: the reach of each agent sits in four configuration screens, and no one diffs them.

The four layers keep the same names on the guide's home page and in what on premise AI covers. The products speak for themselves, one layer at a time, and the fourth layer is left to the company running the stack.

Inference: three servers, one interface

Ollama describes itself as "the most popular way to build with open models", and its home page promises "complete privacy locally or in the cloud". The line that prompts "are never stored or trained on" covers the local run; a cloud model sends the prompt out. Ollama's job is to serve a model, and no agent loop, scheduler or audit record appears on its page.

vLLM is "a fast and easy-to-use library for LLM inference and serving", started at Berkeley's Sky Computing Lab, with "over 2000 contributors". Throughput comes from PagedAttention, continuous batching and prefix caching. Pick it when many requests arrive at once and one workstation no longer keeps up.

LocalAI is "a free, open-source alternative to OpenAI (Anthropic, etc.), functioning as a drop-in replacement REST API for local inferencing". It runs text, images and audio "locally or on-premises with consumer-grade hardware". Of the three, only LocalAI documents a user model: LOCALAI_API_KEY gates access, though "API keys grant full admin access with no role separation", while LOCALAI_AUTH=true adds "multi-user support with admin/user roles, OAuth login, per-user API keys, and usage tracking".

All three expose an OpenAI-compatible surface: LocalAI is a drop-in replacement, Ollama "supports a subset of the OpenAI API", and vLLM ships an "OpenAI-Compatible Server". The shared surface lets the layer above swap servers without a rewrite, so the inference choice rarely decides the rest of the stack.

Open WebUI grew a runtime; the rules stayed yours to write

Open WebUI calls itself "a self-hosted AI platform that's extensible, feature-rich, user-friendly, and built to run entirely offline", with "support for Ollama and OpenAI-compatible APIs". That sentence undersells what its own docs describe today. "Open Terminal gives an agent a real terminal and filesystem, so it can carry out multi-step work instead of only describing it", and "Open WebUI Computer is the cutting-edge agent harness and runtime for your real machine", with "approvals and plan mode" and "scheduled agents that report back to you". For teams, "Terminals is the enterprise layer on top, giving every user an isolated environment with resource limits and automatic lifecycle management."

So the chat page is a runtime, and calling it a chat page is out of date. What its pages do not describe is a file in your repository listing which agent may reach which connector, and an action record one reviewer can read across every person's agents. A Terminals policy is a workspace profile: "It controls the image, tools, resources, storage, environment variables, security context, and idle timeout." It does not say what a named agent may call, or keep the receipt.

Two products the assistants name most sit in the same layer. OpenClaw is "an open-source AI assistant that runs on your own computer" and meets people in Discord, iMessage, Slack, Teams, Telegram, WhatsApp "and 20+ more". "One Gateway runs it as a personal assistant on a laptop or as a shared team deployment; configuration is the only difference." State, memory and credentials "live on your hardware", models and agent harnesses are plugins, and the OpenClaw Foundation stewards it with "no paid tier, hosted service, or token". Hermes Agent is "the open-source, self-hosted AI agent by Nous Research", with persistent memory, auto-generated skills, a messaging gateway over six channels and five sandbox backends: local, Docker, SSH, Singularity and Modal.

Each of the three ships some control. Open WebUI has approvals and plan mode, OpenClaw's docs go as far as policy as code and a durable audit ledger, and Hermes documents the sandboxes its agents run in. The difference is where the rules live and who can change them.

Orchestration: workflows and agent loops

Dify is "an open-source platform for building AI applications". It builds agents, agentic workflows and chatbots over your own data and publishes them as web apps or APIs, and its Community Edition self-hosts through Docker Compose. n8n runs workflows: "You can self-host n8n on your own infrastructure, on-premises, or in a private cloud, using Docker Compose, one-line setup, or other deployment methods". It adds that "All self-hosted installations use the same core product." LangGraph is "a low-level orchestration framework and runtime for building, managing and deploying long-running, stateful agents", and its own docs call it "very low-level, and focused entirely on agent orchestration."

Orchestration answers what should happen next in a workflow. On its own it does not answer whether this person's agent may call this tool with this argument, or where that rule is written down. A workflow that can send email on a schedule can send the wrong email on a schedule.

Where the rules live, and what records a call

Kortix's rules sit in the project's kortix.yaml, in the same repository as the agents they apply to, and the audit page reads that file rule by rule. The question narrows to which of these products ships a place to write the rule and a record of the call.

The answers differ by product, and the differences matter more than the labels. Open WebUI's approvals sit in a chat, and its Terminals policies govern a workspace rather than a named agent. OpenClaw documents a control story of its own: policy as code, exec approvals bound to the exact command, working directory and environment hash, and a durable audit ledger. LocalAI puts the control at its API, where a single key is admin and LOCALAI_AUTH adds roles and per-user keys. Ollama and vLLM serve models and describe nothing above that, and Dify, n8n and LangGraph describe the run rather than the permission behind it.

Kortix puts the rules and the record in the repository a company already owns. "Per-resource permissions for people and agents. Roles, groups, and an audit trail." "Approval gates you set. Off until you set them", so a rule has to exist before a call is held. "Every action is recorded. Reading, exporting and streaming the audit log depend on the plan." And the work arrives the way code does: "Session work reaches main through a change request. Merge is default-deny for agents." A grant widened by an agent then shows up as a diff a person merges, not as a setting somebody flipped at two in the morning.

That is the layer the other products give you a piece of. Owning the pieces is not the same as owning the file they sit in.

Kortix is the row that carries the agents and the rules together, and it is the one a review asks about after the demo. A stack can run entirely on your hardware and still have no answer to who allowed a call.

Where the stack is the wrong pick

A self-hosted Kortix is "the full Kortix platform on a server you control: one Docker Compose stack that the kortix self-host CLI installs, configures and updates." With the CLI installed, three commands finish the setup on a Linux server with a public IP:

kortix self-host configure
kortix self-host start
kortix self-host doctor

The first writes the configuration, the second applies it, and the third checks the instance and lists what fails. What the commands do not do is move execution onto that server: "Sessions run on Platinum sandboxes, not on the server. The instance must be reachable from the internet so those sandboxes can call back."

So part of the stack stays outside your network, and the self-hosting guide says so on the same page. A company whose rule is that nothing executes outside its own perimeter should look elsewhere today. onprem.ai sells "the complete AI stack including hardware" with "reliable operation, even in air-gapped environments". ibl.ai builds agents on a runtime that "deploys inside your perimeter, on your cloud, on-premise or air-gapped, with local models if you want them", and its engineers work alongside your team. For a turnkey GPU box rather than a software stack, onprem.ai, Dell and Supermicro sell one.

Kortix is the right pick for a different requirement: an open-source runtime and a policy layer you run yourself. The model comes from whichever provider a project connects, and execution runs on managed sandboxes, said plainly. The boundary in full covers the questions that decision raises.

The order the stack goes up

1. Hardware first. GPU memory sets the model size, and the rest of the box follows from it. On prem AI server requirements covers the sizing and the recurring costs.
2. Inference second. Point the server at local weights or at a hosted model, which is a cost decision before it is a technical one. On prem LLM vs API is that trade.
3. Runtime third. A single machine holds the control plane, the database, the policy file and the chat page. The DNS records and ports come before the first session, because the sandbox needs a public API to call back to.
4. Policy fourth, before the first workflow. Write the grants and the rules, decide which tools stop for a person, and read the grant lines in the manifest before anyone outside the team gets an account.
5. The first workflow last. One connector, one trigger, one change request you approve, then widen.

Two things still leave the machine on a stack built this way: the model call goes to whichever provider a project connects, and the execution sandbox calls back in. Everything else, including the policy, the permissions and the record, sits on the server you installed. What an audit needs follows that record rather than the network diagram.