llmman CLI
llmman runs Claude Code, Codex and OpenCode on a local or hosted model with one command and stores models as OCI images
- 582 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- Downloads and runs llama.cpp releases and containers, and the installer executes a script from the network
- The daemon listens on 127.0.0.1:17434; when exposed through `LLMMAN_HOST=0.0.0.0` every request has to carry an API key
- A hosted provider key goes only to the local daemon and is not written into the agent's settings
Install
- Works in
Manual install
curl -fsSL https://llmmanorg.github.io/install.sh | shFor Linux and macOS. The script is downloaded from the network, so read it before running. Windows has `irm https://llmmanorg.github.io/install.ps1 | iex`.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Downloads and runs llama.cpp releases and containers, and the installer executes a script from the network
- The daemon listens on 127.0.0.1:17434; when exposed through `LLMMAN_HOST=0.0.0.0` every request has to carry an API key
- A hosted provider key goes only to the local daemon and is not written into the agent's settings
This is third-party code. Review the repository files before installing.
What it does
llmman is a Rust command line tool that starts an inference server and points coding agents at it. `llmman launch claude --model qwen3.8` fetches a tested llama.cpp release for your GPU, loads the model and starts Claude Code with the right environment variables. Codex, OpenCode, GitHub Copilot CLI, Cline, Aider, Qwen Code, Gemini CLI, goose and other agents from the list printed by a bare `llmman launch` connect the same way. You install the agents yourself: llmman only starts what is already on the machine. Models are stored as OCI images. `llmman pull` fetches them from Docker Hub or straight from Hugging Face, `llmman transfer` copies a model from Hugging Face into your registry without a local copy, `llmman push` with a key signs it in cosign format and `llmman verify` checks the signature. Inference runs on unmodified llama.cpp, vLLM, SGLang or mlx-lm. `llmman serve` opens one endpoint on 127.0.0.1:17434 that speaks the Ollama, OpenAI and Anthropic APIs, so any client of those APIs works. The `--provider` flag redirects requests to a hosted provider such as OpenRouter, and the `--overflow-provider` and `--overflow-model` pair sends only the requests that do not fit the local model's context to the cloud. Several machines running `llmman serve` form a shared pool, and the node that already has the model loaded answers.
Who it is for. Developers who want to run Claude Code or Codex on their own GPU or inside a closed network, and teams whose models already live in a container registry.
Good fit when
- The code must not leave the machine, and Claude Code or OpenCode has to run on a local model
- Models sit in a private Harbor or GHCR and need cosign signing with a check on `pull`, like container images
- A laptop, a workstation and a mini server should answer as one endpoint
- One agent has to switch quickly between a local model and OpenRouter or Groq via `--provider`
- The agent has to run isolated: `--sandbox docker` allows writes only to the Git work tree and its settings directory
Not a fit when
- You want an installer wizard for agents: llmman starts installed programs only, except Cline, which it can install with npm after a confirmation
- Ollama with its model library is enough and signing or registry-to-registry transfer is not needed
- You have no GPU and want a model without setup: llmman runs inference on your machine or forwards requests to a provider and hosts no models itself
- You need a ready cloud platform with user accounting and billing: llmman stays a utility on your own hardware
Example request
Install llmman and start Claude Code on the local qwen3.8 model, and show that the answer comes from my machineLimitations
- Building with `cargo install` needs Go 1.27.3 and Rust, plus LLVM on Windows. Prebuilt binaries and package managers need none of it.
- A daemon reachable over the network accepts requests only with an API key (`LLMMAN_API_KEYS`) unless `LLMMAN_AUTH=off` is set.
- Speed and quality depend on your GPU and the chosen model: the README gives no benchmarks.
- The installer `curl -fsSL https://llmmanorg.github.io/install.sh | sh` runs a script from the network, so read it before running.
How to disable. Stop the `llmman serve` process and remove the program with the manager you used: `brew uninstall llmman`, `scoop uninstall llmman` or `cargo uninstall llmman`. llmman sets agent environment variables only for the duration of a launch. For Cline it merges an Ollama provider into `~/.cline/data/settings/providers.json`, so remove that entry by hand.
FAQ
How do I run Claude Code on a local model with llmman?
Run `llmman launch claude --model qwen3.8`. The command starts the daemon, fetches llama.cpp for your GPU, loads the model and opens Claude Code pointed at that address. A bare `llmman launch` lists the supported agents and shows which ones are installed.
How is llmman different from Ollama?
Ollama keeps models in its own layout and pulls them from its own library. llmman keeps unmodified GGUF and safetensors in the standard OCI Image Layout, pulls from Hugging Face and any OCI registry, signs models in cosign format and moves them between registries with `transfer`. Ollama clients still work, because the Ollama API answers on `127.0.0.1:17434` too.
Can only large requests go to the cloud?
Yes. Set `--overflow-provider` and `--overflow-model` next to the local `--model`: a request that does not fit the local model's context goes to the provider and the rest stay on the machine. The `x-llmman-route: local` or `cloud` header pins a request to one side.
Do I install the agents separately?
Yes. llmman starts the Claude Code, Codex, OpenCode and other agents you already have and only points them at the right endpoint. Cline is the one exception: llmman can install it with npm after your confirmation.
How do I check that a model from a registry has not been swapped?
Sign the model when you publish it: `llmman push docker.io/myorg/mymodel:v1 --sign-key signing.key`. Verify with `llmman verify docker.io/myorg/mymodel:v1 --key signing.pub`. A `[verify]` section in the configuration turns on a check at every `pull`, with a warning or a refusal.
Discussion
No questions or reviews yet.
Related
CLI toolsA local model server for Apple Silicon Macs in OpenAI, Anthropic and Ollama formats: Claude Code, Codex and OpenCode connect with one command
A vLLM-style inference server on MLX that runs language models on Apple Silicon Macs and serves them over an API
Ollama runs open models on your own machine and serves them to agents through a local API compatible with OpenAI and Anthropic.
A CLI and Python library for OpenAI, Claude, Gemini, local and other models: prompts, chats, tools, embeddings and a SQLite log
Late is a coding AI agent in one Go binary: an orchestrator plans while disposable subagents code and research, on llama-server or cloud models
LM Studio is a desktop app for running open models locally, with a chat, a local server in OpenAI and Anthropic formats, and MCP support.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| llmman (this card) | no | 582 | |
| Ollama | yes | 186,123 | keeps models in its own layout and pulls from its own library, with no signatures or registry-to-registry transfer |
| Claude Code Router | no | 38,020 | a gateway that routes between providers and does not run local models or store them |
README in short
The README opens with `llmman launch claude --model qwen3.8` and what happens behind it. Then come six install routes, a three-command quick start, image and audio generation and typed questions about text and images. Separate sections cover signed OCI models, a comparison table with Ollama, the server, machine pooling, agent launch and sandboxes.