vllm-mlx: local LLM server for Claude Code on Mac
vllm-mlx
A vLLM-style inference server on MLX that runs language models on Apple Silicon Macs and serves them over an API
- 1,641 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- The server opens a port on the machine and accepts requests with no real key, so keep the address 127.0.0.1
- An agent pointed at a local model still runs commands and edits files with your rights
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/vllm-mlxDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- The server opens a port on the machine and accepts requests with no real key, so keep the address 127.0.0.1
- An agent pointed at a local model still runs commands and edits files with your rights
This is third-party code. Review the repository files before installing.
What it does
vllm-mlx runs language models on Macs with M1–M5 chips through MLX and exposes them over a network API. One process answers OpenAI routes (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/rerank) and Anthropic routes (/v1/messages with streaming and tool calls). This lets you point agents at a local model without changing their code. For agent load there is continuous request batching and a paged KV cache with prefix sharing that can spill to SSD. The server parses tool calls in 19 formats and produces structured output by JSON Schema. It works with image, audio and video models, recognizes speech through Whisper and synthesizes it, and exports metrics to Prometheus. The `vllm-mlx bench-serve` command measures load and writes results to CSV, JSON or SQLite.
Who it is for. For owners of Apple Silicon Macs who want to run Claude Code, Codex or OpenCode on local models and stand up a shared API for them.
Good fit when
- Run Claude Code on a local Qwen or Llama model without calling a cloud API
- Stand up one address that Codex, OpenCode and OpenAI SDK apps all connect to
- Handle several requests in parallel with continuous batching and a shared prefix cache
- Get a strict JSON Schema answer from the model for a script or agent
- Measure speed and memory use of models with bench-serve before choosing
Not a fit when
- The computer runs Windows, Linux or an Intel Mac: the server works on Apple Silicon only
- You need the largest cloud models: local models on a Mac are weaker at coding
- You want one-click model setup with no command line: an app such as Ollama or LM Studio fits better
Example request
Start vllm-mlx with the Qwen3-8B-4bit model on port 8000 and run Claude Code on that local model.Limitations
The server runs only on Macs with Apple Silicon and uses Metal GPU kernels through MLX. Speed and model size are limited by the unified memory of the machine: a model of about 30 billion parameters in 4-bit form needs around 18 GB.
The Python openai client is not part of the package and is installed separately. Speech synthesis in languages other than English needs espeak-ng on macOS.
Models download from Hugging Face, so test downloading weights in advance. The agent test results apply to the stated client versions and settings.
How to disable. Stop the server with Ctrl+C, remove the package with `python -m pip uninstall vllm-mlx` and unset ANTHROPIC_BASE_URL in the environment. Downloaded weights sit in the Hugging Face cache and are deleted separately.
FAQ
How do I run Claude Code on a local model through vllm-mlx?
Start the server: `vllm-mlx serve mlx-community/Qwen3-8B-4bit --port 8000`. In another window set `export ANTHROPIC_BASE_URL=http://localhost:8000` and `export ANTHROPIC_API_KEY=not-needed`, then run claude. The server answers on the /v1/messages route.
How is vllm-mlx different from Ollama?
vllm-mlx is built for Apple Silicon on MLX and aimed at agent workloads: parallel requests and two APIs at once, OpenAI and Anthropic. Ollama is more general and easier to install.
Which agents are tested with vllm-mlx?
OpenCode, pi, Codex, Claude Code, GitHub Copilot CLI, Cline CLI and the OpenClaw embedded agent. All seven completed a streamed tool call and an exact file edit on a 4-bit Qwen 27B model in a check dated September 19, 2026. The version matrix is in docs/guides/client-acceptance.md.
Which computers does it support?
Macs with an Apple Silicon chip: M1, M2, M3, M4 or M5. The server uses unified memory and Metal kernels, and no model conversion is needed.
How do I turn on continuous batching?
Add the `--continuous-batching` flag to the serve command. Cache options are covered in docs/guides/continuous-batching.md, and `--ssd-cache-dir` spills the prefix cache to SSD.
Discussion
No questions or reviews yet.
Related
CLI toolsA local model server for Apple Silicon Macs in OpenAI, Anthropic and Ollama formats: Claude Code, Codex and OpenCode connect with one command
A local model server for Apple Silicon Macs: Qwen 3.8 and Gemma 4 with MTP decoding and OpenAI and Anthropic APIs for Claude Code, Codex and Cursor
A CLI and Python library for OpenAI, Claude, Gemini, local and other models: prompts, chats, tools, embeddings and a SQLite log
LM Studio is a desktop app for running open models locally, with a chat, a local server in OpenAI and Anthropic formats, and MCP support.
The Apple Intelligence model built into a Mac as a terminal command and a local OpenAI-compatible server: no keys, no cloud
A local-first AI agent running on your own machine via llama.cpp that drives a browser, edits files, runs approved commands and remembers context across session
Comparison with similar tools
Entries that share the main search query, and how they differ.
| Entry | Official | Repository stars | Main difference |
|---|---|---|---|
| vllm-mlx: local LLM server for Claude Code on Mac (this entry) | no | 1,641 | |
| mlx-serve | yes | 2,151 | also a local Mac server with OpenAI, Anthropic and Ollama APIs; vllm-mlx adds continuous batching and a paged KV cache |
| Ollama | yes | 185,242 | general model serving across systems, while vllm-mlx works on Apple Silicon only and targets agent workloads |
README in short
The project calls vllm-mlx a vLLM-style server for Apple Silicon. Measurements on an M4 Max range from 205 tokens per second on a small model to 127 on a 30-billion-parameter model. Whisper runs dozens of times faster than real time.