SkillFoxx

Inference Autopilot skill

Inference Autopilot tunes SGLang launch parameters for your GPU and workload and returns a launch command backed by benchmarks.

  • 142 ★ on the repository
  • Last commit
  • Apache-2.0
Skill

Extended access

Why this level

  • A run starts SGLang servers and loads the GPUs, which takes a lot of time and resources.
  • Conclusions about the best configuration must be validated on your own workload before production.
All reasons and checks

rednote-machine-learning/inference-autopilotAuthor:rednote-machine-learning

Install

In your terminal, with SkillFoxx CLI

npx skillfoxx add skills/inference-autopilot

Detects the agents on your machine, shows the access level and pins the version.

Other ways to install

Access and checks

How we review

Extended accessThe tool runs code or makes network calls and only reads external data.

  • A run starts SGLang servers and loads the GPUs, which takes a lot of time and resources.
  • Conclusions about the best configuration must be validated on your own workload before production.

This is third-party code. Review the repository files before installing.

What it does

Inference Autopilot (`inferopt`) is a CLI and agent skill for optimizing SGLang inference on one or several nodes. The user supplies a model, GPUs, a representative workload, latency SLOs and an experiment budget. The tool checks feasibility, measures a baseline, captures an Nsight Systems profile and finds the bottleneck. It then selects mechanisms and parameter values, and confirms the winner with ABBA windows and Bayesian checks. The output is `report.md`, `final.json` and a ready launch command, or a decision to keep the baseline. `SKILL.md` walks the agent through `inferopt init`, `doctor`, `plan`, `run` and `report`.

Who it is for. MLOps engineers and teams who deploy LLMs on SGLang themselves and want to change parameters from measurements rather than guesses.

Good fit when

  • A new model is deployed on your own GPUs and the starting SGLang parameters are unclear.
  • p99 latency exceeds the SLO and the cause is unknown.
  • After an SGLang upgrade the parameters must be rechecked for the current version.
  • An agent drafts the experiment plan and monitors the run without deciding for a human.
  • The optimization result is needed as a reproducible report with logs and a profile.

Not a fit when

  • There is no local SGLang or compatible model.
  • AMD cards must be tuned automatically: profiling and tuning for them are not implemented yet.
  • Production rollout is required: rollout orchestration and Kubernetes or Slurm adapters are not included.

Example request

Prepare an inferopt task for a model on two GPUs with a p99 TTFT SLO of 500 ms, run `doctor` and `plan`, and show the plan before any experiment starts.

Limitations

Needs Python 3.9 or newer, a local SGLang, a compatible model and NVIDIA GPUs with Nsight Systems (`nsys`) on `PATH`. Automatic AMD profiling and tuning are not implemented. The result applies only to the recorded model, SGLang version, hardware, workload and budget, not universally. Precision-changing candidates are opt-in and need separate quality evidence. Run artifacts hold model paths and workload details, so keep the directory private.

How to disable. Remove the skill from the agent's skills directory and the package with `pip uninstall`. Clean the run output directories by hand.

FAQ

Is an agent required?

No. The CLI is fully standalone; the `SKILL.md` skill only helps collect inputs, review the plan, monitor the run and explain evidence.

Which optimization goals exist?

`online_latency` looks for a configuration within the SLO, `offline_throughput` maximizes throughput, with optional latency or error limits.

How long does an experiment take?

It depends on intensity: `fast` defaults to 24 trials, `balanced` to 40, `max` to 96. All modes share the same acceptance gates.

Is `inferopt run` safe to start?

`doctor` and `plan` are read-only. `run --yes` starts only processes of the current experiment, installs no packages, leaves drivers and weights alone and kills no foreign processes.

Does it support several nodes?

Yes. Run one process per node through `torchrun` or a scheduler, and the report emits one command per node.

Discussion

No questions or reviews yet.

Skills

A GPU profiler for AI agents: the agent runs vLLM or SGLang under graphsignal-run, reads the metrics and tunes flags for speed

SkillExtended access276

Skill that powers AutoDL instances on and off from a plain-language request, keeps a budget and shuts down idle machines.

SkillBroad access95

A CLI and skill for the Inspire compute platform: an agent creates notebooks, training jobs and services without web clicks.

SkillExtended access563
Official

Agent skills for running, benchmarking and profiling Hugging Face models on Intel Arc GPUs, plus CUDA to XPU code migration.

PluginBroad access51

Glyd’s CLI runs language models on NVIDIA GPUs with compressed weights and cache to reduce the memory needed for inference.

CLIExtended access200

The official Vast.ai skill: an agent searches GPU offers, rents instances and manages them through vastai.

SkillBroad access231

README in short

The tool stores no fixed best configurations, only a process: rules are selected from live deployment evidence, and each candidate passes compatibility, benchmarking and statistical checks.

README badge

Original SKILL.md

---
name: inference-autopilot
description: Analyze, benchmark, diagnose, and optimize large-model inference deployments from hardware inventory, model details, workload traces, and latency or throughput SLOs.
---

Report a problem