SkillFoxx

Spark History Server MCP

An MCP server and shs CLI for Apache Spark History Server: an agent investigates Spark jobs, stages and failures in plain words

  • 207 ★ on the repository
  • Last commit
  • Apache-2.0
MCP server

Minimal access

Why this level

  • The tools read metrics, jobs and stages of Spark applications and have no commands to run or change jobs
  • The History Server address and TLS settings live in your configuration
All reasons and checks

kubeflow/mcp-apache-spark-history-serverAuthor:kubeflow

Install

In your terminal, with SkillFoxx CLI

npx skillfoxx add mcp/spark-history-server-mcp

Detects the agents on your machine, shows the access level and pins the version.

Other ways to install

Access and checks

How we review

Minimal accessThe tool runs nothing and makes no network calls.

  • The tools read metrics, jobs and stages of Spark applications and have no commands to run or change jobs
  • The History Server address and TLS settings live in your configuration

This is third-party code. Review the repository files before installing.

What it does

The Kubeflow Spark AI Toolkit gives two interfaces to Spark History Server data. The MCP server exposes it to AI agents as tools: for questions like "why did the application fail" the agent picks tools, routes requests between servers and gathers findings. It installs with `uvx --from mcp-apache-spark-history-server spark-mcp` and connects to Claude Code with `claude mcp add`. The second interface, the `shs` CLI in Go as a single static binary, works without MCP and without a daemon: `shs apps`, `shs jobs -a APP_ID --status failed`, `shs stages -a APP_ID --sort duration`, `shs compare apps`. A ready skill is attached with `shs setup skill`. By default the server talks to a History Server at `http://localhost:18080`, and addresses are set in a config file or `SHS_SERVERS__LOCAL__URL`.

Who it is for. Data engineers and platform teams who investigate failures and slow Apache Spark jobs and want to hand the first pass to an agent or a script.

Good fit when

  • A Spark application crashed overnight and the failed jobs and the cause must be found
  • Two versions of one pipeline run with different timings and must be compared by stage
  • Data is needed by a script or CI without MCP and without a daemon process
  • Several History Servers should answer in one conversation

Not a fit when

  • Spark application history is not enabled: the server reads only History Server data
  • A thread dump of a running driver is needed: it is available only through the driver UI and the History Server answers 503
  • Running and changing Spark jobs is required: the tools read history and metrics

Example request

Find the failed jobs of application application_1234 and compare it with the previous run by stage duration.

MCP

Transport
stdio
Authentication
not required

Environment variables

SHS_MCP__TRANSPORT
The MCP transport, with the value stdio for a stdio run.
SHS_SERVERS__LOCAL__URL
The Spark History Server address, http://localhost:18080 by default.

Limitations

The MCP server requires Python 3.12 or newer and uv, and the `shs` CLI installs as a binary from the project releases (Go). Each server entry in the configuration has a fixed address, so covering all running applications needs a proxy that routes requests to the driver UI while the application runs and to the History Server afterwards. If a tool does not name a server, the first configured server that has the application is used, so name the server explicitly. For a private CA set `ssl_ca_cert`. The latest commit in the repository is from November 2026.

How to disable. Remove the `spark-history` entry from the client configuration (`claude mcp remove spark-history`) and the `shs` binary from `/usr/local/bin`. The `~/.claude/skills/spark-history.md` file created by `shs setup skill` is deleted by hand.

FAQ

How do I run the MCP server?

With `uvx --from mcp-apache-spark-history-server spark-mcp` without installing, or `uv tool install mcp-apache-spark-history-server` and then `spark-mcp`. The package is published on PyPI.

What is the difference between MCP and shs?

MCP is built for multi-step investigations by an agent, while `shs` suits commands, scripts and CI: a single static binary with no dependencies or daemon.

Where does the server connect by default?

To a Spark History Server at `http://localhost:18080`. Another address is set in a config file or the `SHS_SERVERS__LOCAL__URL` variable.

Which MCP versions are supported?

Protocol 2026-07-28 and 2025-11-25 along with earlier ones: the server negotiates the version with each client, so no configuration is needed.

How do I attach the CLI as a skill?

With `shs setup skill > ~/.claude/skills/spark-history.md`. Kiro works the same way.

Discussion

No questions or reviews yet.

MCP servers
Official

A DataHub MCP server: an agent searches tables and columns by metadata, traces lineage and finds analysts' SQL queries

MCP serverExtended access85
Official

Confluent MCP server for Kafka: topics, messages, Flink SQL, Schema Registry and connectors

MCP serverBroad access170
Official

An MCP server from AntV for generating charts and diagrams: 25+ visualization types, from lines and bars to maps, mind maps and network graphs

MCP serverExtended access4.4K

MCP server for SAP HANA and HANA Cloud: schema browsing, SQL with row caps and 34 tools, with writes blocked by default

MCP serverBroad access75
Official

Jitsu MCP configures and debugs data pipelines: destinations, streams, functions, live events and syncs.

MCP serverExtended access5.1K

A local Databricks API emulator with a 67-tool MCP server: agents run SQL and jobs without a real workspace

MCP serverBroad access148

README in short

The README explains two interfaces (the `shs` CLI and the MCP server) in a table and gives CLI installation, example commands and skill attachment. For MCP it covers installation, registration in agents, configuration and limits for live applications.

README badge

Report a problem