RAG and RLM MCP server comparison for 1C
perform_comparison_1c_rag_mcp
A methodology and workflow for a Claude orchestrator that compares 1C code-analysis MCP servers by answer quality and token spend
Low risk
We rate an entry low when it mostly gives the agent instructions and reference material.
Why this level
- Tests third-party MCP servers against a test configuration export, not a live base
- The orchestrator only reads the tested servers' results and writes reports
Install
Manual install
git clone https://github.com/Dach-Coin/perform_comparison_1c_rag_mcp.gitNext you need to bring up Docker, LM Studio with bge-m3, and configure e2e_tests/servers.yaml from the example.
This is third-party code. Review the repository files before installing.
What it does
The project provides a two-role orchestration protocol: an orchestrator agent on Claude Opus launches parallel analyzer agents on Claude Sonnet, one per tested MCP server, each running the same business questions against a 1C configuration using only its own server, and writing a call log and a business report. The orchestrator then scores quality by a shared methodology and compiles a comparison summary of answer quality, token spend and approach type (RAG, RLM, graph). It includes a WORKFLOW.md protocol, ten test business prompts, a ready servers_example.yaml server configuration sample, and already-run comparison rounds on a 1C:Document Management CORP configuration, recorded in reports with concrete per-server results.
Who it is for. For 1C developers and teams choosing between several 1C code-analysis MCP servers who want an objective comparison rather than marketing claims.
Good fit when
- You need to compare several 1C code-analysis MCP servers on your own configuration
- You are building your own 1C MCP server and need an objective testing methodology
- You want to understand the trade-off between answer quality and token spend across approaches (RAG, RLM, graph)
Not a fit when
- You need an actual MCP server for working with 1C, not a tool to compare them
- You lack Docker, LM Studio with the bge-m3 model, and, for graph servers, Neo4j
- You are not ready to spend time running dozens of test passes
Example request
Run the analysis using the e2e_tests/WORKFLOW.md workflow, tests 1-3, for the cloud-embeddings, rlm-tools-bsl and 1c-metacode MCP serversLimitations
The ready reports in the repository are based on one 1C:Document Management CORP 3.0 configuration; results on other configurations may differ. Some compared servers (Comol) are paid and distributed only as Docker images without source. The methodology relies on the Claude Opus orchestrator's own scoring, not an independent third-party judge.
How to disable. Simply do not run the workflow; the tool installs nothing persistent beyond the test servers you yourself stand up for comparison.
Security check
- Tests third-party MCP servers against a test configuration export, not a live base
- The orchestrator only reads the tested servers' results and writes reports
README in short
The README describes the two-role orchestration model, test bench setup (Docker, LM Studio with bge-m3, Neo4j for graph servers), preparing configuration source files by exporting to XML, installing the compared MCP servers (paid ones from Comol and free open-source ones), a quick start that clears the examples and runs the analysis, result tables from two comparison rounds with concrete quality and token numbers, and the repository structure with the protocol, prompts and reports.
FAQ
Which server performed best in the included reports?
In the April round, rlm-tools-bsl, code-metadata (version 08.04) and 1c-mcp-metacode all scored the maximum 10 out of 10, with rlm-tools-bsl using the fewest tokens.
Are paid servers required for testing?
No, the example also includes free open-source servers like rlm-tools-bsl and 1c-mcp-metacode.
Related
A CLI for testing and red teaming LLM applications: compares models, runs automated CI checks and finds vulnerabilities
Playwright CLI
playwright-cli
Microsoft's official Playwright CLI with an agent skill: drive a browser through short commands without heavy MCP schemas
The official MCP server debugger: web UI, automation CLI and terminal UI in one package
SonarSource's official MCP server: quality and security issues, quality gates and code analysis from SonarQube Server and Cloud