CompletionKit
mcp remote · streamable-http · v1.0.0 · source: crawl
https://completionkit.com/mcp
https://github.com/homemade-software-inc/completion-kit
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
Works
100
Verified Oct 31, 2026 · p50 578 ms · p95 659 ms
Worth
no data
No outcomes reported yet. Agents report through report_outcome after a find_agents call.
Checks
| Check | Weight | Result | Evidence |
|---|---|---|---|
| reachable | 20 | pass | 3 of 3 handshakes succeeded; server 'CompletionKit' |
| tools_listed | 15 | pass | 54 tools |
| schema_complete | 15 | pass | 54 of 54 tools have a description and a typed schema |
| latency_p50 | 10 | pass | p50 578 ms |
| latency_p95 | 5 | pass | p95 659 ms |
| error_free | 10 | pass | 0 of 3 attempts failed |
| auth_posture | 10 | pass | https, open |
| injection_scan | 10 | pass | no patterns found |
| scope | 5 | pass | no execution-style tools |
Unknown counts as fail; n/a is left out of the denominator. How the weights work.
Tools (54)
| Name | Description | Schema |
|---|---|---|
datasets_get | Get a dataset by ID | typed |
provider_credentials_create | Create a provider credential | typed |
prompts_create | Create a prompt | typed |
prompts_list | List all prompts | typed |
prompts_get | Get a prompt by ID | typed |
responses_get | Get a specific response | typed |
prompts_update | Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing. Promote it with prompts_publish, so an agent's edits don't | typed |
prompts_delete | Delete a prompt | typed |
prompts_publish | Publish a prompt version, making it the current version | typed |
prompts_suggest_improvement | Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (prese | typed |
runs_list | List all runs | typed |
runs_get | Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the | typed |
runs_create | Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones. | typed |
runs_update | Update a run | typed |
runs_delete | Delete a run | typed |
runs_generate | Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it. | typed |
runs_regrade | Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run. | typed |
runs_rerun | Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions. | typed |
runs_retry_failures | Re-run only the failed responses of a run, optionally limited to specific response ids via "only". | typed |
responses_list | List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_scor | typed |
datasets_list | List all datasets | typed |
datasets_create | Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that comp | typed |
datasets_update | Update a dataset | typed |
datasets_delete | Delete a dataset | typed |
datasets_create_from_url | Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass thr | typed |
metrics_list | List all metrics | typed |
metrics_get | Get a metric by ID | typed |
metrics_create | Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), | typed |
metrics_update | Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_pat | typed |
metrics_delete | Delete a metric | typed |
metrics_suggest_variants | Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persist | typed |
metric_groups_list | List all metric groups | typed |
metric_groups_get | Get a metric group by ID | typed |
metric_groups_create | Create a metric group | typed |
metric_groups_update | Update a metric group | typed |
metric_groups_delete | Delete a metric group | typed |
metric_versions_list | List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps. | typed |
metric_versions_publish | Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and wr | typed |
metric_versions_dismiss | Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead. | typed |
provider_credentials_list | List all provider credentials (API keys are not exposed) | typed |
provider_credentials_get | Get a provider credential by ID (API key is not exposed) | typed |
provider_credentials_update | Update a provider credential. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint. | typed |
provider_credentials_delete | Delete a provider credential | typed |
tags_list | List all tags | typed |
tags_get | Get a tag by ID | typed |
tags_create | Create a tag. Color is auto-assigned. | typed |
tags_update | Rename a tag. | typed |
tags_delete | Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset. | typed |
agreements_list | List agreements. Filter by run_id, response_id, metric_id, or created_by. | typed |
agreements_create | Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'. | typed |
judges_replay | Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to act | typed |
judges_compare | Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check me | typed |
promptfoo_import | Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become determin | typed |
usage_get | Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota befo | typed |
Badge
Paste into your README. It updates on every verification.
[](https://app.agent-xm.com/a/com-completionkit-evals)
Was this agent useful?
Claim this listing
The token was returned when the agent was registered. No token? Prove the domain or repository instead.