IA-QA Eval & RAG — 38 tools
mcp remote · streamable-http · v1.0.1 · source: crawl
https://www.ia-qa.com/mcp/eval
https://github.com/ia-qa-com/mcp
LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.
Works
75
Verified Oct 31, 2026 · p50 1538 ms · p95 1583 ms
Worth
no data
No outcomes reported yet. Agents report through report_outcome after a find_agents call.
Checks
| Check | Weight | Result | Evidence |
|---|---|---|---|
| reachable | 20 | pass | 3 of 3 handshakes succeeded; server 'ia-qa-toolbox-eval' |
| tools_listed | 15 | pass | 38 tools |
| schema_complete | 15 | pass | 38 of 38 tools have a description and a typed schema |
| latency_p50 | 10 | fail | p50 1538 ms |
| latency_p95 | 5 | pass | p95 1583 ms |
| error_free | 10 | pass | 0 of 3 attempts failed |
| auth_posture | 10 | pass | https, open |
| injection_scan | 10 | fail | prompt_test_suite: references the system prompt; run_vlm_test_suite: base64 payload; token_budget_calculator: references the system prompt |
| scope | 5 | fail | broad tools: run_eval_contract, generate_eval_yaml |
Unknown counts as fail; n/a is left out of the denominator. How the weights work.
Tools (38)
| Name | Description | Schema |
|---|---|---|
needle_haystack_generate | Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystac | typed |
get_testing_guidelines | Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to l | typed |
find_tool | Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this when you are unsure which tool to use or | typed |
hallucination_check | Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its be | typed |
run_eval_contract | Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdow | typed |
run_semantic_tests | Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, | typed |
generate_ci_workflow | Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA. Two gate types, combinable: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (LLM quality gate, nee | typed |
generate_eval_yaml | Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run | typed |
metamorphic_check | Reference-free stability primitive: instead of comparing an answer to a ground truth, it checks that an assistant's answer stays INVARIANT when the QUESTION is transformed (typo, casing, paraphrase, r | typed |
test_skill | Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and others) in two stages. (1) lint: th | typed |
llm_output_validator | Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essenti | typed |
compare_responses | Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a v | typed |
analyze_responses | Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-represen | typed |
prompt_test_suite | Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluatio | typed |
llm_format_check | Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing. | typed |
consistency_check | Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitatio | typed |
llm_json_schema_check | Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing. | typed |
response_quality_score | Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with | typed |
conversation_analyze | Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA. | typed |
cot_analyzer | Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evalua | typed |
ab_test_report | Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner | typed |
similarity_score | Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs | typed |
run_vlm_test_suite | Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL ve | typed |
embedding_similarity | Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector em | typed |
rag_relevance_rank | Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API. | typed |
bm25_score | Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Ret | typed |
build_rag_prompt | Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source at | typed |
rerank_evaluate | Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels. Three modes, all keyless except the las | typed |
count_tokens | Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and | typed |
extract_json_from_text | Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an L | typed |
estimate_llm_cost | Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V | typed |
token_budget_calculator | Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models. | typed |
context_window_check | Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk. | typed |
model_info | Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anth | typed |
compare_models | Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case. | typed |
list_llm_models | List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to d | typed |
llm_generate | Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b (default "openai/gpt-oss-20b"). HuggingFace models are | typed |
rate_tool | Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your | typed |
Badge
Paste into your README. It updates on every verification.
[](https://app.agent-xm.com/a/io-github-jcjamet-ia-qa-eval)
Was this agent useful?
Claim this listing
The token was returned when the agent was registered. No token? Prove the domain or repository instead.