Agent Reliability documentation

Overview

Agent Reliability remembers the MCP tool contract your team approved and compares every future change against it. Install the GitHub App, register a target, and every push or pull request gets a GitHub Check that explains what changed, why it matters, and whether your deployment policy allows it.

Quickstart

Install the GitHub App from the Agent Reliability overview page and grant it access to the repositories you want covered. The first scan for a repository establishes its baseline; every scan after that is compared against it.

• Install the GitHub App: /agent-reliability
• Register at least one target (a public HTTPS MCP endpoint) for the repository.
• Open a pull request that changes your MCP server -- the Check reports the result.

Repository setup

A target is the MCP endpoint Agent Reliability scans on your behalf. Targets are public HTTPS MCP endpoints today; each repository can register multiple targets (for example, staging and production).

Baselines

Agent Reliability remembers the tool contract your team approved. Every future scan is compared against that accepted state, not against whatever the server happens to look like today. A completed, non-superseded, non-pull-request scan against your default branch that passes policy automatically promotes its snapshot as the new baseline -- pull-request scans never do, so a change under review can't silently redefine what "approved" means before it's merged.

Semantic diff

Every scan produces a semantic contract comparison, not a raw JSON diff. Changes are classified two ways: how severe (info/low/medium/high/critical) and whether they're breaking for existing callers -- these are independent, because a low-severity change can still break callers and a high-severity one can be fully backward compatible.

• Tool surface: tools added, removed, or renamed; changed descriptions.
• Capability & risk: new write capability, new destructive operation, permission expansion.
• Contract: required field added, optional -> required, field removed, type or enum change.
• Authentication: anonymous -> authenticated (or the reverse), OAuth/API-key changes.
• Transport & runtime: endpoint/transport changes, latency regression.
• Trust & evidence: Verify score or readiness regression on the underlying server.

Policies

You define what an unacceptable change looks like in a repository deployment policy. Each scan evaluates to one decision: pass, warn, approval_required, or block.

• Block new destructive tools.
• Warn on new write capabilities.
• Block breaking schema changes.
• Require a minimum Verify readiness on the underlying server.
• Block severe authentication regression.

GitHub Checks

The Check is where you see the result. It reports a title summarizing the decision and change count, a one-line summary of the most important finding, and a full Markdown body organized by severity (Critical/High/Medium/Low/Info) naming exactly which policy rule fired and what to do next.

Scheduled monitoring

Your code does not have to change for your agent dependencies to change. Once a target is registered, you can enable scheduled scans that recheck it on an interval, independent of repository activity, and alert when an upstream MCP server introduces contract, permission, authentication, or trust changes outside your repository.

Evidence

Every run produces a structured evidence artifact (JSON and Markdown) recording the diff, the policy evaluation, and both the baseline and candidate snapshot fingerprints. Evidence is hash-chain protected today. Evidence artifacts can be cryptographically signed for independent verification on supported deployments.

Security & privacy

The GitHub App requests the minimum permissions needed: Metadata (read), Checks (read/write), and Contents (read, used only to fetch a repository-committed deployment policy file when that feature is enabled). Installation access tokens are short-lived and kept in process memory only -- never persisted to application tables, logs, or evidence. Credentials for scanning authenticated targets are encrypted at rest using the same vault pattern as Verify's other stored credentials. Uninstalling the GitHub App from a repository stops all future scans for it immediately.

Quotas

Free Beta and Team are the two available plans today.

• Free Beta: 3 repositories, 5 targets per repository, 100 scans/month, concurrency 2.
• Team: 25 repositories, 25 targets per repository, 1000 scans/month, concurrency 10. Upgrade from your workspace dashboard.

Troubleshooting

A scan result is always one of three different kinds of outcome, and they are never confused with each other: a policy decision (pass/warn/approval_required/block) is a finding about your MCP server's contract; a configuration error means the scan couldn't run because of how it's set up (for example, an invalid policy file); an infrastructure error means the scan itself failed to complete. A broken scan or an infrastructure problem is reported as needing attention -- it is never presented as a policy block, so it never reads as "your MCP server is unsafe" when the real problem was on the scanning side.

API

Supported external routes. All require authentication except the public overview and install pages.

• GET /v1/agent-reliability/overview
• GET /v1/agent-reliability/dashboard
• GET /v1/agent-reliability/billing
• POST /v1/agent-reliability/billing/checkout
• GET/POST /v1/agent-reliability/repositories
• GET/PUT/DELETE /v1/agent-reliability/repositories/{id}
• GET/POST /v1/agent-reliability/repositories/{id}/targets
• PUT/DELETE /v1/agent-reliability/repositories/{id}/targets/{target_id}
• GET/PUT /v1/agent-reliability/repositories/{id}/targets/{target_id}/schedule
• GET/PUT /v1/agent-reliability/repositories/{id}/deployment-policy
• POST /v1/agent-reliability/scans
• GET /v1/agent-reliability/runs
• GET /v1/agent-reliability/runs/{id}
• GET /v1/agent-reliability/runs/{id}/evidence
• GET /v1/agent-reliability/runs/{id}/evidence.html
• POST /v1/agent-reliability/runs/{id}/baseline/promote
• POST /v1/agent-reliability/diffs
• POST /v1/agent-reliability/deployment-policy/evaluate

Get started

Install the GitHub App from the Agent Reliability overview.