Ostra in your coding agent
Connect Claude Code, Cursor or VS Code to the Ostra MCP server, then teach your agent when to triage a failure, pick impacted tests, check selectors and ask Ostra Pro — with ready-made CLAUDE.md and Cursor rules.
With the Ostra MCP server connected, your coding agent can ask Ostra the same questions your pipeline does, while it works: is this failure flaky or real? which tests should I run for this change? will this locator survive? how do I fix this test? Tool calls are billed and rate limited exactly like the REST API.
1. Connect the server
Use a Full access key (read-only keys can list the tools but not call them). Keep the key in the OSTRA_API_KEY environment variable.
claude mcp add --transport http ostra https://ostra.testrelic.ai/mcp --header "Authorization: Bearer $OSTRA_API_KEY"On Windows PowerShell use $env:OSTRA_API_KEY. Keep the command on one line. Check with claude mcp list: ostra should show ✓ Connected.
Ask the agent: "Call ostra_list_models and tell me the Ostra Decision release." If it answers, you're connected. Other clients and troubleshooting: MCP server.
2. Tell the agent when to use each tool
Agents use tools better when the project says when to. Add a short section to your agent instructions.
## Ostra (testing models, via the `ostra` MCP server)
- When a test fails, call `ostra_triage_failure` before changing code. Pass the test id, the raw
error message, `framework`, the failing `action`, and any run history you can find
(`memory.runs_50`, `failed_count_50`, `flake_rate_50`). Then:
- `flaky` / `likely_flaky`: don't change app code; suggest a retry or quarantine.
- `product_bug`: investigate the application code, not the test.
- `test_bug` / `selector_drift`: fix the test or the locator.
- `environment`: fix the setup (browsers, dependencies, ports), not the code.
- `defer`: say the evidence is inconclusive and show what you found.
- Before running the whole suite after a change, call `ostra_select_impacted_tests` with the diff
stats and the test files as candidates; run `output.impacted` first.
- When adding or changing locators, call `ostra_check_selector`; replace any that come back `breaks`
with role or test-id locators.
- For hard test-writing or failure-analysis questions, ask `ostra_pro_chat`, passing the relevant
code and logs in the prompt and the same `session` value for follow-ups on the same problem.
- Don't call Ostra tools in loops over unchanged inputs: identical requests within 24 hours return
the cached answer.Example workflows
"The checkout test is failing again. What's going on?"
The agent reads the last failure, calls ostra_triage_failure, gets flaky at 83%, and suggests quarantining the test and stabilising the wait instead of editing the checkout code.
"I changed the coupon service. Which tests should I run?"
The agent summarises git diff --numstat, lists the spec files as candidates (with co-change rates from git log if it can), calls ostra_select_impacted_tests, and runs the two checkout specs it returns.
"Write a Playwright test for password reset."
The agent asks ostra_pro_chat with the page's markup and your existing test style, writes the test, then calls ostra_check_selector on the locators it used and swaps any that come back breaks.
Cost and control
- Every call shows up in the Console under MCP servers → Recent tool calls and Logs (channel MCP), with its cost and the client that made it.
- Owners, admins and developers can turn any tool off for the workspace under MCP servers → Tools.
- Give agents their own key with a monthly spend cap, so an agent stuck in a loop can't spend more than you allow. See API keys.
Related
- MCP server and each tool's reference:
ostra_triage_failure,ostra_select_impacted_tests,ostra_check_selector,ostra_pro_chat. - MCP servers in the Console.
Selector checks
Keep a manifest of the locators your tests depend on, check them against a preview build, record their history per release, and fail the pull request when Ostra Decision predicts one will break.
SDK clients
Use Ostra Pro from the OpenAI and Anthropic SDKs, LangChain, the Vercel AI SDK and any tool with an OpenAI-compatible base URL — base URLs, keys, sessions, streaming and cold starts.