Changelog

tr_diagnose_run could list a different number of failures from the run's own total, which it prints right beside them. It now matches.

• A test that passed on retry isn't a failure. Each test is counted once, by its final attempt, so a test that failed and then passed is flaky, as the run says, not failed.
• No failures go missing on large runs, and steps with no test attached are no longer counted as extra failures. A failure past the run's first 1,000 steps is counted with its message, but without a stack trace.
• The same list feeds the other tools that read a run's failures: tr_compare_runs, tr_search_failures, tr_ai_rca, tr_create_jira, tr_heal_run and tr_replay_failure.

Released as @testrelic/mcp 3.4.7.

tr_dry_run_test type-checks a generated test before you commit it. In a project on TypeScript 6 or 7 with a tsconfig.json, it answered FAIL for every file, valid or not. It now gives a real verdict: a valid test passes, and a broken one fails with its own compiler error.

• It uses the TypeScript you already have. The compiler is found in the project the server is started in, or failing that in a global npm install, and nothing is downloaded. Start the server in your project directory, with TypeScript installed (npm i -D typescript). A global TypeScript behind a pnpm, asdf or Volta wrapper script isn't found.
• Results change on TypeScript 5. Every version now checks with the same options: strict, ES modules, an ESNext target and bundler module resolution. Your project's tsconfig.json is not applied. On TypeScript 5, modern code such as .at(), replaceAll and top-level await now passes, while strict-mode errors (a value that may be null or undefined, an untyped parameter) and import x = require() now fail.
• NO VERDICT instead of a false FAIL. When the check can't run (no TypeScript found, the compiler won't start, or it runs past the timeouts.analysis setting, 30 seconds by default), the result reads "NO VERDICT" and status is unavailable, so a file that was never checked isn't reported as broken. ok is still false in that case: read status to tell NO VERDICT from FAIL. The hosted server at mcp.testrelic.ai has no TypeScript and none of your project's dependencies, so it answers NO VERDICT: type-check generated tests with the server running locally.
• It runs on Windows, where it answered FAIL for every file on any TypeScript version.
• Cypress specs can pass. cy, describe and it are recognised when Cypress is installed. A *.cy.ts file is always checked as Cypress. For a Cypress spec with another name, pass framework: "cypress", as tr_generate_test now says in its next step.
• More in the result. Next to ok, the structured result has status (pass, fail or unavailable) and, when a compiler was found, typescript_version. Compiler output in the structured result is capped at 32,000 characters, with truncated: true when it was cut.

Released as @testrelic/mcp 3.4.6.

If you run @testrelic/mcp yourself, call these tools from scripts, or read their structured results, check the following before upgrading to 3.4.5.

• Node 20.19 is now the minimum (it was 20.0). Node 22.12 or later is recommended: on Node 20 the on-disk cache falls back to memory. The hosted server and the Docker image run on Node 24.
• tr_create_jira: project_key no longer defaults to ENG. Pass it to choose the Jira project, or leave it out to use the project your Jira integration is set up with. priority P1 to P4 becomes Highest, High, Medium or Low. The result shows the ticket's key, summary, status, priority and link instead of "undefined", and closed tickets are no longer treated as duplicates.
• tr_production_signal reports a real peak. The peak is the busiest minute, in lines per minute, not a percentage, and no peak is shown when nothing matched. time_range takes minutes, hours, days or weeks (30m, 24h, 7d, 2w). A total that reached the 500-line limit shows as "500+", and the label "Total errors" is now "Matching lines". In the structured result, read peak_per_minute. error_rate_peak is still present but is always 0.
• tr_user_impact only searches logs when you ask. Pass the new log_query (LogQL) to add a log signal covering the time since the run started. Without it no log search is made, where a fixed query used to run every time. The Amplitude figure now covers the run's own dates and is labelled as error events, which is what it counts.
• tr_risk_score and tr_analyze_diff can answer UNKNOWN. A project with no coverage data used to get a risk level anyway, usually LOW, which read as an all-clear. risk_level can now be unknown. In that case risk_score is 0, which means no data, not low risk: check risk_level first.
• Coverage tools say when there is nothing to measure. For a project with no coverage map, tr_test_map, tr_coverage_gaps and tr_coverage_report answer "Coverage not available" instead of 0% and every journey listed as a gap. tr_coverage_report adds coverage_available: false in that case.
• tr_heal_run needs your test's source to return a patch. Pass the failing test file's current contents as source and its path as file_path. The patch comes back as a diff, and nothing on disk is changed. Without source the tool returns the error and stack trace and asks for it.

Released as @testrelic/mcp 3.4.5.

Several MCP tools returned placeholders, blanks or misleading numbers. Each now returns the real answer, or the platform's own reason for not having one.

• Errors say why. When the platform explains a failure (sign-in, not-found and server errors), the tool includes that explanation instead of a bare status line. A missing endpoint or resource gives a clear error rather than "Cannot read properties of undefined", and a feature the platform doesn't offer yet reads "This feature is not available on the platform yet." Five failed lookups in a row, such as a mistyped run ID, no longer block every other tool for 30 seconds.
• tr_ai_rca and tr_suggest_fix stop returning placeholders. The 0%-confidence result that read "not yet available" is gone. tr_ai_rca falls back to an analysis of the run's failures written by your assistant, marked "(sampled fallback)". This needs an assistant that supports MCP sampling. Without it the result reads "RCA not available." tr_suggest_fix returns the reason it has no suggestion.
• tr_recent_runs pages. Passing the cursor from one page returns the next one, not the first again.
• tr_ask_ai says why an answer stopped, as "Ask AI stopped:" followed by the reason, instead of a blank reply.
• tr_replay_failure gives a command you can run for the run's framework: npx playwright test -g, npx jest -t or npx vitest run -t with the test title quoted, or npx cypress run --spec with the spec file.
• tr_user_journeys lists the paths your tests walk, with how many times tests took each one and the share that passed, in place of placeholder rows. They come from test runs, not user analytics.
• Failure text is clean. Terminal colour codes and a doubled "Error: Error:" no longer appear in failure messages, and error_type is the error's own type, such as TimeoutError.
• Smaller fixes. tr_ai_usage no longer shows a budget of 0 when the platform doesn't report one, tr_marketplace_validate gives the reason credentials were rejected, and tr_flaky_audit no longer fails when the platform returns no scores.

Released as @testrelic/mcp 3.4.5. The same release raises the minimum Node version and changes what some tools take and return, covered in its own entry for this date.

Calling tr_plan_test and then tr_generate_test now works with the parameter names the tools return, and the tools answer straight away with assistants that don't support MCP sampling.

• tr_generate_test accepts what tr_plan_test returns. Passing cache_key, file_path and framework used to fail with "No plan found". cache_key now works like plan_cache_key, and file_path like file_name. Any directory in file_path is ignored: the file is still written to the generated folder of the output directory. framework (playwright, cypress, jest or vitest) overrides the plan's framework. The older names keep working.
• tr_plan_test names the key to pass next. Its last line now gives the exact plan_cache_key for tr_generate_test.
• No more dropped connections. MCP sampling is the server asking your assistant's model to write text. With an assistant that doesn't support it, the tools that rely on it (tr_plan_test, tr_generate_test, tr_generate_assertion, tr_heal_run and tr_suggest_locator) could wait a minute for a reply that never came, and on the hosted server the connection was lost. They now fall back to their built-in result straight away. Assistants that do support sampling are asked as before.
• tr_diagnose_run always returns the same fields. Every result now has run, failures and flakiness, where a missing run or an all-passed run used to leave some out. If you checked for a missing failures or flakiness, check for an empty list instead.

Released as @testrelic/mcp 3.4.3, the first npm release since 3.4.1. If tr_generate_test still answers "Output validation error", your assistant is running an older copy: start it with npx -y @testrelic/mcp@latest.

If you run @testrelic/mcp as an HTTP server on anything other than localhost, upgrading from 3.4.1 changes how clients connect. The default npx setup over stdio and servers bound to localhost need no token, as before. The cache key and rate limit changes below apply to them too.

• Requests to /mcp need a token. When the server listens on a non-loopback address, such as --host 0.0.0.0 or the Docker image, every request must send Authorization: Bearer <token>, using the token the server itself is configured with (--token, TESTRELIC_MCP_TOKEN or a saved login). A request without it gets 401 with the code UNAUTHORIZED. The token is checked with TestRelic Cloud when a session opens, so one the platform rejects gets 401 too. /healthz stays open.
• The server won't start without one. Beyond localhost with no token configured, startup stops and says why, unless the server is multi-tenant (below). If a proxy in front of the server already authenticates callers, set TESTRELIC_MCP_REQUIRE_AUTH=false.
• New: one server for several people. On a server that requires a token, optionally set TESTRELIC_MCP_MULTI_TENANT=true, and each caller sends their own tr_mcp_ token instead of the server's. Tools then see that caller's organization and repositories, and cached results are kept apart per caller. The hosted server at mcp.testrelic.ai works this way.
• The default repository no longer applies on a server that requires a token. --default-repo-id and TESTRELIC_DEFAULT_REPO_ID are ignored there, so pass project_id when the token can see more than one repository.
• Clients that send an Origin header are no longer refused by a server reachable beyond localhost. It answered 403 "Invalid Origin header" to browser-based and editor clients. A localhost-only server still checks Origin, and the Host check is unchanged: behind a proxy, list your public hostname in TESTRELIC_MCP_PUBLIC_HOSTS.
• Saved cache keys from 3.4.1 stop working, on any transport. A cache_key now belongs to the caller it was issued to, and tr_fetch_cached answers that an older key "was not issued to this session". Run the tool that issued it again instead of reusing a stored key.
• Requests are rate limited. /mcp allows 600 requests a minute per connecting address. Beyond that the answer is 429 with the code RATE_LIMITED. Behind a proxy the server sees one address, so all callers share the limit.

Released as @testrelic/mcp 3.4.3. To create a tr_mcp_ token, see MCP authentication. To use the hosted server instead of running your own, see the MCP quickstart.

tr_replay_failure failed with "Cannot read properties of undefined (reading 'map')" on runs with a failed test, so no replay plan came back. It now returns the plan.

• The plan lists the failing test's own files. Under "Artefacts" you get links to up to 10 of the files uploaded for the test being replayed, such as its trace, video and screenshots, and never another test's recording. They are download links that expire. Call the tool again for fresh ones.
• A test with no uploaded files says so. The plan still comes back, with "No uploaded artefacts for this test." in place of the list.
• A run with no commit recorded still gets a plan. The checkout step says the run has no commit instead of printing an empty git checkout, and a run whose details can't be loaded no longer fails the whole call.
• Also in this release: tr_generate_test writes a spec that parses even when a test name or step contains quotes, backslashes or line breaks.

Released as @testrelic/mcp 3.4.4.

• tr_diagnose_run listed every failure with a blank test name, an error_type stuck on "Error", an empty message, and zero duration — the count was right, so it looked like a working tool returning useless data. That's fixed.
• Timed-out tests were previously dropped entirely; they're now counted and correctly inferred as TimeoutError.
• Failures now collapse per test instead of per timeline step, so one failing test no longer produces multiple duplicate "failures."
• This also fixes tr_compare_runs and both healing tools.

Released as @testrelic/mcp v3.4.1.