TestRelic now speaks Python: pytest, Playwright, DeepEval, and Appium analytics
Today TestRelic speaks Python. We're shipping four pytest-native packages — testrelic-pytest, testrelic-playwright, testrelic-deepeval, and testrelic-appium — that capture the same rich test analytics our JavaScript reporters do, and stream them to the same cloud workspace. Each one installs from PyPI, activates through pytest's plugin system with no conftest.py wiring, and starts uploading on your next pytest run.
Why Python, and why now
Half the test suites we get asked about are Python. API teams live in pytest + httpx. Browser teams have moved to pytest-playwright. The LLM teams run deepeval. Mobile teams drive Appium from the Python client. Until now, all of them had to choose between rich local reports and TestRelic's cloud — because the cloud only understood our Node SDKs.
So we ported the whole thing. Not a thin shim — a field-for-field port of the analytics model, so a Python run opens the exact same Session Workspace, AI Insights, and history views as a JavaScript run. Pick the package that matches your stack and you're done. The new Python SDK section of the docs walks through each one.
pytest: zero-config analytics for any suite
testrelic-pytest is the generic reporter. It works with any pytest project — unit, integration, or API — and the headline is that it needs no configuration at all:
pip install testrelic-pytest
export TESTRELIC_API_KEY=tr_live_...
pytest
That's the whole setup. The plugin auto-registers, captures every test's outcome, markers, captured output, and CI/git metadata, and uploads the run.
The part we're most excited about: automatic HTTP API detection. If a test makes httpx or requests calls, TestRelic classifies it as a rest API test and opens it in the API Session Workspace instead of the web view — capturing request and response headers and bodies, with sensitive headers masked. Real assert statements are recorded too, with the source expression and the evaluated values, so a failing check reads like this in the Assertions tab:
response.status_code == 201
→ 200 == 201 (failed)
No fixtures, no decorators. Plain pytest tests just become observable. There's more in HTTP API Auto-Capture.
Playwright, ported to Python
testrelic-playwright is a full port of @testrelic/playwright-analytics (schema 1.3.0 — field-for-field parity). It's a pytest plugin for pytest-playwright that captures navigation timelines, network stats, action steps, console logs, expect() assertions, screenshots, and video — then renders a single-file interactive HTML report and uploads to the cloud.
pip install pytest-playwright testrelic-playwright
playwright install
pytest
Output lands in test-results/analytics-timeline.json and .html. For large suites it streams to disk automatically (configurable via reportMode and streamingThreshold), and the bundled testrelic-playwright serve command hosts the report locally. BrowserStack and LambdaTest runs are detected automatically. Full reference: Configuration and Get Started.
LLM evaluations with DeepEval
testrelic-deepeval captures DeepEval evaluation results and uploads them to your TestRelic org instead of Confident AI. The pytest plugin captures DeepEval's in-memory run at session finish — no code changes — and there's a drop-in evaluate() wrapper for programmatic runs:
pip install testrelic-deepeval
testrelic login
deepeval test run tests/
Eval runs land in the new LLM Evaluations workspace, render through the same Run Detail and Session Workspace as everything else, and track an Eval Stability metric over time. We wrote a whole post on this — see Own your LLM evals.
Mobile testing with Appium
testrelic-appium (Beta) brings analytics to Appium + pytest. It auto-wraps any Appium WebDriver your test creates and records a full command timeline, Android logcat / iOS syslog, CDP network capture, screenshots, and video — with a named-assertion helper:
from testrelic_appium import assert_that
with assert_that("toHaveText", expected="Welcome"):
assert el.text == "Welcome"
Pure-Selenium tests are never touched. See Get Started with Appium (Python).
One cloud, every language
| Package | Install | Tests it covers |
|---|---|---|
testrelic-pytest | pip install testrelic-pytest | Any pytest suite; auto-detects API tests |
testrelic-playwright | pip install testrelic-playwright | Playwright Python browser + API tests |
testrelic-deepeval | pip install testrelic-deepeval | DeepEval LLM evaluations |
testrelic-appium | pip install testrelic-appium | Appium native mobile tests |
All four share the same conventions: set TESTRELIC_API_KEY, runs go to https://platform.testrelic.ai, failed uploads queue locally and replay with a drain command, and CI metadata is auto-detected. Once a run uploads it shows up in the Test Runs dashboard alongside your JavaScript runs.
Get started
Pick your package on the Python SDK overview, pip install, set your API key, and run your tests. If you're already a TestRelic user, your existing org, repos, and AI features work with Python runs immediately.
Related documentation
- Python SDK overview — choose your package
- pytest — HTTP API Auto-Capture
- Playwright (Python) — Get Started
- DeepEval — Get Started
- Appium (Python) — Get Started
- Cloud Platform overview
FAQ: Do I need to change my tests?
No. All four packages are pytest plugins that activate on install. You set TESTRELIC_API_KEY and run pytest as usual.
FAQ: Which Python versions are supported?
pytest, Playwright, and Appium packages support Python 3.9+. DeepEval requires Python 3.10+ (and is tested on 3.9–3.12).
FAQ: Can Python and JavaScript runs live in the same org?
Yes. They upload to the same workspace and appear together in the Test Runs feed, with framework badges to tell them apart.