Skip to main content
Ask AI

Pytest Integration

testrelic-deepeval registers a pytest plugin through a pytest11 entry point. Once the package is installed in the same environment as DeepEval, every pytest invocation — including deepeval test run — captures DeepEval's evaluation result at session finish and uploads it to TestRelic. There is nothing to import and no conftest.py wiring.

Credential resolution​

The plugin resolves your credentials from the first source that provides a value:

  1. Environment variables — TESTRELIC_API_KEY (and optional endpoint overrides).
  2. Credentials file — ~/.testrelic/credentials.toml, written by testrelic login.

If no credentials are found, the plugin still loads but uploads nothing — DeepEval runs exactly as before. See Configuration for the full precedence rules.

It never fails your tests​

Capture happens after your tests finish, so it can never change an outcome:

  • With no credentials present, the plugin logs a message and no-ops.
  • Any upload error is swallowed — a reporting problem can't turn a green suite red.

Skipping upload for a run​

To run your suite without uploading, simply run without credentials — for example, unset TESTRELIC_API_KEY. The plugin loads but skips the upload.

Terminal
TESTRELIC_API_KEY= deepeval test run tests/

Project and repo identity​

Attach a run to a specific project or repository with environment variables:

Terminal
export TESTRELIC_PROJECT=rag-experiments
export TESTRELIC_REPO_GIT_ID=my-org/llm-pipeline

In CI, branch, commit, and run URL are auto-detected, so these are usually inferred for you.

Hyperparameters pass through​

DeepEval's own hyperparameters flow through to TestRelic unchanged and appear on the eval run, so you can group and compare runs by model, temperature, or prompt version:

run_eval.py
from testrelic import evaluate

evaluate(
    test_cases=cases,
    metrics=metrics,
    hyperparameters={"model": "gpt-4o", "temperature": 0.2, "prompt": "v3"},
)

Programmatic (non-pytest) usage​

When you aren't running under pytest, use the evaluate() wrapper instead — it runs the eval and uploads on completion:

run_eval.py
from testrelic import evaluate

results = evaluate(test_cases, metrics)
# results is whatever deepeval.evaluate() returns; upload is automatic

Next steps​