Skip to main content
Ask AI

TestRelic for DeepEval

testrelic-deepeval captures your DeepEval LLM-evaluation results and uploads them to your TestRelic org — instead of Confident AI. It reads DeepEval's in-memory TestRun only: there is no monkey-patching and no HTTP redirect. Results are sent to TestRelic's native /api/v1/evals/* API.

Why use it​

  • Own your eval data. Evaluation runs land in your TestRelic org, alongside your test runs and repositories — not in a third-party service.
  • Track quality over time. Eval runs surface in the DeepEval / eval workspace and contribute to the Eval Stability metric, so you can watch metric pass rates trend across commits.
  • Zero rewiring. Keep your existing DeepEval suites. You either install the pytest plugin (no code changes) or swap one import for the evaluate() wrapper.

Requirements​

RequirementVersion
Python>=3.10 (tested 3.9–3.12)
DeepEval (optional extra)>=2.0,<4.0

DeepEval itself is an optional extra. To pull it in alongside the SDK:

Terminal
pip install "testrelic-deepeval[deepeval]"

The base dependencies (httpx, pydantic, typer, platformdirs) are installed automatically.

Two ways to use it​

There are two usage modes — pick whichever fits how you already run DeepEval:

  • pytest plugin — installed automatically with the package. It auto-captures DeepEval's global TestRun at session finish and uploads it. No code changes, no pytest flags, and it never fails your tests.
  • evaluate() wrapper — a drop-in replacement for deepeval.evaluate() that runs the eval and then auto-uploads the result.
Programmatic example
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from testrelic import evaluate

results = evaluate(
    test_cases=[LLMTestCase(input="Hi", actual_output="Hello")],
    metrics=[AnswerRelevancyMetric(threshold=0.7)],
)
# results is whatever deepeval.evaluate() returns; upload is automatic
note

The CLI entry point is testrelic (not testrelic-deepeval). Use it to log in, view runs, and replay queued uploads.

Next steps​