Skip to main content
Ask AI

Examples

Copy-pasteable patterns for the most common ways to run DeepEval through TestRelic.

A pytest DeepEval suite (auto-upload)​

With testrelic-deepeval installed, the pytest plugin auto-registers. Write a normal DeepEval test — no imports from TestRelic, no flags — and the run uploads at session finish:

tests/test_evals.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric


def test_answer_relevancy():
    test_case = LLMTestCase(
        input="What is the capital of France?",
        actual_output="The capital of France is Paris.",
    )
    assert_test(test_case, [AnswerRelevancyMetric(threshold=0.7)])

Run it:

Terminal
deepeval test run tests/
note

If no API key or credentials are present, the plugin logs a message and no-ops. It never fails your tests — upload exceptions are swallowed.

The programmatic evaluate() wrapper​

Outside pytest, use the evaluate() drop-in. It runs the eval and auto-uploads:

run_eval.py
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from testrelic import evaluate

results = evaluate(
    test_cases=[LLMTestCase(input="Hi", actual_output="Hello")],
    metrics=[AnswerRelevancyMetric(threshold=0.7)],
)
# results is whatever deepeval.evaluate() returns; upload is automatic

Offline, then testrelic drain​

If the cloud is unreachable, uploads are written to the offline queue at ~/.testrelic/queue/. Run your evals as usual — they're queued locally:

Terminal
# Network down or no connectivity — runs are queued to ~/.testrelic/queue/
deepeval test run tests/

Later, once you're back online, replay everything in the queue:

Terminal
testrelic drain

Successfully delivered runs are cleared from the queue.

A GitHub Actions step​

In CI, set TESTRELIC_API_KEY from a secret and run your suite. The SDK auto-detects branch, commit, and CI URL for GitHub Actions:

.github/workflows/evals.yml
- name: Run DeepEval suite
  env:
    TESTRELIC_API_KEY: ${{ secrets.TESTRELIC_API_KEY }}
  run: |
    pip install "testrelic-deepeval[deepeval]"
    deepeval test run tests/

See GitHub Actions for the broader CI setup.

Next steps​