Nightly Observatory
The Nightly Observatory is the most feature-rich monitoring dashboard in TestRelic. It is designed for teams that run comprehensive overnight test suites and need deep visual analysis of long-running test data.
Accessing the Nightly Observatory
Navigate to Monitoring in the sidebar and select the Nightly tab.
Dashboard components
KPI strip
The KPI strip at the top of the Nightly Observatory provides a summary of the last nightly run:
| KPI | Description |
|---|---|
| Pass rate | Overall pass rate for the nightly run |
| Total sessions | Total test sessions in the nightly suite |
| Avg. duration | Mean session duration |
| Failure rate | Percentage of failing sessions |
| Flaky rate | Percentage of sessions classified as flaky |
Heatmaps
The heatmap visualization shows test pass/fail patterns over time in a grid format:
- Rows — Each row represents a test suite or test file.
- Columns — Each column represents a nightly run (day).
- Cell color — Green for pass, red for fail, amber for flaky, gray for skipped.
Heatmaps make it immediately obvious if a specific suite is consistently failing on certain days (e.g. weekend runs only) or if failures are randomly distributed (suggesting flakiness).
Transition arcs
Transition arcs visualize test status changes between consecutive nightly runs:
- An arc from green to red = a newly failing test.
- An arc from red to green = a resolved failure.
- An arc staying red = a persistent failure.
This visualization is particularly useful for identifying tests that are newly broken vs. tests that have been broken for multiple nights.
Failure constellation
The failure constellation is a force-directed graph where:
- Each node represents a failing test.
- Nodes cluster together when tests fail in the same run (suggesting a common root cause).
- Node size indicates how many times a test has failed across the window.
Tight clusters often indicate a shared infrastructure issue, a code regression, or a broken test data dependency.
Duration fingerprint
The duration fingerprint chart shows the execution time distribution for each test in the nightly suite. It helps identify:
- Tests that are consistently slow (outliers on the right side of the distribution).
- Tests whose duration varies widely across runs (potential performance instability).
- Sudden duration spikes that may indicate resource contention during overnight runs.
Suite burn rings
Suite burn rings are radial progress charts for each nightly test suite. Each ring shows:
- The overall pass rate as a filled arc.
- The suite name and total session count.
- Color coding for health (green / amber / red thresholds).
Flakiness matrix
The flakiness matrix is a table showing the flakiness score for each test across the monitored window. Columns represent days (or runs), and rows represent individual tests. Each cell shows:
- Green — Passed consistently.
- Red — Failed consistently.
- Striped — Alternated between pass and fail — the strongest indicator of flakiness.
The matrix is sorted by flakiness score descending, putting the most problematic tests at the top.
Interpreting nightly results
- Cluster of failures on the same night → likely a deployment issue or infrastructure problem.
- Same tests failing every night → persistent bugs or broken test setup.
- Tests alternating pass/fail on consecutive nights → genuine flakiness requiring stabilization.
Use the heatmap and flakiness matrix together to distinguish between these patterns efficiently.