Features | Cerberus CI

Features

Everything you need to trust your test suite

A comprehensive observability layer that sits above your existing test infrastructure.

Three-Tier Classification Pipeline

Core

The three heads of Cerberus: rules → cache → AI. Each tier is progressively more expensive. The rule-based pre-filter resolves ~40% of failures with zero API cost. The verdict cache prevents re-classifying the same recurring failure shape. Only genuinely new error signatures reach the AI provider.

Rule-based tier: retry-pass, consecutive failures, timeout patterns

Verdict cache: SHA-256 error signature with 30-day TTL

AI tier: only for new, ambiguous error signatures

Average API cost: <1 call per unique failure shape

1 "comment"># The classifier "keyword">runs automatically
2 "function">cerberus "keyword">classify --"keyword">run-id $CI_RUN_ID
3 
4 "comment"># Verdicts stored: flaky | regression | unknown
5 "comment"># Each classified_by: rules | cache | ai | mock

Deterministic Gate

The gate NEVER calls an AI provider. It reads stored verdicts and applies deterministic rules. This ensures gate behavior is 100% reproducible — the same code change always produces the same outcome, regardless of AI provider availability or latency.

Exit code 0 = pass, 1 = fail

Configurable: fail_on_regression, fail_on_unknown, fail_on_perf_regression

max_new_flaky_tests threshold

Audit trail: every verdict records which provider classified it

cerberus.config.yml

yaml

1 gate:
2   fail_on_regression: true
3   fail_on_unknown: false
4   fail_on_perf_regression: true
5   max_new_flaky_tests: 3

Performance Regression Detection

New

Statistical comparison against branch baselines. Uses rolling median (robust to outliers) computed from the target branch's history. Supports manual baselines from known-good runs. Cold start is handled gracefully — insufficient history warns but doesn't fail.

Rolling median comparison (robust to outliers)

Per-metric threshold overrides

Manual baseline support (trust 1 run)

Metric denylist for noisy metrics

Cold start: warn, don't fail

1 "comment"># Set a known-good "keyword">baseline
2 "function">cerberus "keyword">baseline "keyword">set --"keyword">run-id $GOOD_RUN --label "v1.0 release"
3 
4 "comment"># Check performance
5 "function">cerberus "keyword">gate --"keyword">run-id $CI_RUN_ID
6 
7 "comment"># page_load_ms: 800ms → 1100ms (+37.5%)
8 "comment"># ❌ Regression flagged (>20% threshold)

Provider-Agnostic AI

Two adapters cover the entire market. ClaudeProvider (native SDK) and OpenAICompatibleProvider (raw fetch, covers OpenAI, Groq, Ollama, and 20+ others). Switch with a config change — no code changes needed.

Claude: native @anthropic-ai/sdk

OpenAI-compatible: raw fetch, chat/completions shape

Covers OpenAI, OpenRouter, Groq, Together AI, DeepSeek, Ollama, LM Studio

MockProvider: deterministic heuristic, zero cost

Automatic fallback: missing API key → mock mode with warning

cerberus.config.yml

yaml

1 # Switch from Claude to Ollama
2 ai:
3   provider: openai-compatible
4   base_url: http://localhost:11434/v1
5   model: llama3
6   api_key_env: null

GitHub Action

One line to add to your workflow. Cerberus handles the full pipeline: ingest, classify, gate, report. Works with any test framework that outputs JUnit XML or Playwright JSON.

Composite action — no Docker needed

Supports all AI providers

Configurable inputs for every option

Gate result as output for downstream steps

GitHub Actions annotations for inline PR feedback

.github/workflows/ci.yml

yaml

1 - uses: EvertonSt/cerberus-ci-action@v1
2   with:
3     ai-provider: claude
4     ai-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
5     github-token: ${{ secrets.GITHUB_TOKEN }}
6     test-results-path: ./test-results/results.json
7     annotations: true

PR Comments

Plain-English quality reports posted on every pull request. Deduplicated (updates existing comment, no spam). Includes gate status, flaky count, regression count, performance deltas, and AI-generated analysis.

Markdown format with collapsible details

Hidden marker for deduplication

Updates existing comment, no spam

AI-generated trend analysis (collapsible)

Provider attribution footer

Run Comparison

Diff two CI runs side-by-side. See new failures, resolved issues, status changes, and performance deltas at a glance. Auto-selects the previous run on the same branch.

New failures (🔴), resolved (🟢), unchanged (⚪)

Performance deltas with percentage

Auto-select previous run on same branch

JSON output for CI scripts

1 "function">cerberus "keyword">compare --"keyword">run-id $RUN_B
2 "comment"># BEFORE: "keyword">run-A (aaa111) @ 2026-01-10
3 "comment"># AFTER:  "keyword">run-B (bbb222) @ 2026-01-11
4 "comment"># 🔴 New failures: 1
5 "comment"># 🟢 Resolved: 1
6 "comment"># 📈 page_load_ms: 800ms → 1100ms (+37.5%)

Trends Analysis

Analyze flaky test patterns across multiple runs. Detect worsening, improving, and stable tests. See the worst offenders by fail rate and get actionable recommendations.

Worsening / improving / stable detection

Recent vs. overall fail rate comparison

Worst offenders by fail rate

JSON output for CI integration

1 "function">cerberus "keyword">trends --branch main
2 "comment"># Total "keyword">runs analyzed: 50
3 "comment"># Overall fail rate: 12%
4 "comment"># 📈 checkout.spec.ts:42 — 40% fail rate (worsening)

Zero-Cost Mock Mode

Every feature works without any API key. MockProvider uses deterministic local heuristics to classify failures. Full pipeline is testable and demoable in CI with zero external API cost.

Deterministic heuristic: retry→flaky, assertion→regression

CERBERUS_MOCK=1 or no API key → auto fallback

All 237 tests run in mock mode

Perfect for demos, CI, and development

1	"comment"># The classifier "keyword">runs automatically
2	"function">cerberus "keyword">classify --"keyword">run-id $CI_RUN_ID
3
4	"comment"># Verdicts stored: flaky \| regression \| unknown
5	"comment"># Each classified_by: rules \| cache \| ai \| mock

1	gate:
2	fail_on_regression: true
3	fail_on_unknown: false
4	fail_on_perf_regression: true
5	max_new_flaky_tests: 3

1	"comment"># Set a known-good "keyword">baseline
2	"function">cerberus "keyword">baseline "keyword">set --"keyword">run-id $GOOD_RUN --label "v1.0 release"
3
4	"comment"># Check performance
5	"function">cerberus "keyword">gate --"keyword">run-id $CI_RUN_ID
6
7	"comment"># page_load_ms: 800ms → 1100ms (+37.5%)
8	"comment"># ❌ Regression flagged (>20% threshold)

1	# Switch from Claude to Ollama
2	ai:
3	provider: openai-compatible
4	base_url: http://localhost:11434/v1
5	model: llama3
6	api_key_env: null

1	- uses: EvertonSt/cerberus-ci-action@v1
2	with:
3	ai-provider: claude
4	ai-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
5	github-token: ${{ secrets.GITHUB_TOKEN }}
6	test-results-path: ./test-results/results.json
7	annotations: true

1	"function">cerberus "keyword">compare --"keyword">run-id $RUN_B
2	"comment"># BEFORE: "keyword">run-A (aaa111) @ 2026-01-10
3	"comment"># AFTER: "keyword">run-B (bbb222) @ 2026-01-11
4	"comment"># 🔴 New failures: 1
5	"comment"># 🟢 Resolved: 1
6	"comment"># 📈 page_load_ms: 800ms → 1100ms (+37.5%)

1	"function">cerberus "keyword">trends --branch main
2	"comment"># Total "keyword">runs analyzed: 50
3	"comment"># Overall fail rate: 12%
4	"comment"># 📈 checkout.spec.ts:42 — 40% fail rate (worsening)