Platform · Evaluations & compliance

Don't just monitor AI. Measure how well it performs.

Score real turns with live evaluations, gate releases with golden tests from your CI pipeline, and review completed turns with configured compliance assessments—each with its own purpose and timing.

Book a live demo View the quickstart

Quality at a glance

goal_completion92

context_adherence86

golden: smoke (pass rate)94

Copper marker = pass threshold

Illustrative product scenario — synthetic data

The problem

Uptime and latency say nothing about whether the answer was right.

Quality regressions slip in with prompt, model and code changes. Without measurement in production and in CI, teams find out from users.

No objective quality signal on real traffic
Model or prompt changes shipped without regression tests
No routine review of completed agent activity

Capabilities

What Evaluations & compliance does

Live evaluations

Judge models score a sample of real turns on metrics such as goal completion and context adherence, against your thresholds.

Golden test suites

Scripted conversations with assertions on tool calls, text and judged criteria, declared in an Evaluation file.

CI release gates

Run golden tests from your pipeline with JUnit output; the build fails below your minimum pass rate.

Replay and re-score

Replay an agent's scenarios on a new model in a registration dry run, and re-score stored turns.

Compliance assessments

Five post-turn checks: sensitive data, toxicity, safety, prompt attacks and off-purpose requests.

Review and alerts

A review queue for owners, critical alerts and daily summaries—delivered to your team's channels.

Technical detail

Three mechanisms, kept distinct

Guardrails act during the request. Evaluations and compliance run after the turn completes and never block the user.

Run golden tests from CI

intsen eval run --agent field-service/fleet-service-advisor --dataset smoke \  --version $COMMIT --adapter "python tests/run_case.py" --junit out/evals.xml

Live evaluations: after the turn · sampled · scores against thresholds · breaches alert
Golden tests: in your CI · scripted scenarios · pass rate gates the release
Compliance: after the turn · opt-in per agent · findings go to a review queue
Judge models: called through the gateway on your registration; spend shown separately
Eval traffic: tagged and kept out of production numbers

Why it matters

For the people accountable for it

AI platform

A shared quality bar across teams, with trends per agent.

Engineering

Catch regressions in CI before a prompt or model change ships.

Risk / compliance

A routine, recorded review of completed agent turns, with triage.

Boundaries

What to know before you evaluate

We would rather you hear the limits from us than discover them in a pilot.

Compliance assessments are configured checks, not a statutory or regulatory certification.
Live evaluations score a sample you choose; they are not a guarantee about every turn.
Judge-model scores are an estimate of quality and should be calibrated against your own reviewed examples.
Running golden tests from the Console and scheduled runs are planned for after the first release.

FAQ

Common questions

Guardrails Agent tracing Spend management

Who pays for judge-model calls?

They run on your registration with the model you choose. Each call is tagged, so the Console shows evaluation spend separately from your agent's own cost.

Do evaluations slow down the user?

No. Live evaluations and compliance run after the turn completes.

What does a golden test adapter look like?

A few lines that call your agent with the scenario input. The helper package adds the IntSen headers.

Ready to make your enterprise AI accountable?

See how IntSen can help your teams govern access, safety, quality and spending across the AI applications and agents you already run.

Book a live demo Discuss an enterprise pilot