Scenario suites
Representative tasks with explicit setup, permissions, success conditions, prohibited outcomes, and cleanup.
Agent Evaluation
Evaluate whether the application completed the real task.
DualWeb builds scenario suites and environment outcome checks, with traces, failure tests, and release gates around the systems the application must actually change.
A fluent answer is not the outcome
An evaluation should inspect the final environment state, required artifacts, action history, policy compliance, and human intervention—not only the language of the last response.
Environment checks
Verify records, files, messages, bookings, configurations, or other real task outcomes.
Multiple trials
Repeat variable tasks enough times to expose reliability, cost, and intervention patterns.
Failure injection
Exercise unavailable tools, invalid data, timeouts, stale state, partial writes, and revoked permissions.
Recovery checks
Measure whether the runtime retries safely, resumes from checkpoints, escalates, or stops without compounding harm.
Release gates
Prevent deployment when critical task, security, cost, latency, or regression thresholds fail.
Metrics tied to completed work
Task completion
The proportion of scenarios whose required environment state and artifacts are correct.
Human intervention
Where a person had to correct, unblock, approve, or complete the workflow.
Failure and recovery
Tool errors, policy violations, unsafe retries, abandoned tasks, and successful recovery paths.
Latency and cost
End-to-end time and resource cost per completed task, separated from failed attempts.
Trace quality
Whether operators can understand tool selection, state changes, approvals, and the cause of failure.
Regression
Changes in task outcomes across prompts, models, tool versions, policies, and releases.
Evaluation is part of the release lifecycle
- 01
Define outcomes and risk
Write acceptance scenarios around the real workflow, permissions, and consequences.
- 02
Build deterministic checks
Inspect the resulting environment and artifacts with stable assertions wherever possible.
- 03
Run repeated and adversarial trials
Include normal tasks, ambiguous inputs, tool failures, permission denials, and interrupted execution.
- 04
Review traces and interventions
Identify why the runtime succeeded or failed and where controls changed the outcome.
- 05
Gate and monitor releases
Keep critical scenarios in regression testing and monitor production outcomes with the same definitions.
Make task completion measurable before launch.
We can include evaluation in a complete application build or scope a reliability engagement around an existing agentic product.
Discuss a project