Define the decision
Name the workflow, intended users, consequences of error and the decision the evaluation should support.
Waltrump AI OrchestratorData readiness for AI workflows
For teams evaluating a RAG application, an AI assistant or submitted agent behavior: start with the sources, examples and decisions behind one workflow. Make gaps visible before treating a promising answer as proof of readiness.
A practical starting point
Outdated sources, conflicting definitions, missing context, unsupported answers and too few representative examples can all complicate a review. Start with one business objective, a permitted evidence sample, expected outcomes and an accountable reviewer.
Good information alone does not prove an AI workflow will perform correctly: retrieval, model behavior, instructions, evaluation design, permissions and workflow controls also matter. These are suggested review steps, not a certification or a claim that WAO automatically performs each one. A reviewer may need to inspect source versions, permissions and expectations manually.
Name the workflow, intended users, consequences of error and the decision the evaluation should support.
Identify source owners, permissions, versions and freshness. Record missing, conflicting or inaccessible material.
Choose customer-safe examples that cover common cases, edge cases and known gaps. Document expected outcomes and who reviewed them.
Check whether the workflow receives relevant evidence, whether answers are supported by it and where a human should intervene.
Keep the scope, evidence, unresolved gaps and next review visible as sources, workflows and models change.
Fictional example
Every detail below is invented to illustrate a possible review. The issue and next check are hypothetical, not observed results or a completed WAO evaluation.
Proposed question: Can an unopened item be returned 45 days after delivery?
Expected answer: Yes, within 60 days of delivery under the current fictional policy, subject to its other conditions.
Current: Fictional Returns Policy v2 allows 60 days. Obsolete: v1 allowed 30 days and should not be used for this question.
Possible issue: The assistant could cite v1 and give the wrong window. Proposed next check: Review the indexed source versions and test the answer and citation against v2, including an uncertain case for human review.
Proposed owner: The support policy owner, with the evaluation team.
Why the review matters
NIST suggests empirically evaluating model capability claims and reviewing sources in generated outputs during risk measurement and ongoing monitoring.
In interviews with 53 practitioners working in high-stakes AI, researchers documented data issues that can compound and cause downstream problems.
In a 2024 study of two tasks, some language models performed worse when relevant information appeared in the middle of a long input. Results depend on the models and tasks tested.
These sources do not evaluate or endorse WAO, establish universal performance, or show that WAO prevents these issues.
Explore in WAO
Explore current capability states and their evidence boundaries. Access depends on the applicable plan and rights; provider execution requires separate authorization.
15 capability records
Evaluate submitted evidence with a bounded evaluation pack
Submit evidence to a supported pack for a scoped evaluation.
What you get: A pack evaluation result for the submitted evidence.
Provider and agent runs remain separately activated.
When to use it: Use it when a defined pack fits the evidence you already have.
Illustrative example: Evaluate submitted examples using an eligible pack before discussing the findings.
See how your AI is performing and what needs checking
Bring available evaluation results into a scoped health review and see where evidence is missing.
What you get: A review of measured findings and testing gaps to help you choose what to investigate next.
Decision support for measured evidence; missing dimensions remain unmeasured.
When to use it: Use it when you need a concise view before expanding an AI workflow.
Illustrative example: A support assistant has evidence for common questions but not refund exceptions; review the gap before deciding what to test.
Turn a health finding into an accountable next step
Record an action against a scoped health finding, then review applicable evidence of completion.
What you get: A saved action and evidence-based verification state for follow-up.
An action is verified only with applicable evidence.
When to use it: Use it when a health review identifies an issue someone needs to address.
Illustrative example: Assign a coverage gap to an owner and check the supporting evidence before marking it verified.
See which readiness questions still need supporting evidence
Provide declarations and available evidence for a scoped readiness review.
What you get: A saved readiness view that separates declared answers from supporting evidence.
Declared answers do not replace measured evidence.
When to use it: Use it when a team needs to organize readiness questions before a review.
Illustrative example: A team records its review process and identifies where evidence has not yet been attached.
Evaluate an agent's proposed steps without executing tools
Submit a proposal-only trace for a scoped evaluation and save the result.
What you get: A saved trace evaluation for reviewing proposed behavior.
No live tool or computer action is executed.
When to use it: Use it when examining an agent's proposed tool use before any action is authorized.
Illustrative example: Review a submitted trace that proposes a tool call without making the call.
Run a bounded comparison after the required activation
Set up a bounded Direct run for an authorized project and provider, then recover its saved result.
What you get: A saved result for the eligible run, with failures visible.
Three-case customer comparison is unfinished; no complete tally is claimed for failed cases.
When to use it: Use it when an approved project needs a scoped provider evaluation.
Illustrative example: An activated team runs an eligible case and reopens the saved result for review.
Prepare a focused benchmark before running any provider
Create a finite Direct-policy setup and save it as a draft.
What you get: A saved preparation draft that can be reviewed before execution is considered.
Preparation does not execute a provider or certify a Benchmark.
When to use it: Use it when turning an evaluation question into a bounded setup.
Illustrative example: Draft an evaluation of support answers without sending a provider request.
Keep generated reports available for later review
Save and retrieve generated reports within your authorized scope.
What you get: A saved report and governed retrieval path for later reference.
Reports are evidence summaries, not certifications.
When to use it: Use it when stakeholders need to revisit a prior generated result.
Illustrative example: Open a saved technical report after a team discussion rather than generating it again.
Bring accountable people into an evidence review
Review scoped AI evidence with the people responsible for the decision.
What you get: A documented review that helps a team discuss findings and unresolved questions.
Review does not replace representative inputs or approval.
When to use it: Use it when a result needs human judgment before a decision.
Illustrative example: A reviewer considers unusual support responses alongside the available evaluation evidence.
Review quality factors supported by your existing evidence
Use scoped evidence to inspect a basic quality projection.
What you get: A view of quality factors supported by the submitted evidence.
The factor taxonomy is not universal measurement.
When to use it: Use it when you need to discuss output quality with your team.
Illustrative example: Review measured response-quality factors for a set of support answers.
Consider measured quality alongside costs you actually know
Review measured quality together with available cost evidence for the scoped workload.
What you get: A quality-and-cost view that makes missing cost information visible.
Unavailable provider cost is never inferred as zero.
When to use it: Use it when comparing trade-offs among measured options.
Illustrative example: Review two evaluated approaches using their known costs rather than treating an unknown cost as free.
Review a cited answer against its scoped retrieval evidence
Use the bounded cited-answer retrieval path, then save and reopen the result.
What you get: A saved answer and citations tied to the tested corpus and evidence window.
Grounding applies to the tested corpus and evidence window.
When to use it: Use it when checking how a retrieval-based answer is supported.
Illustrative example: Review a saved answer against citations from the corpus used in its evaluation.
Organize release evidence before a human decision
Review scoped release evidence and decision-support information.
What you get: A release-oriented evidence view for deciding what still needs attention.
It does not approve or deploy a release.
When to use it: Use it before deciding whether an AI change is ready for release.
Illustrative example: A team reviews missing evaluation evidence before approving a prompt change.
Turn submitted evidence into a reviewable report analysis
Provide evidence for a scoped report-intelligence review.
What you get: A report intelligence result bounded by the evidence supplied.
The report remains bounded by its source evidence.
When to use it: Use it when explaining the implications of evidence to a stakeholder.
Illustrative example: Review a submitted evaluation summary and its limitations before sharing a conclusion.
Keep the evidence behind a decision together
Use scoped evidence to create and save a decision scorecard.
What you get: A saved scorecard for discussing the evidence and decision with your team.
A scorecard is not WAO certification.
When to use it: Use it when a decision needs an accessible evidence summary.
Illustrative example: A release team records its criteria and the evidence it considered in a scorecard.
Take the next step
Bring customer-safe examples and a clear decision to discuss a reviewed evaluation. Requesting a review does not run a provider.