WAO logoWaltrump AI Orchestrator

Data readiness for AI workflows

Give your AI workflow evidence it can be evaluated against.

For teams evaluating a RAG application, an AI assistant or submitted agent behavior: start with the sources, examples and decisions behind one workflow. Make gaps visible before treating a promising answer as proof of readiness.

A practical starting point

Review readiness in the context of one workflow.

Outdated sources, conflicting definitions, missing context, unsupported answers and too few representative examples can all complicate a review. Start with one business objective, a permitted evidence sample, expected outcomes and an accountable reviewer.

Good information alone does not prove an AI workflow will perform correctly: retrieval, model behavior, instructions, evaluation design, permissions and workflow controls also matter. These are suggested review steps, not a certification or a claim that WAO automatically performs each one. A reviewer may need to inspect source versions, permissions and expectations manually.

01

Define the decision

Name the workflow, intended users, consequences of error and the decision the evaluation should support.

02

Inventory the sources

Identify source owners, permissions, versions and freshness. Record missing, conflicting or inaccessible material.

03

Prepare representative examples

Choose customer-safe examples that cover common cases, edge cases and known gaps. Document expected outcomes and who reviewed them.

04

Review context and answers

Check whether the workflow receives relevant evidence, whether answers are supported by it and where a human should intervene.

05

Record limits and revisit

Keep the scope, evidence, unresolved gaps and next review visible as sources, workflows and models change.

Fictional example

A proposed review of a policy assistant.

Every detail below is invented to illustrate a possible review. The issue and next check are hypothetical, not observed results or a completed WAO evaluation.

Question and expected answer

Proposed question: Can an unopened item be returned 45 days after delivery?

Expected answer: Yes, within 60 days of delivery under the current fictional policy, subject to its other conditions.

Current and obsolete sources

Current: Fictional Returns Policy v2 allows 60 days. Obsolete: v1 allowed 30 days and should not be used for this question.

Hypothetical issue and next check

Possible issue: The assistant could cite v1 and give the wrong window. Proposed next check: Review the indexed source versions and test the answer and citation against v2, including an uncertain case for human review.

Proposed owner: The support policy owner, with the evaluation team.

Why the review matters

Research informs the questions, not a product guarantee.

Test use of long context

In a 2024 study of two tasks, some language models performed worse when relevant information appeared in the middle of a long input. Results depend on the models and tasks tested.

Liu et al.: Lost in the Middle

These sources do not evaluate or endorse WAO, establish universal performance, or show that WAO prevents these issues.

Explore in WAO

Connect your review to existing capabilities.

Explore current capability states and their evidence boundaries. Access depends on the applicable plan and rights; provider execution requires separate authorization.

15 capability records

Available in beta

AI Application and Agent Evaluation Packs

Evaluate submitted evidence with a bounded evaluation pack

Submit evidence to a supported pack for a scoped evaluation.

What you get: A pack evaluation result for the submitted evidence.

Provider and agent runs remain separately activated.

See how AI Application and Agent Evaluation Packs works

When to use it: Use it when a defined pack fits the evidence you already have.

Illustrative example: Evaluate submitted examples using an eligible pack before discussing the findings.

Technical details
Evidence
Versioned scenario, bounded trace and human-review evidence
Credential boundary
Customer-managed provider credential encrypted in the exact project is required only for approved provider execution
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
Yes — only after exact project activation
Availability
Available in Controlled Beta
Execution dependency
exact project byok and tuple activation for provider or agent runs
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Provider and agent runs remain separately activated.
Open Capability
Available in beta

AI Health

See how your AI is performing and what needs checking

Bring available evaluation results into a scoped health review and see where evidence is missing.

What you get: A review of measured findings and testing gaps to help you choose what to investigate next.

Decision support for measured evidence; missing dimensions remain unmeasured.

See how AI Health works

When to use it: Use it when you need a concise view before expanding an AI workflow.

Illustrative example: A support assistant has evidence for common questions but not refund exceptions; review the gap before deciding what to test.

Technical details
Evidence
Dimension states, confidence and missing evidence
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Decision support for measured evidence; missing dimensions remain unmeasured.
Open Capability
Available in beta

AI Health Action Center

Turn a health finding into an accountable next step

Record an action against a scoped health finding, then review applicable evidence of completion.

What you get: A saved action and evidence-based verification state for follow-up.

An action is verified only with applicable evidence.

See how AI Health Action Center works

When to use it: Use it when a health review identifies an issue someone needs to address.

Illustrative example: Assign a coverage gap to an owner and check the supporting evidence before marking it verified.

Technical details
Evidence
Evidence-linked action state and verification record
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none unless a separate retest is requested
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
An action is verified only with applicable evidence.
Open Capability
Available in beta

AI Readiness Pulse

See which readiness questions still need supporting evidence

Provide declarations and available evidence for a scoped readiness review.

What you get: A saved readiness view that separates declared answers from supporting evidence.

Declared answers do not replace measured evidence.

See how AI Readiness Pulse works

When to use it: Use it when a team needs to organize readiness questions before a review.

Illustrative example: A team records its review process and identifies where evidence has not yet been attached.

Technical details
Evidence
Declared and verified readiness with missing evidence
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none unless a separate benchmark is started
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Declared answers do not replace measured evidence.
Open Capability
Available in beta

Agent and Tool Intelligence

Evaluate an agent's proposed steps without executing tools

Submit a proposal-only trace for a scoped evaluation and save the result.

What you get: A saved trace evaluation for reviewing proposed behavior.

No live tool or computer action is executed.

See how Agent and Tool Intelligence works

When to use it: Use it when examining an agent's proposed tool use before any action is authorized.

Illustrative example: Review a submitted trace that proposes a tool call without making the call.

Technical details
Evidence
Capability-specific evidence and known limitations
Credential boundary
Customer-managed provider credential encrypted in the exact project is required only for approved provider execution
Provider/model boundary
One exact Google Gemini proposal-only agent tuple · Exact model/version confirmed during controlled access review
BYOK required
Yes — only after exact project activation
Availability
Available in Controlled Beta
Execution dependency
exact project byok and tuple activation proposal only
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
No live tool or computer action is executed.
Open Capability
Activation required

Benchmark

Run a bounded comparison after the required activation

Set up a bounded Direct run for an authorized project and provider, then recover its saved result.

What you get: A saved result for the eligible run, with failures visible.

Three-case customer comparison is unfinished; no complete tally is claimed for failed cases.

See how Benchmark works

When to use it: Use it when an approved project needs a scoped provider evaluation.

Illustrative example: An activated team runs an eligible case and reopens the saved result for review.

Technical details
Evidence
Benchmark observations, scope and limitations
Credential boundary
Customer-managed provider credential encrypted in the exact project is required only for approved provider execution
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
Yes — only after exact project activation
Availability
Limited Controlled Access
Execution dependency
exact project byok and tuple activation for live runs
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Three-case customer comparison is unfinished; no complete tally is claimed for failed cases.
Request Activation
Available in beta

Benchmark Express

Prepare a focused benchmark before running any provider

Create a finite Direct-policy setup and save it as a draft.

What you get: A saved preparation draft that can be reviewed before execution is considered.

Preparation does not execute a provider or certify a Benchmark.

See how Benchmark Express works

When to use it: Use it when turning an evaluation question into a bounded setup.

Illustrative example: Draft an evaluation of support answers without sending a provider request.

Technical details
Evidence
Benchmark observations, scope and limitations
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
Yes — only after exact project activation
Availability
Available in Controlled Beta
Execution dependency
exact project byok and tuple activation only when a live run starts
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Preparation does not execute a provider or certify a Benchmark.
Open Capability
Available in beta

Generated Reports

Keep generated reports available for later review

Save and retrieve generated reports within your authorized scope.

What you get: A saved report and governed retrieval path for later reference.

Reports are evidence summaries, not certifications.

See how Generated Reports works

When to use it: Use it when stakeholders need to revisit a prior generated result.

Illustrative example: Open a saved technical report after a team discussion rather than generating it again.

Technical details
Evidence
Safe report metadata and evidence references
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Reports are evidence summaries, not certifications.
Open Capability
Available in beta

Human Review

Bring accountable people into an evidence review

Review scoped AI evidence with the people responsible for the decision.

What you get: A documented review that helps a team discuss findings and unresolved questions.

Review does not replace representative inputs or approval.

See how Human Review works

When to use it: Use it when a result needs human judgment before a decision.

Illustrative example: A reviewer considers unusual support responses alongside the available evaluation evidence.

Technical details
Evidence
Independent review and adjudication record
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Review does not replace representative inputs or approval.
Open Capability
Available in beta

Quality Metrics

Review quality factors supported by your existing evidence

Use scoped evidence to inspect a basic quality projection.

What you get: A view of quality factors supported by the submitted evidence.

The factor taxonomy is not universal measurement.

See how Quality Metrics works

When to use it: Use it when you need to discuss output quality with your team.

Illustrative example: Review measured response-quality factors for a set of support answers.

Technical details
Evidence
Capability-specific evidence and known limitations
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
The factor taxonomy is not universal measurement.
Open Capability
Available in beta

Quality-Adjusted Cost

Consider measured quality alongside costs you actually know

Review measured quality together with available cost evidence for the scoped workload.

What you get: A quality-and-cost view that makes missing cost information visible.

Unavailable provider cost is never inferred as zero.

See how Quality-Adjusted Cost works

When to use it: Use it when comparing trade-offs among measured options.

Illustrative example: Review two evaluated approaches using their known costs rather than treating an unknown cost as free.

Technical details
Evidence
Explicit quality, validity and cost-availability states
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none for existing evidence; live reruns require exact activation
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Unavailable provider cost is never inferred as zero.
Open Capability
Available in beta

RAG and Retrieval Intelligence

Review a cited answer against its scoped retrieval evidence

Use the bounded cited-answer retrieval path, then save and reopen the result.

What you get: A saved answer and citations tied to the tested corpus and evidence window.

Grounding applies to the tested corpus and evidence window.

See how RAG and Retrieval Intelligence works

When to use it: Use it when checking how a retrieval-based answer is supported.

Illustrative example: Review a saved answer against citations from the corpus used in its evaluation.

Technical details
Evidence
Capability-specific evidence and known limitations
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
Grounding applies to the tested corpus and evidence window.
Open Capability
Available in beta

Release Intelligence

Organize release evidence before a human decision

Review scoped release evidence and decision-support information.

What you get: A release-oriented evidence view for deciding what still needs attention.

It does not approve or deploy a release.

See how Release Intelligence works

When to use it: Use it before deciding whether an AI change is ready for release.

Illustrative example: A team reviews missing evaluation evidence before approving a prompt change.

Technical details
Evidence
Capability-specific evidence and known limitations
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
It does not approve or deploy a release.
Open Capability
Available in beta

Report Intelligence

Turn submitted evidence into a reviewable report analysis

Provide evidence for a scoped report-intelligence review.

What you get: A report intelligence result bounded by the evidence supplied.

The report remains bounded by its source evidence.

See how Report Intelligence works

When to use it: Use it when explaining the implications of evidence to a stakeholder.

Illustrative example: Review a submitted evaluation summary and its limitations before sharing a conclusion.

Technical details
Evidence
Capability-specific evidence and known limitations
Credential boundary
No provider credential; WAO does not execute a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
The report remains bounded by its source evidence.
Open Capability
Available in beta

Shareable Decision Scorecards

Keep the evidence behind a decision together

Use scoped evidence to create and save a decision scorecard.

What you get: A saved scorecard for discussing the evidence and decision with your team.

A scorecard is not WAO certification.

See how Shareable Decision Scorecards works

When to use it: Use it when a decision needs an accessible evidence summary.

Illustrative example: A release team records its criteria and the evidence it considered in a scorecard.

Technical details
Evidence
Evidence hash, decision, alternatives, limitations and validity
Credential boundary
No provider credential is needed unless a separately approved workflow executes a provider
Provider/model boundary
Provider-neutral; exact combinations are separately qualified · Workload-selected and entitlement controlled
BYOK required
No for catalog review
Availability
Available in Controlled Beta
Execution dependency
none
Evidence freshness
Reviewed for interim classification; release acceptance pending
Known limitation
A scorecard is not WAO certification.
Open Capability

Take the next step

Start with one workflow and its open questions.

Bring customer-safe examples and a clear decision to discuss a reviewed evaluation. Requesting a review does not run a provider.