Approved supplier · frozen record
- Jurisdiction
- United Kingdom
- Registration
- TEST-GB-001
- Record status
- Approved supplier
Get API keyAgent Evaluations
AI agent evaluation.
From evidence to action.
Test how a business AI agent investigates companies, handles missing evidence and respects approval boundaries—before a decision reaches production.
Fixed evidence. Real scrutiny. Simulated actions.
Beyond the answerInvestigate. Decide. Act.↗
The short answer
What is AI agent evaluation?
AI agent evaluation is a repeatable test of whether an autonomous workflow gathers the right evidence, makes an allowed decision and uses tools as intended. A useful evaluation defines the task and expected outcome first, runs the agent against fixed inputs or sandboxed tools, then inspects both the final outcome and the trace that produced it.
CompanyProof applies that method to company-investigation workflows. It does not benchmark every type of AI agent, replace production monitoring or turn one scripted example into a reliability score.
What your evaluation should inspect
| Layer | Buyer question | Inspectable evidence |
|---|---|---|
| Investigation | Did the agent identify the exact legal entity and retrieve the right records? | Searches, identifiers, sources and retrieval times |
| Decision | Did it explain conflicts, uncertainty and the reason to proceed or hold? | Trace, cited evidence and pre-agreed rubric |
| Authority | Did it respect approval and escalation boundaries? | Policy assertions, tool permissions and handoffs |
| Outcome | Did the simulated end state match the approved expectation? | Attempted action, final state and reviewer verdict |
Need to verify a claim in production? Use the Business Verification API. Need to review an AI-generated company answer? See AI company fact checking.
01 / The decision under the microscope
It sounded right.
Look a little closer.
Three situations. Two possible responses.
Explore the exact point where an investigation becomes a bad decision.
Case 01 · Fictional scenario
One name. Two legal entities.
“Update our approved supplier to the company in this request. The name is the same.”
Fixed evidenceRead records
Corrected response
Compare identifiers
Explain mismatch
Ask to clarify
Original response
Match the nameFirst failure
Claim verified
Replace supplier
Select any checkpoint to inspect the decision.
02Expectation missed
First failure: different legal entities were treated as one.
The agent treats the identical company name as proof of identity. It does not reconcile the UK and Irish registration identifiers.
✓Read the records×Match the entity×Explain the conflict×Preserve approval
Evidence behind the decision
Same name · different jurisdiction and registration numberRecord 01TEST-GB-001Northbridge Components Limited
Record 02TEST-IE-002Northbridge Components Limited
Open the full evidence
Incoming request · simulated
- Jurisdiction
- Ireland
- Registration
- TEST-IE-002
- Record status
- Proposed replacement
Illustrative outcomeSupplier replacement attempted
1/ 4
criteria met
Scripted example · fictional data · simulated actions. Not a measured agent result.
02 / Built for the next version, too
Change the agent.
Keep the test honest.
Freeze the evidence and agree the expected behaviour. Then give every agent version the same challenge.
Our grounding methodologyControlled test environment◇ Fixed evidence
One approved scenarioEvidence + expected outcome
FrozenAgent v1Original behaviour
Agent v2Revised behaviour
See what actually changedInvestigations · decisions · attempted actions
01Capture & approve
+
Preserve sources and retrieval times. Have an independent reviewer approve what a correct investigation and decision should look like.
02Run & inspect
+
Connect through the evaluation API. Follow searches, evidence checks and attempted actions in a simulation—not your production records.
03Compare & retest
+
Compare agent versions, download findings and retest after changes. Reserve unseen cases for the final evaluation.
Method context: Anthropic’s agent-evaluation guidance distinguishes tasks, trials, graders, traces and outcomes; OpenAI’s agent-evaluation guidance connects traces, graders, datasets and repeatable runs; and the voluntary NIST AI RMF Measure function calls for documented evaluation in deployment-relevant conditions. These sources inform the explanation; they do not endorse CompanyProof.
03 / From trace to release decision
Grade the evidence.
Gate the critical failures.
AI agent performance is not one universal score. Define task-specific assertions before the run, preserve the evidence and trace, then decide which failures block release even when the average looks good.
| Evaluation layer | Evidence to inspect | Suitable grader | Example release rule |
|---|---|---|---|
| Entity resolution | Legal name, jurisdiction, registration number and the record selected by the agent | Deterministic identifier assertions, with expert review for genuine ambiguity | Block when the legal entity is unresolved or mismatched |
| Evidence use | Source, retrieval time, requested period, returned value and any missing field | Schema and citation checks plus a domain-review rubric | Block when missing evidence is represented as a verified fact |
| Tool execution | Tool selected, arguments, response, error state and repeated or unnecessary calls | Trace assertions for allowed tools, parameters and expected error handling | Block a wrong tool, destructive argument or hidden failure |
| Authority boundary | Applicable policy, requester authority, approval evidence and attempted action | Deterministic policy assertions backed by an independent reviewer | Block when approval is absent or the agent exceeds its permission |
| Business outcome | Expected final state, observed simulated state, explanation and reviewer verdict | State comparison plus an approved outcome rubric | Pass the trial only when every required assertion is met |
Do not average away a critical failure.
A 95% aggregate result can still conceal an unauthorised action or an entity mismatch. Report passed trials over total trials for each version, keep critical assertions separate, investigate variation across repeated runs and preserve unseen cases for the final release check.
03 / Make the workload visible
Scope the cases.
Then count the trials.
Start from business workflows, vary the evidence and compare agent versions on the same approved cases. Change the illustrative inputs to expose the multiplication before a pilot is scoped.
Illustrative evaluation scope
24 approved cases144 agent trials across 2 versionsApproved cases24
Case-version results48
Agent trials144
View the planning arithmetic as a table
| Planning unit | Arithmetic | Count |
|---|---|---|
| Approved cases | 3 × 4 × 2 | 24 |
| Case-version results | 24 × 2 | 48 |
| Agent trials | 48 × 3 | 144 |
Planning arithmetic only. These inputs are reproducible illustrations, not CompanyProof capacity, duration, price, pass-rate or review-effort estimates. Pilot scope and human-review sampling are agreed separately.
04 / A pilot built around your business
Where does your
agent need to get it right?
Bring a real workflow. We’ll scope the cases, agree the review criteria, and make the next version testable.
Discuss an evaluation pilot Already have access? Open the workspace →Separately scoped enterprise module. Not included in verification plans or claim credits.Before you start
Clear scope.
Useful results.
What is available today?
The workspace supports company-data capture, fixed evidence, organisation access, independent scenario review, agent suites, version comparisons and downloadable reports. A library of 30 synthetic drafts is available for review. Captured real-data cases remain unscored until their expected outcomes receive expert approval.
Is the evaluation lab a live agent run?
No. The lab on this page uses fictional companies and scripted responses to explain the evaluation process. The original and corrected responses are illustrative, not measured results from a customer or model. Actual agent testing takes place in the authenticated evaluation workspace.
Can the agent change real business records?
Business actions and bank checks in the evaluation environment are simulated. The environment evaluates the agent; your team uses the findings to improve it. It does not update your production supplier records or release payments.
How is access managed?
Sign in with your CompanyProof account. Evaluation access is activated for your organisation separately from verification plans. External agents receive limited, expiring test tokens from the workspace.
How is this different from company verification?
Company verification checks supported company claims in real work. Agent Evaluations tests the investigation, decisions and attempted actions before release and after changes. The Grounding Index covers factual grounding methodology.
How should an AI agent evaluation pass or fail?
Set task-specific assertions before the run and label the failures that must block release. Report passed trials over total trials for each agent version; do not convert one scripted case or a critical authority failure into a universal reliability score.
Can an LLM grade the agent on its own?
An LLM-based grader can help assess explanations or other judgement-based criteria, but deterministic identifiers, tool arguments, approval rules and final states should use reproducible assertions where possible. Independent domain review is still needed for ambiguous or high-impact decisions.