10.1 Objective
A working AI application is not automatically a reliable AI system.
ACAI must be evaluated systematically.
The goal of this chapter is to establish a framework that answers:
What does ACAI do?
How accurately does it do it?
Where does it fail?
How often does it fail?
How fast is it?
Does a new change improve or reduce performance?
Can the results be reproduced?
The architecture therefore becomes:
System
↓
Evaluation Dataset
↓
Test Execution
↓
Metrics
↓
Failure Analysis
↓
Improvement
↓
Regression Testing
10.2 Evaluation Architecture
ACAI
│
▼
Evaluation Runner
│
┌────────────┼────────────┐
▼ ▼ ▼
Quality Speed Reliability
│ │ │
└────────────┼────────────┘
▼
Metrics
│
▼
Evaluation Report
10.3 Evaluation Dataset
Create a controlled set of test cases.
Example:
tests/evaluation/
basic_questions.json
reasoning.json
retrieval.json
memory.json
workflow.json
failure_cases.json
A test case can contain:
{
"id": "basic_001",
"category": "general",
"input": "Explain photosynthesis.",
"expected_behavior": "Provide a correct concise explanation."
}
The expected behavior should be specific enough to evaluate.
10.4 Categories
ACAI should not be tested only with simple questions.
A useful evaluation suite can contain:
General Questions
Reasoning
Planning
Retrieval
Memory
Multi-step Tasks
Tool Usage
Long Context
Ambiguous Requests
Invalid Inputs
Failure Recovery
Safety Constraints
This produces a more representative picture of system behavior.
10.5 Test Case Structure
A stronger format is:
{
"id": "reasoning_001",
"category": "reasoning",
"input": "Example task...",
"expected": {
"must_contain": [],
"must_not_contain": [],
"criteria": [
"correct conclusion",
"valid reasoning"
]
}
}
Not every task can be reduced to exact string matching.
Therefore ACAI should support multiple evaluation methods.
10.6 Exact-Match Evaluation
For deterministic outputs:
def exact_match(
actual: str,
expected: str,
) -> bool:
return (
actual.strip().lower()
== expected.strip().lower()
)
Useful for:
Classification
Fixed-format output
Simple transformations
Deterministic calculations
It is not sufficient for open-ended generation.
10.7 Keyword Evaluation
A simple evaluator:
def contains_required_terms(
text: str,
required: list[str],
) -> bool:
normalized = text.lower()
return all(
term.lower() in normalized
for term in required
)
This can be useful for basic smoke tests.
However, keyword presence does not prove semantic correctness.
10.8 Rule-Based Evaluation
For structured output, define explicit rules.
Example:
Required:
- answer field
- confidence field
- explanation field
Forbidden:
- invalid JSON
- empty answer
This provides deterministic validation.
10.9 Semantic Evaluation
For natural-language answers, evaluation may require semantic comparison.
Possible dimensions:
Correctness
Relevance
Completeness
Clarity
Groundedness
Instruction following
For research purposes, the exact evaluation protocol should be documented rather than relying on an unspecified "AI score."
10.10 Human Evaluation
Some tasks require human judgment.
A human evaluator can assign scores such as:
1 = Poor
2 = Weak
3 = Acceptable
4 = Good
5 = Excellent
Example dimensions:
Accuracy
Relevance
Clarity
Completeness
Usefulness
The evaluation instructions should remain consistent between evaluators.
10.11 Automated Evaluation
Automated evaluation is useful for large test suites.
Architecture:
Test Input
↓
ACAI
↓
Generated Output
↓
Evaluator
↓
Score
The evaluator itself can be:
Rule-based
Reference-based
Programmatic
Model-assisted
Model-assisted evaluation should be validated because an evaluator model can also make mistakes.
10.12 Task Success Rate
One important metric is:
Task Success Rate
=
Successful Tasks
÷
Total Tasks
× 100
For example, if 92 out of 100 tasks meet the predefined success criteria:
92 / 100 × 100 = 92%
The definition of "successful" must be specified before reporting the result.
10.13 Accuracy
For classification tasks:
Accuracy
=
Correct Predictions
÷
Total Predictions
For example:
90 correct
100 total
Accuracy = 90%
Accuracy can be misleading when classes are highly imbalanced, so additional metrics may be necessary.
10.14 Precision and Recall
For classification:
Precision
=
True Positives
÷
(True Positives + False Positives)
Recall
=
True Positives
÷
(True Positives + False Negatives)
For some classification tasks, F1 can also be useful.
10.15 Latency Measurement
For every evaluation run, record latency.
Example:
import time
start = time.perf_counter()
result = await acai.run(
request
)
latency = (
time.perf_counter()
- start
)
Store:
latency_ms
10.16 Average Latency
If five requests take:
100 ms
200 ms
150 ms
250 ms
300 ms
the average is:
200 ms
Average latency is useful but does not show the tail of the distribution.
10.17 Percentile Latency
Production systems often examine:
P50
P95
P99
Conceptually:
P50 → typical request
P95 → slower tail
P99 → extreme tail
This helps identify occasional severe slowdowns.
10.18 Reliability
Reliability should be measured independently from quality.
Example:
100 requests
97 completed
3 failed
Then:
Completion Rate = 97%
Failure Rate = 3%
A response that technically completes but is incorrect should not automatically count as a successful task.
10.19 Quality vs Reliability
These are different:
Reliability:
"Did the system complete?"
Quality:
"Was the result good?"
Therefore:
Request
│
├── Completed?
│
└── Correct?
A system can have:
High reliability
Low quality
or:
High quality
Low reliability
The evaluation framework should measure both.
10.20 Regression Testing
Every significant code change should run the evaluation suite.
Architecture:
Code Change
↓
Tests
↓
Evaluation
↓
Compare With Baseline
↓
Accept / Reject
Example:
Version A
Task Success: 88%
Version B
Task Success: 91%
This suggests improvement, but the difference should be tested for statistical and practical significance where appropriate.
10.21 Evaluation Baseline
Create a baseline report:
ACAI Evaluation
Version: 1.0
Task Success: measured value
Accuracy: measured value
Failure Rate: measured value
P50 Latency: measured value
P95 Latency: measured value
These numbers should only be populated after actual evaluation runs.
10.22 Evaluation Runner
Create:
app/evaluation/runner.py
Example:
class EvaluationRunner:
def __init__(
self,
system,
evaluator,
):
self.system = system
self.evaluator = evaluator
async def run_case(
self,
case,
):
result = await self.system.run(
case["input"]
)
score = self.evaluator.evaluate(
case,
result,
)
return {
"id": case["id"],
"result": result,
"score": score,
}
10.23 Evaluation Results
A complete evaluation record can look like:
{
"id": "reasoning_001",
"success": true,
"score": 0.9,
"latency_ms": 742,
"provider": "mock",
"failure": null
}
In production, provider/model information can be added to make comparisons possible.
10.24 Aggregate Metrics
Create:
def aggregate_results(
results,
):
total = len(results)
successful = sum(
1
for result in results
if result["success"]
)
success_rate = (
successful / total
if total
else 0
)
return {
"total": total,
"successful": successful,
"success_rate": success_rate,
}
10.25 Failure Analysis
A good evaluation system does not stop at:
Success = 84%
It should answer:
What failed?
Why did it fail?
Which component failed?
Can the failure be reproduced?
How can it be fixed?
Create failure categories:
PLANNING_ERROR
RETRIEVAL_ERROR
MEMORY_ERROR
MODEL_ERROR
TOOL_ERROR
VALIDATION_ERROR
TIMEOUT
SYSTEM_ERROR
10.26 Failure Pipeline
Failed Test
↓
Capture Input
↓
Capture Output
↓
Capture Logs
↓
Identify Component
↓
Classify Failure
↓
Create Fix
↓
Add Regression Test
The final step is particularly important.
A fixed failure should become a permanent test case so that the same bug does not silently return.
10.27 Reproducibility
A research-oriented system should record enough information to reproduce a run.
Useful metadata:
ACAI version
Git commit
Model provider
Model identifier
Configuration
Prompt/template version
Dataset version
Timestamp
Random seed, when applicable
Not every model API exposes deterministic behavior, so exact reproducibility may not always be possible.
The goal is to maximize reproducibility and document remaining variability.
10.28 Dataset Versioning
Evaluation datasets should also be versioned.
Example:
evaluation-v1
evaluation-v2
evaluation-v3
Then a result can state:
ACAI 1.4
on evaluation-v3
rather than simply:
ACAI is 95% accurate.
The second statement is incomplete without the evaluation conditions.
10.29 Benchmark Structure
A benchmark report can contain:
System
Version
Dataset
Number of Tasks
Task Categories
Metrics
Results
Failure Analysis
Limitations
Environment
This makes the result understandable to someone who did not build the system.
10.30 Example Benchmark Table
| Category | Cases | Success Rate | Avg. Latency |
|---|
| General | measured | measured | measured |
| Reasoning | measured | measured | measured |
| Retrieval | measured | measured | measured |
| Memory | measured | measured | measured |
| Workflow | measured | measured | measured |
Do not replace "measured" with invented numbers.
10.31 Comparing Two Versions
Suppose:
Version A
Success = 80%
Version B
Success = 85%
Absolute improvement:
85 - 80 = 5 percentage points
Relative improvement:
(85 - 80) / 80 × 100
= 6.25%
Both descriptions are valid, but they communicate different things.
10.32 A/B Evaluation
For larger experiments:
Users
│
┌─────┴─────┐
▼ ▼
Version A Version B
│ │
▼ ▼
Metrics Metrics
│ │
└─────┬─────┘
▼
Compare
The groups should be defined carefully to avoid introducing confounding factors.
10.33 Statistical Considerations
A difference in benchmark scores does not automatically prove that one system is better.
For example:
System A = 90%
System B = 91%
If only a tiny number of examples were tested, the difference may be unstable.
A serious evaluation should consider:
Sample size
Variance
Confidence intervals
Task composition
Repeated trials
Statistical tests where appropriate
10.34 Human Evaluation Agreement
When multiple people evaluate outputs, agreement matters.
If evaluator A gives:
5
and evaluator B gives:
1
the evaluation instructions may be unclear.
Therefore human evaluation should include:
Clear rubric
Examples
Independent scoring
Agreement analysis
Adjudication process
10.35 Evaluation Dashboard
Eventually, ACAI can expose:
/evaluation
with:
Total Tests
Success Rate
Failure Rate
Average Latency
P95 Latency
Fallback Rate
Provider Usage
Category Performance
Conceptually:
┌─────────────────────────────────┐
│ ACAI Evaluation Dashboard │
├─────────────────────────────────┤
│ Success Rate measured │
│ Failure Rate measured │
│ P95 Latency measured │
│ Fallback Rate measured │
├─────────────────────────────────┤
│ Category Performance │
│ │
│ Reasoning measured │
│ Retrieval measured │
│ Memory measured │
│ Workflow measured │
└─────────────────────────────────┘
10.36 Continuous Evaluation
Evaluation should not happen only once.
A mature workflow is:
Developer Change
↓
Automated Tests
↓
Evaluation Suite
↓
Compare Baseline
↓
Review
↓
Deploy
↓
Production Monitoring
↓
New Failures
↓
Add Tests
↓
Next Release
This creates a feedback loop.
10.37 Real-World Validation
Before claiming that ACAI performs a capability reliably, define:
Capability
Success criteria
Test dataset
Evaluation method
Minimum acceptable performance
Failure conditions
For example:
Capability:
Multi-step planning
Success criteria:
All required steps identified
Dependencies respected
Invalid steps rejected
Evaluation:
100 predefined tasks
Result:
Measured after execution
This is much stronger than saying:
"It seems intelligent."
10.38 Research Documentation
If ACAI is presented as a research project, document:
Architecture
Implementation
Training/fine-tuning, if any
Models
Datasets
Evaluation protocol
Results
Limitations
Known failures
Hardware
Software versions
If a capability was not tested, state that clearly.
10.39 What Counts as Evidence?
Evidence can be ranked roughly as:
Claim
↓
Demonstration
↓
Repeatable Test
↓
Benchmark
↓
Independent Reproduction
A video demonstration can show that something happened once.
A repeatable benchmark provides much stronger evidence about consistent performance.
Independent reproduction provides stronger external validation.
10.40 Claims Must Match Evidence
Avoid claims such as:
"ACAI is guaranteed intelligent."
"ACAI never makes mistakes."
"ACAI has human-level reasoning."
unless there is extraordinary evidence supporting such claims.
Prefer measurable statements:
"ACAI achieved X on dataset Y under configuration Z."
This makes the project technically credible.
10.41 Evaluation Test Flow
The complete test process is:
TEST DATASET
│
▼
LOAD TEST CASE
│
▼
ACAI RUN
│
┌───────────┼───────────┐
▼ ▼ ▼
Output Latency Logs
│ │ │
└───────────┼───────────┘
▼
Evaluator
│
▼
Score
│
▼
Aggregate Results
│
▼
Failure Analysis
│
▼
Final Report
10.42 End-to-End Evaluation
At this stage the complete ACAI development cycle becomes:
Idea
↓
Architecture
↓
Implementation
↓
Testing
↓
Evaluation
↓
Failure Analysis
↓
Improvement
↓
Re-evaluation
↓
Deployment
↓
Monitoring
This is the transition from a prototype to an engineering/research process.
10.43 Chapter 10 Success Criteria
Chapter 10 is complete when:
[✓] Evaluation dataset defined
[✓] Test categories defined
[✓] Automated runner created
[✓] Rule-based evaluation supported
[✓] Exact-match evaluation supported
[✓] Semantic evaluation approach defined
[✓] Human evaluation approach defined
[✓] Quality metrics defined
[✓] Reliability metrics defined
[✓] Latency metrics defined
[✓] Regression testing defined
[✓] Failure taxonomy defined
[✓] Reproducibility metadata defined
[✓] Benchmark reporting defined
[✓] Results are based on actual measurements
10.44 Current ACAI Architecture
USER
│
▼
API GATEWAY
│
▼
ORCHESTRATOR
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
PLANNER MEMORY RETRIEVAL
│ │ │
│ ▼ │
│ DATABASE │
└─────────────────────────┬─────────────────────────┘
▼
WORKFLOW ENGINE
│
▼
MODEL ROUTER
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Provider A Provider B Local
│ │ │
└──────────────┼──────────────┘
▼
GENERATION
│
▼
VERIFICATION
│
┌────────┴────────┐
▼ ▼
PASS REVISE
│ │
▼ ▼
RESULT RETRY
│
┌────────────┼────────────┐
▼ ▼ ▼
CACHE LOGS METRICS
│
▼
EVALUATION
│
┌────────┴────────┐
▼ ▼
Benchmark Failure Analysis
10.45 Next Chapter
ACAI now has a complete evaluation layer.
The next step is to make the system more capable of interacting with external tools and structured environments.
That leads to:
Chapter 11 — Tool Calling, Agents, External Actions, and Controlled Execution
It will cover:
Tool Interface
Tool Registry
Tool Discovery
Function Calling
Arguments Validation
Permission Control
Tool Execution
Timeouts
Sandboxing
External APIs
Agent Loops
Planning → Action → Observation
Tool Failure Recovery
Audit Logging
The central architecture will become:
User
↓
Planner
↓
Select Tool
↓
Validate Arguments
↓
Permission Check
↓
Execute
↓
Observe Result
↓
Reason
↓
Next Action / Final Answer
The critical principle will be:
An AI model should not be given unrestricted control over external actions.
Every tool should have:
Defined interface
Validated inputs
Explicit permissions
Execution limits
Timeouts
Logging
Failure handling
End of Chapter 10
Comments
Post a Comment