42.1 Introduction
The document and RAG architecture established in Chapter 41 provides the evidence layer of the AI system.
The next requirement is a reliable inference layer.
A production AI application should not allow every API route to directly call a model provider. That approach creates tight coupling, inconsistent prompts, duplicated error handling, uncontrolled costs, and weak security boundaries.
Instead, the application should introduce an AI Orchestration Layer.
The conceptual architecture is:
Application
│
▼
AI Orchestrator
│
├── Model Router
├── Prompt Manager
├── Context Manager
├── Token Manager
├── Safety Policy
├── Provider Adapter
├── Retry/Fallback Manager
├── Usage Meter
└── Observability
│
▼
Model Provider(s)
The purpose of this layer is to make model invocation controlled, observable, replaceable, and secure.
42.2 Why Direct Model Calls Are a Problem
A weak architecture might look like:
API Route
↓
AI Provider
Then another feature might implement:
Another API Route
↓
Different AI Provider
Eventually the application contains dozens of independent model calls.
This creates problems:
duplicated authentication logic,
duplicated prompts,
inconsistent safety controls,
inconsistent timeouts,
inconsistent retries,
difficult provider migration,
difficult cost tracking,
difficult testing,
inconsistent logging.
The preferred architecture is:
Feature
↓
AI Orchestrator
↓
Provider Adapter
↓
Model
Every AI feature therefore passes through a common control layer.
42.3 Provider Abstraction
The application should define a provider-neutral interface.
Conceptually:
AIProvider
├── generateText()
├── generateStructured()
├── generateEmbeddings()
└── healthCheck()
Individual providers implement the interface.
The application should not need to know the implementation details.
For example:
Provider Adapter
│
├── Cloud Model Adapter
├── Local Model Adapter
└── Backup Model Adapter
This makes the system portable.
42.4 Model Abstraction
Provider and model should be separate concepts.
For example:
Provider
└── Model
A provider may expose multiple models.
The application should therefore use an internal model identifier such as:
fast-general
quality-general
reasoning
vision
embedding
rather than hard-coding provider-specific names throughout the codebase.
The routing layer can translate internal identifiers into actual configured models.
42.5 Model Capability Registry
The orchestrator should maintain metadata about model capabilities.
Example:
ModelProfile
├── modelId
├── provider
├── capability
├── contextLimit
├── supportsVision
├── supportsStructuredOutput
├── supportsTools
├── latencyClass
└── costClass
This allows the router to select an appropriate model based on task requirements.
For example:
Image Question
↓
Vision-capable model
while:
Simple Classification
↓
Fast low-cost model
and:
Complex Reasoning
↓
High-capability model
42.6 Model Routing
Model routing determines which model should process a request.
A basic routing function can consider:
Task
User Plan
Latency Requirement
Context Size
Required Capability
Safety Requirement
Availability
Cost Budget
Conceptually:
Request
│
▼
Task Classification
│
▼
Capability Check
│
▼
Policy Check
│
▼
Model Selection
│
▼
Inference
The router should not select a model merely because it is currently popular or inexpensive.
It should satisfy the application's explicit requirements.
42.7 Task Classes
A practical system can define task classes such as:
CHAT
RAG_QA
SUMMARIZATION
CLASSIFICATION
STRUCTURED_EXTRACTION
VISION_ANALYSIS
CODE_ASSISTANCE
EMBEDDING
AGENT_PLANNING
Each task can have a default model policy.
For example:
RAG_QA
→ quality-general
CLASSIFICATION
→ fast-general
EMBEDDING
→ embedding-model
VISION_ANALYSIS
→ vision-model
The mapping should remain configurable.
42.8 Fallback Routing
External AI services can fail.
Possible causes include:
A resilient architecture can use controlled fallback routing.
Example:
Primary Model
│
├── Success → Return
│
└── Retryable Failure
↓
Backup Model
│
├── Success → Return
└── Failure → Safe Error
Fallback should never bypass authorization or safety controls.
Every provider must operate under the same application policy.
42.9 Retry Policy
Not every failure should be retried.
Retryable conditions may include certain:
transient network failures,
temporary service unavailability,
controlled rate-limit responses.
Non-retryable conditions may include:
invalid request,
invalid authentication configuration,
malformed structured output,
authorization failure,
policy rejection.
Retries should use bounded exponential backoff.
A system should also enforce a maximum retry count.
Otherwise, one failed request could generate unnecessary provider traffic and cost.
42.10 Timeouts
Every model request should have a timeout.
Conceptually:
Request
↓
Timeout Boundary
↓
Model Provider
Without a timeout, an application worker can remain occupied indefinitely.
The timeout should be appropriate for the operation.
For example:
Simple classification
→ short timeout
Long document summarization
→ longer timeout
The exact values should be configurable.
42.11 Prompt Architecture
Prompts should not be scattered throughout API handlers.
Instead, create a prompt layer.
Conceptually:
PromptRegistry
├── ChatPrompt
├── RagPrompt
├── SummaryPrompt
├── ExtractionPrompt
└── AgentPrompt
Each prompt should have:
an identifier,
a version,
an intended task,
required variables,
expected output format.
Example:
rag.answer.v3
is more manageable than an anonymous string embedded inside an API route.
42.12 Prompt Versioning
Prompts are part of application behavior.
Changing a prompt can change model output.
Therefore, prompt versions should be tracked.
Example:
rag.answer.v1
rag.answer.v2
rag.answer.v3
The system can then determine:
Which prompt produced this answer?
This is valuable for:
debugging,
evaluation,
regression testing,
research,
reproducibility.
42.13 Prompt Construction
A RAG request may contain:
System Policy
Task Instructions
User Question
Retrieved Evidence
Output Requirements
The orchestrator should construct these components explicitly.
Conceptually:
SYSTEM
↓
TASK
↓
USER INPUT
↓
RETRIEVED EVIDENCE
↓
OUTPUT CONTRACT
The exact implementation depends on the model interface.
The important architectural principle is that different content types retain their roles.
42.14 Instruction Hierarchy
The system must distinguish between trusted instructions and untrusted content.
For example:
Trusted Application Policy
↓
Application Task
↓
User Request
↓
Retrieved Documents
↓
External Tool Output
Document content, search results, webpages, uploaded files, and tool outputs should not automatically gain authority simply because they appear inside the model context.
This is particularly important for agentic systems.
42.15 User Input Is Also Data
Although the user controls their own request, the application should not blindly interpolate arbitrary user input into privileged instructions.
User input may contain:
The application should therefore separate:
Application Instructions
from:
User Content
rather than concatenating everything into one indistinguishable string.
42.16 Structured Output
Whenever an AI operation has a machine-readable result, structured output should be preferred.
Instead of asking for:
"Return the answer somehow."
the application should define an expected structure.
For example:
{
"summary": "string",
"confidence": 0.0,
"sources": []
}
The application can then validate the result before using it.
This is especially important for:
database updates,
workflow decisions,
tool calls,
document extraction,
automated actions.
42.17 Output Validation
Model output must be considered untrusted until validated.
A structured response should pass through:
Model Output
↓
Parser
↓
Schema Validator
↓
Business Rules
↓
Application
If validation fails:
Validation Failure
↓
Repair / Retry
↓
Second Validation
↓
Safe Failure
The system should never assume that a model will always obey an output schema perfectly.
42.18 Token Management
Token usage affects:
latency,
cost,
context capacity,
model reliability.
The orchestrator should calculate or estimate:
Input Tokens
Output Tokens
Total Tokens
for every inference operation where the provider exposes appropriate usage data.
A usage record can contain:
requestId
userId
projectId
model
inputTokens
outputTokens
totalTokens
duration
status
42.19 Context Budget
A request may contain:
Conversation History
+
User Question
+
Retrieved Documents
+
System Instructions
All of these consume context.
Therefore, the application should maintain a context budget.
Conceptually:
Maximum Context
├── System Policy
├── Conversation
├── Retrieved Evidence
└── Output Reserve
If the input exceeds the budget, the context manager should reduce it systematically.
Possible strategies include:
remove redundant history,
summarize older conversation,
retrieve fewer chunks,
compress evidence,
prioritize relevant sources.
42.20 Context Prioritization
Not every piece of information has equal importance.
A priority system can be:
Priority 1:
Security / system requirements
Priority 2:
Current user request
Priority 3:
Relevant retrieved evidence
Priority 4:
Recent conversation
Priority 5:
Older optional context
The exact ordering depends on application design, but it should be deterministic and testable.
42.21 Conversation Management
Long conversations can become expensive.
Instead of sending the entire conversation on every request, the system can maintain:
Recent Messages
+
Conversation Summary
+
Relevant Retrieved History
For example:
Conversation
├── Summary
├── Recent 10 messages
└── Relevant historical messages
This reduces unnecessary context consumption.
42.22 Model Context and RAG Context
Conversation context and document context should remain distinguishable.
Conceptually:
Conversation Context
+
Document Evidence
+
Current Request
The model can then understand which information came from where.
This also improves citation generation.
42.23 AI Request Lifecycle
Every AI request should pass through a controlled lifecycle:
1. Receive Request
2. Authenticate User
3. Authorize Operation
4. Validate Input
5. Apply Rate Limits
6. Determine Task Type
7. Select Model
8. Retrieve Context
9. Build Prompt
10. Apply Token Budget
11. Call Provider
12. Validate Output
13. Record Usage
14. Record Audit Event
15. Return Response
This creates a consistent security boundary.
42.24 AI Request Object
Internally, the system can normalize requests into a common structure:
AIRequest
├── requestId
├── userId
├── projectId
├── task
├── input
├── context
├── modelPolicy
├── outputSchema
└── metadata
This allows the orchestrator to process different AI features consistently.
42.25 AI Response Object
Likewise:
AIResponse
├── requestId
├── content
├── structuredData
├── model
├── usage
├── citations
├── finishReason
└── metadata
This provides a stable contract for the frontend and backend.
42.26 Usage Accounting
AI operations should be connected to the usage system established earlier.
A request can record:
User
Project
Task
Model
Input Tokens
Output Tokens
Duration
Status
This enables:
quotas,
billing,
analytics,
abuse detection,
capacity planning.
Usage records should not contain unnecessary sensitive prompt content.
42.27 Rate Limiting
AI endpoints require stronger controls than ordinary read APIs because inference can be expensive.
Rate limits can operate at multiple levels:
IP
User
Project
API Key
Task
Model
For example:
User Limit
↓
Project Limit
↓
Model Limit
↓
Global Capacity Limit
The system should return a controlled response when a limit is exceeded.
42.28 Budget Controls
A project may have a monthly AI budget.
The orchestrator can check:
Current Usage
+
Estimated Request Cost
≤
Allowed Budget
before starting expensive operations.
For exact cost accounting, the implementation should use provider-reported usage and configured pricing data where applicable.
42.29 Safety Policy Layer
Safety controls should not depend entirely on the model.
The application can have a policy layer that determines whether a requested operation is permitted.
Conceptually:
Request
↓
Application Policy
↓
Model Policy
↓
Provider
The policy layer can enforce application-specific restrictions such as:
42.30 Tool Permissions
When the AI system eventually gains access to tools, model output must not automatically execute arbitrary actions.
Instead:
Model Request
↓
Tool Authorization
↓
Parameter Validation
↓
Approval Policy
↓
Tool Execution
For sensitive operations, human approval can be required.
This creates a crucial separation:
Model proposes
versus:
Application authorizes
42.31 Agentic Inference
A future agent system may perform multiple inference steps:
Goal
↓
Plan
↓
Retrieve
↓
Reason
↓
Tool Proposal
↓
Verification
↓
Action
Each step should be separately observable.
The model should not receive unrestricted authority over the entire application.
Instead, the orchestrator should provide bounded capabilities.
42.32 Model Independence
One major benefit of orchestration is model independence.
The application can evolve from:
One Model
to:
Multiple Models
without rewriting every feature.
The architecture becomes:
Feature
↓
Task
↓
Model Policy
↓
Router
↓
Selected Model
This is particularly valuable as model capabilities and pricing change over time.
42.33 Local and Remote Models
The abstraction layer can support both remote and locally hosted models.
Conceptually:
AI Orchestrator
│
├── Remote Provider
│
└── Local Provider
This can support different deployment strategies.
For sensitive workloads, an organization may choose a locally hosted model.
For high-capability workloads, it may use a remote model.
The application architecture should remain consistent.
42.34 Provider Health Monitoring
The router should maintain basic provider health information.
Possible states:
HEALTHY
DEGRADED
RATE_LIMITED
UNAVAILABLE
DISABLED
A provider that repeatedly fails can temporarily be removed from normal routing.
This prevents every request from repeatedly encountering the same failure.
42.35 Circuit Breaker Concept
A circuit breaker can protect the application from an unhealthy dependency.
Conceptually:
Normal
↓
Repeated Failures
↓
Open Circuit
↓
Stop Requests Temporarily
↓
Health Check
↓
Recovery
↓
Normal
This should be carefully implemented so that a provider outage does not cascade into application-wide resource exhaustion.
42.36 Caching
Some AI operations may be safely cacheable.
Examples could include deterministic operations where the same input and configuration produce reusable results.
A cache key might include:
task
model
promptVersion
inputHash
contextHash
configurationVersion
Sensitive data should not be placed into shared caches without appropriate isolation.
Caching should never bypass authorization.
42.37 Determinism and Reproducibility
Model outputs can vary.
For research and testing, the system should record relevant configuration.
For example:
model
promptVersion
temperature
maxOutputTokens
retrievalVersion
embeddingVersion
applicationVersion
Not every provider exposes identical controls, so the architecture should record what is actually available.
This makes experiments easier to reproduce.
42.38 AI Audit Trail
The system should maintain an AI operation record.
A conceptual audit entry:
AI Audit Event
├── requestId
├── userId
├── projectId
├── task
├── model
├── promptVersion
├── retrievalVersion
├── status
├── duration
└── timestamp
The audit record should avoid storing full sensitive prompts or outputs unless explicitly required and appropriately protected.
42.39 Error Classification
AI failures should be categorized.
Example:
INPUT_VALIDATION_ERROR
AUTHORIZATION_ERROR
RATE_LIMIT_ERROR
PROVIDER_TIMEOUT
PROVIDER_UNAVAILABLE
MODEL_OUTPUT_INVALID
CONTEXT_LIMIT_ERROR
POLICY_REJECTION
INTERNAL_ERROR
This allows the frontend to provide meaningful responses without exposing internal implementation details.
42.40 Secure Error Messages
The system should not expose:
A user-facing response should be safe.
Internally:
Detailed diagnostic
Externally:
Controlled error
This follows the same security boundary established for the API layer.
42.41 AI Service Architecture
The resulting backend architecture can now be:
src/
├── app/
│ └── api/
├── services/
│ ├── ai/
│ │ ├── orchestrator
│ │ ├── router
│ │ ├── prompts
│ │ ├── context
│ │ ├── token-manager
│ │ ├── safety
│ │ ├── usage
│ │ └── providers/
│ ├── documents/
│ ├── retrieval/
│ └── storage/
├── repositories/
└── lib/
This keeps model-specific implementation away from business-facing API routes.
42.42 Example AI Request Flow
A RAG question might follow:
User
↓
POST /api/ai/ask
↓
Authenticate
↓
Authorize Project
↓
Validate Question
↓
Check Usage Limit
↓
Retrieve Authorized Evidence
↓
Select RAG Model
↓
Build Versioned Prompt
↓
Apply Context Budget
↓
Call Provider
↓
Validate Response
↓
Attach Citations
↓
Record Usage
↓
Audit
↓
Return Answer
This flow integrates the architecture established in previous chapters.
42.43 Testing Strategy
The orchestration layer requires multiple levels of testing.
Unit Tests
Test:
model routing,
token budgeting,
prompt construction,
output validation,
retry decisions.
Integration Tests
Test:
Failure Tests
Simulate:
timeout,
provider unavailable,
malformed output,
rate limit,
context overflow.
Security Tests
Test:
unauthorized project access,
cross-user retrieval,
untrusted document instructions,
tool authorization,
sensitive error leakage.
42.44 Evaluation Matrix
A useful evaluation matrix is:
Task
Model
Prompt Version
Retrieval Version
Expected Quality
Actual Quality
Latency
Token Usage
Failure Rate
This enables systematic comparison.
For example:
RAG-QA
Model A
Prompt v3
Retriever v2
Accuracy: ...
Latency: ...
The exact metrics should be determined by the research objective.
42.45 Operational Dashboard
The system can expose metrics such as:
AI Requests / Minute
Success Rate
Failure Rate
Average Latency
P95 Latency
Input Tokens
Output Tokens
Provider Errors
Fallback Rate
Retrieval Latency
Model Distribution
Sensitive user content should not be displayed in operational dashboards.
42.46 Configuration Management
AI configuration should be environment-controlled.
Examples include:
AI_DEFAULT_MODEL
AI_TIMEOUT_MS
AI_MAX_RETRIES
AI_MAX_CONTEXT_TOKENS
AI_MAX_OUTPUT_TOKENS
AI_DEFAULT_TEMPERATURE
AI_PROVIDER_MODE
Secrets should remain outside source control.
Configuration should be validated when the application starts.
42.47 No Secrets in Prompts or Logs
API keys, passwords, session tokens, and private credentials must never be intentionally inserted into model prompts.
Similarly, logs should avoid storing credentials or unnecessary sensitive content.
The AI layer should assume that model providers and logging systems are separate trust boundaries.
42.48 Data Minimization
Before sending information to a model, the application should ask:
Does the model actually need this data?
If not, it should be omitted.
For example:
Full User Profile
may not be necessary for:
Simple Document Summarization
Reducing unnecessary context improves:
privacy,
latency,
cost,
security.
42.49 The AI Orchestrator as a Security Boundary
The orchestrator should become the central checkpoint for AI operations.
AI REQUEST
│
▼
┌──────────────┐
│ Orchestrator │
└──────────────┘
│ │ │
┌───────┘ │ └────────┐
▼ ▼ ▼
Authorization Routing Safety
│ │ │
└────────────┼─────────────┘
▼
Context
│
▼
Model
│
▼
Validation
│
▼
Application
This architecture makes AI behavior significantly easier to control.
42.50 Complete AI Architecture
Combining previous chapters produces the following architecture:
USER
│
▼
API
│
▼
Authentication/AuthZ
│
▼
AI Orchestrator
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
Model Router Context Manager Safety Policy
│ │ │
▼ ▼ │
Provider Adapter RAG Retrieval │
│ │ │
└──────────────┬───┴──────────────────┘
▼
Prompt
│
▼
Token Manager
│
▼
Selected Model
│
▼
Output Validation
│
┌─────────┴─────────┐
▼ ▼
Usage Audit
│ │
└─────────┬─────────┘
▼
Response
42.51 Architectural Principles
Principle 1 — Centralize AI access
Features should call the orchestrator rather than providers directly.
Principle 2 — Keep providers replaceable
Provider-specific details belong inside adapters.
Principle 3 — Separate model selection from application logic
Use task and capability policies.
Principle 4 — Treat prompts as versioned software artifacts
Prompt changes should be traceable.
Principle 5 — Control context
Never send unnecessary information to the model.
Principle 6 — Validate model output
Model-generated data is not automatically trusted.
Principle 7 — Enforce authorization before retrieval
The model must never become an access-control mechanism.
Principle 8 — Keep tools behind authorization
Model proposals are not automatically authorized actions.
Principle 9 — Record usage and operational metadata
AI behavior must be measurable.
Principle 10 — Design for failure
Provider outages, timeouts, rate limits, and malformed outputs are expected conditions.
42.52 Chapter Summary
The system now contains a dedicated AI inference architecture.
The major components are:
AI Orchestrator
Model Router
Provider Adapters
Prompt Registry
Context Manager
Token Manager
Safety Policy
Output Validator
Usage Meter
Audit Layer
Observability
The resulting lifecycle is:
Request → Authenticate → Authorize → Validate → Route → Retrieve → Build Context → Construct Prompt → Infer → Validate → Meter → Audit → Respond
This architecture prevents AI providers from becoming deeply embedded throughout the application.
It also establishes the foundation for the next stage of the project: agent architecture, tool calling, planning, execution boundaries, approvals, task state, and safe autonomous workflows.
END OF CHAPTER 42
Implementation snippet — provider abstraction
Implementation snippet — simple model router
Implementation snippet — orchestrator skeleton
Comments
Post a Comment