12.1 Introduction
The ACAI system requires a reliable infrastructure layer capable of storing users, sessions, memories, documents, embeddings, model configurations, evaluation results, and system events.
The infrastructure should be designed around three principles:
-
Separation of responsibilities
-
Security and controlled access
-
Scalability without unnecessary complexity
A practical prototype can begin with a small number of services and databases. As usage increases, individual components can be separated and scaled independently.
12.2 High-Level Infrastructure
12.3 Database Architecture
ACAI should not treat every piece of information as the same kind of data.
A practical architecture separates:
Each storage layer has a different purpose.
12.4 Relational Database
The relational database stores structured application information.
Possible tables:
Example:
Relationships should be explicitly defined.
12.5 Conversation Storage
A conversation can be represented as:
A message record might contain:
However, sensitive conversation data should not be retained indefinitely without a legitimate reason and appropriate user controls.
12.6 Memory Database
Long-term memory requires additional metadata.
Example:
Possible memory types:
The system should distinguish between information explicitly provided by the user and information inferred by the model.
12.7 Vector Database
Semantic retrieval requires embeddings.
Conceptually:
Each vector should retain a reference to its source.
This allows retrieved information to be traced back to its original document.
12.8 Document Storage
Large files should generally be stored separately from the relational database.
Examples:
The database stores metadata:
while the actual file remains in object storage or an equivalent file-storage system.
12.9 Cache Layer
A cache can store temporary or frequently accessed information.
Possible cache entries:
The cache should never become the only copy of information that must persist.
12.10 API Architecture
The API should expose clearly separated resources.
Example:
Each API should validate requests before passing them to internal services.
12.11 Chat API
Conceptually:
Input:
Processing:
The actual production API should also enforce rate limits and authorization.
12.12 Document API
A document pipeline could use:
Upload flow:
12.13 Retrieval API
Example:
The retrieval system can accept:
The service then returns ranked results.
12.14 Model API
The Model Service abstracts the underlying foundation models.
The application should avoid hard-coding the rest of the system to one specific model provider.
Conceptually:
This makes model replacement easier.
12.15 Model Registry
The Model Registry stores information about available models.
Example:
The router can use this information when making model-selection decisions.
12.16 Task API
Complex requests may create multiple tasks.
Example:
Each task can have its own status:
12.17 Evaluation API
The evaluation system needs its own interface.
Possible endpoints:
This enables automated benchmark runs without manually interacting with the application.
12.18 Authentication
The API should authenticate every protected request.
Conceptually:
Authentication and authorization are separate:
Authentication
Who is the user?
Authorization
What is the user allowed to access?
12.19 Authorization
A user should only access resources they are permitted to access.
Example:
Authorization should be enforced on the backend, not merely hidden in the frontend.
12.20 Rate Limiting
Without rate limiting, a single client could consume excessive resources.
Example:
Different limits may apply to:
-
Authentication
-
Chat
-
File uploads
-
Search
-
Model generation
-
Evaluation jobs
12.21 Queue Infrastructure
Long-running tasks should not necessarily block normal API requests.
Example:
Workers can process:
-
Document ingestion
-
Embedding generation
-
Large evaluations
-
Video processing
-
Batch inference
12.22 Worker Architecture
A worker receives a job:
Failed jobs can be retried according to a controlled retry policy.
12.23 GPU Infrastructure
If local or self-hosted models are used, GPU resources become important.
Conceptually:
The router should know which models are available on which nodes.
12.24 Horizontal Scaling
When demand increases, multiple API instances can operate simultaneously.
Stateless API services are easier to scale horizontally.
Persistent state should remain in shared storage systems.
12.25 Reliability
A production system should avoid single points of failure where practical.
Example:
If one API instance fails, the others can continue serving traffic.
The same principle can be applied to model servers and workers when resources justify it.
12.26 Backup & Recovery
Critical data should have a recovery strategy.
A backup that has never been tested should not automatically be considered recoverable.
Important recovery metrics include:
RPO — Recovery Point Objective
How much data can potentially be lost?
RTO — Recovery Time Objective
How quickly should service be restored?
12.27 Monitoring Infrastructure
The infrastructure should monitor:
A centralized dashboard can provide an operational overview.
12.28 Infrastructure Security
Security should exist at multiple layers.
Sensitive credentials should be stored in secure secret-management systems rather than source code.
12.29 Development Environment
A practical development environment can initially contain:
Development and production configurations should be separated.
12.30 Production Environment
A production deployment may eventually look like:
12.31 Practical Scaling Strategy
Do not begin with the full production architecture.
A realistic progression is:
The system should only move to the next stage when actual workload requires it.
12.32 Cost Optimization
Infrastructure should be optimized according to actual measurements.
Possible strategies:
-
Cache repeated operations
-
Use smaller models for simple tasks
-
Batch embedding operations
-
Compress retrieved context
-
Scale workers according to demand
-
Shut down unused development resources
The objective is not simply minimizing cost, but finding an appropriate quality–latency–cost balance.
12.33 Infrastructure Testing
Before production, test:
A system that works with one request but fails under 100 simultaneous requests is not production-ready.
12.34 Final Infrastructure Blueprint
12.35 Chapter Summary
The infrastructure layer provides the foundation on which the ACAI software architecture operates. A practical implementation should separate structured data, vector data, large files, cache data, and asynchronous workloads according to their specific requirements.
The most important engineering principle is progressive scaling. A prototype should begin with the smallest infrastructure capable of validating the research hypothesis. More complex infrastructure should be introduced only when measurements demonstrate a need for it.
This makes the proposed architecture technically achievable without requiring the initial project to operate as a massive AI infrastructure company.
End of Chapter 12
Comments
Post a Comment