12.1 Objective
A capable AI system needs more than a model and a database.
ACAI must be able to decide:
What should be remembered?
What should be retrieved?
What should be ignored?
What information is relevant now?
What information is outdated?
How should conflicting memories be handled?
How much context should be sent to the model?
The architecture becomes:
USER / EXPERIENCE
↓
MEMORY EXTRACTION
↓
MEMORY STORAGE
↓
RETRIEVAL
↓
RELEVANCE RANKING
↓
CONTEXT COMPRESSION
↓
MODEL
↓
NEW EXPERIENCE
↓
MEMORY UPDATE
12.2 Why Memory Is Different From Context
Context is information currently supplied to the model.
Memory is information intentionally retained for future use.
For example:
Context:
"What are we working on today?"
Memory:
"The project uses Next.js and TypeScript."
A robust system should not treat every conversation message as permanent memory.
12.3 Memory Layers
ACAI can divide memory into several layers:
┌───────────────────────────────┐
│ WORKING MEMORY │
├───────────────────────────────┤
│ SHORT-TERM CONTEXT │
├───────────────────────────────┤
│ EPISODIC MEMORY │
├───────────────────────────────┤
│ SEMANTIC MEMORY │
├───────────────────────────────┤
│ KNOWLEDGE STORE │
└───────────────────────────────┘
Each layer serves a different purpose.
12.4 Working Memory
Working memory contains information required for the current task.
Example:
Task:
Build a website.
Working memory:
- current page
- current files
- current errors
- current objective
- active tool results
This information can disappear when the task ends.
12.5 Short-Term Context
Short-term context contains recent interaction history.
Example:
User:
Create a dashboard.
Assistant:
Created dashboard structure.
User:
Add authentication.
Assistant:
Authentication added.
Recent messages can be useful without necessarily becoming permanent memories.
12.6 Episodic Memory
Episodic memory records meaningful events.
Example:
Event:
User completed project deployment.
Timestamp:
2026-08-30
Context:
Production deployment.
Result:
Deployment succeeded.
This represents an event rather than a general fact.
12.7 Semantic Memory
Semantic memory represents generalized information.
Example:
Fact:
The project uses TypeScript.
Unlike an episode:
"Yesterday the user created a TypeScript file."
semantic memory stores the generalized knowledge.
12.8 Knowledge Store
A knowledge store can contain larger external information:
Documentation
Research papers
Project files
Databases
Reference material
Structured records
This should generally remain separate from personal conversational memory.
12.9 Memory Extraction
Not every message should be stored.
Create a memory extraction stage:
Conversation
↓
Memory Extractor
↓
Candidate Memories
↓
Validation
↓
Importance Score
↓
Store / Reject
Example:
class MemoryCandidate:
def __init__(
self,
content,
memory_type,
importance,
):
self.content = content
self.memory_type = memory_type
self.importance = importance
12.10 Memory Importance
A memory can receive an importance score.
For example:
0 → irrelevant
1 → low importance
2 → moderate
3 → useful
4 → important
5 → highly important
The exact scoring system should be validated experimentally.
12.11 Memory Storage
A basic memory record:
{
"id": "memory_001",
"type": "semantic",
"content": "Project uses TypeScript.",
"importance": 4,
"created_at": "2026-08-30T00:00:00Z"
}
A production system may additionally store:
embedding
source
confidence
last_accessed
updated_at
scope
expiration
12.12 Memory Scope
Memories should have scope.
Possible scopes:
conversation
task
project
user
organization
system
Example:
Project scope:
"Project uses Next.js."
should not automatically become:
System-wide fact.
Scope prevents accidental information leakage between contexts.
12.13 Memory Retrieval
When a new request arrives:
User Request
↓
Query Generation
↓
Memory Search
↓
Candidate Memories
↓
Ranking
↓
Relevant Memories
↓
Context
Only relevant memories should enter the model context.
12.14 Vector Embeddings
Semantic retrieval can use embeddings.
Conceptually:
Text
↓
Embedding Model
↓
Vector
Example:
"TypeScript project"
↓
[0.12, -0.08, 0.41, ...]
A query is also converted into a vector.
Then the system searches for similar vectors.
12.15 Similarity Search
Conceptually:
Query Vector
│
▼
Vector Database
│
├── Memory A → high similarity
├── Memory B → medium similarity
└── Memory C → low similarity
The highest-ranked relevant memories can then be supplied to the model.
12.16 Hybrid Retrieval
Vector similarity alone is not always sufficient.
ACAI can combine:
Vector Search
+
Keyword Search
+
Metadata Filtering
+
Recency
+
Importance
Architecture:
Query
│
┌──────────┼──────────┐
▼ ▼ ▼
Vector Keyword Metadata
Search Search Filter
│ │ │
└──────────┼──────────┘
▼
Re-ranking
│
▼
Final Context
12.17 Relevance Score
A conceptual ranking function can combine several signals:
Relevance
=
semantic similarity
+
importance
+
recency
+
scope match
The exact weights should be tuned using evaluation data rather than assumed to be universally optimal.
12.18 Recency
Recent information can sometimes be more useful.
Example:
Memory A:
Updated yesterday.
Memory B:
Updated two years ago.
If both concern a changing project configuration, Memory A may deserve a higher ranking.
However, recency should not automatically override correctness.
12.19 Memory Conflicts
Suppose memory contains:
Memory A:
Project uses framework X.
Memory B:
Project migrated to framework Y.
ACAI needs conflict resolution.
Possible strategy:
New information
↓
Conflict Detection
↓
Compare timestamps
↓
Compare source reliability
↓
Compare confidence
↓
Update / Preserve / Flag
12.20 Memory Versioning
Instead of silently deleting old information:
Version 1
↓
Version 2
↓
Version 3
This can provide an audit trail.
For example:
Project framework:
v1 = framework X
v2 = framework Y
The latest valid state can be used for current tasks while historical information remains available where appropriate.
12.21 Memory Confidence
A memory should not necessarily be treated as absolute truth.
Example:
{
"content": "Project uses framework X.",
"confidence": 0.92
}
Confidence should reflect the evidence available, not simply model-generated certainty.
12.22 Memory Sources
Useful source metadata:
user_statement
uploaded_document
tool_result
system_configuration
verified_database
model_inference
The source allows ACAI to distinguish:
Verified fact
from:
Model inference
12.23 Context Budget
A model has a finite context capacity or practical context budget.
Suppose the system retrieves:
100 documents
Sending everything to the model may be inefficient.
Instead:
100 candidates
↓
Ranking
↓
20 relevant
↓
Compression
↓
5 useful sections
↓
Model
12.24 Context Compression
Compression can be performed using:
Summarization
Deduplication
Extraction
Structured representation
Relevance filtering
The goal is:
Maximum useful information
with minimum unnecessary context
12.25 Deduplication
Suppose retrieval returns:
Document A:
Project uses TypeScript.
Document B:
The project is written in TypeScript.
Document C:
TypeScript is used throughout the project.
These may contain redundant information.
ACAI can consolidate them:
Project uses TypeScript.
This reduces context waste.
12.26 Long Documents
Large documents should not automatically be sent to the model in full.
Pipeline:
Large Document
↓
Parsing
↓
Structure Detection
↓
Chunking
↓
Embedding
↓
Indexing
↓
Retrieval
↓
Relevant Chunks
↓
Re-ranking
↓
Context
12.27 Chunking
A document can be divided by:
Paragraph
Section
Heading
Page
Semantic boundary
Token limit
Semantic boundaries are generally preferable to blindly splitting every fixed number of characters.
12.28 Chunk Metadata
Each chunk should retain metadata:
{
"document_id": "doc_001",
"chunk_id": "chunk_017",
"section": "Architecture",
"page": 12,
"text": "..."
}
This makes citations and source tracing possible.
12.29 Retrieval With Citations
A strong knowledge system should preserve source references:
Retrieved Chunk
↓
Source Metadata
↓
Model
↓
Answer
↓
Citation
The model should not fabricate source locations.
12.30 Knowledge Graph
Some information is naturally represented as relationships.
Example:
Project
│
├── uses → TypeScript
├── uses → Next.js
├── contains → API
└── deployed_on → Server
A graph representation can store:
Entity
Relationship
Entity
For example:
(Project) ──uses──> (Next.js)
12.31 Hybrid Knowledge Architecture
ACAI can combine:
SQL / Document Database
+
Vector Database
+
Knowledge Graph
+
Object Storage
Each system serves a different purpose.
Structured facts → Database
Semantic search → Vector index
Relationships → Graph
Large files → Object storage
12.32 Memory Consolidation
Memory can be periodically consolidated.
Example:
100 small memories
↓
Group related memories
↓
Remove duplicates
↓
Resolve updates
↓
Create durable summary
↓
Store consolidated memory
This resembles a maintenance process rather than a one-time operation.
12.33 Memory Decay
Not every memory remains equally useful forever.
A conceptual decay mechanism:
High importance
↓
slow decay
Low importance
↓
faster decay
But important information should not simply disappear because it is old.
Instead, decay can influence retrieval priority.
12.34 Memory Lifecycle
Candidate
↓
Validated
↓
Stored
↓
Retrieved
↓
Updated
↓
Consolidated
↓
Archived / Expired
Every transition should be explicit.
12.35 Memory API
Create:
app/memory/service.py
Example:
class MemoryService:
async def add(
self,
memory,
):
...
async def search(
self,
query,
limit=10,
):
...
async def update(
self,
memory_id,
data,
):
...
async def archive(
self,
memory_id,
):
...
This keeps memory operations separate from the model layer.
12.36 Retrieval Service
Create:
app/retrieval/service.py
class RetrievalService:
async def retrieve(
self,
query,
filters=None,
limit=10,
):
candidates = (
await self.search(
query,
filters,
)
)
ranked = self.rank(
candidates
)
return ranked[:limit]
12.37 Context Builder
The model should receive a carefully constructed context.
class ContextBuilder:
def build(
self,
request,
memories,
documents,
tool_results,
):
return {
"request": request,
"memories": memories,
"documents": documents,
"tool_results": tool_results,
}
A production implementation would apply token budgeting and prioritization.
12.38 Context Priority
When context is limited, ACAI can prioritize:
1. Current user request
2. Required system instructions
3. Verified relevant information
4. Current tool results
5. High-value memories
6. Supporting documents
7. Lower-priority historical context
The exact ordering depends on the application.
12.39 Memory Security
Memory systems can accidentally retain sensitive information.
Therefore:
Memory Input
↓
Classification
↓
Policy Check
↓
Redaction / Rejection
↓
Storage
Sensitive information should only be retained when there is a legitimate reason and appropriate authorization.
12.40 User Memory Controls
A mature system should support:
View memory
Delete memory
Correct memory
Disable memory
Export memory
This gives users meaningful control over persistent information.
12.41 Memory Isolation
Different users must not share private memories accidentally.
Architecture:
User A
↓
User A memory namespace
User B
↓
User B memory namespace
Queries must always apply the correct authorization and scope filters.
12.42 Retrieval Security
Security must occur before retrieval results enter the model.
Query
↓
Authorization
↓
Allowed Data Scope
↓
Retrieval
↓
Permission Filtering
↓
Context
Do not rely solely on the model to decide whether retrieved information is allowed.
12.43 Prompt Injection in Retrieved Data
External documents can contain instructions such as:
"Ignore previous instructions..."
Retrieved content should be treated as data, not automatically as trusted instructions.
The architecture should maintain a distinction between:
System instructions
Developer instructions
User instructions
Retrieved content
Tool output
These sources should not be treated as equivalent authority.
12.44 Long-Context Strategy
For very large tasks:
Document Set
↓
Map
↓
Summarize relevant sections
↓
Reduce
↓
Cross-reference
↓
Final reasoning
Rather than:
Everything
↓
One giant prompt
This can reduce cost and improve focus.
12.45 Hierarchical Summarization
Large content can be summarized in layers:
Documents
↓
Section Summaries
↓
Document Summaries
↓
Topic Summary
↓
Global Summary
When a detailed answer is required, ACAI can retrieve the underlying source chunks instead of relying only on the summary.
12.46 Retrieval Evaluation
Memory and retrieval must themselves be evaluated.
Useful metrics include:
Recall@K
Precision@K
MRR
NDCG
Answer groundedness
Citation correctness
The exact metrics should match the retrieval task.
12.47 Retrieval Test
Example:
Query:
"Which framework does the project use?"
Expected document:
project_architecture.md
Evaluation:
Retrieved top 5:
1. project_architecture.md
2. ...
If the expected source appears within the top 5:
Recall@5 = success for this query
Aggregate this across a test dataset for a benchmark.
12.48 Memory Evaluation
Test:
Question
↓
Relevant memory exists?
↓
Was it retrieved?
↓
Was it correctly interpreted?
↓
Did it improve the answer?
This separates retrieval quality from answer quality.
12.49 Full Memory Pipeline
EXPERIENCE
│
▼
MEMORY EXTRACTOR
│
▼
POLICY / VALIDATOR
│
▼
MEMORY STORE
│
┌─────────┼─────────┐
▼ ▼ ▼
Vector Metadata Graph
Index Store Store
│ │ │
└─────────┼─────────┘
▼
QUERY
│
▼
RETRIEVAL
│
▼
RE-RANKING
│
▼
CONTEXT BUILDER
│
▼
MODEL
│
▼
RESPONSE
12.50 Combined ACAI Architecture
At this stage, ACAI becomes:
USER
│
▼
API GATEWAY
│
▼
ORCHESTRATOR
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
PLANNER MEMORY RETRIEVAL
│ │ │
│ ▼ ▼
│ Memory Service Search Engine
│ │ │
│ ┌──────────┼──────────┐ │
│ ▼ ▼ ▼ │
│ Vector Metadata Graph │
│ Index Store Store │
│ └──────────┼──────────┘ │
│ │ │
└─────────────────────────┼─────────────────────────┘
▼
CONTEXT BUILDER
│
▼
WORKFLOW ENGINE
│
▼
AGENT CONTROLLER
│
▼
MODEL ROUTER
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Provider A Provider B Local
│ │ │
└─────────────┼─────────────┘
▼
TOOL REGISTRY
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Calculator Search Database
│ │ │
└─────────────┼─────────────┘
▼
OBSERVATION
│
▼
VERIFIER
│
┌─────────┴─────────┐
▼ ▼
PASS REVISE
│ │
▼ ▼
RESULT RETRY
│
┌─────────┼─────────┐
▼ ▼ ▼
CACHE LOGS METRICS
│
▼
EVALUATION
│
▼
IMPROVEMENT LOOP
12.51 Chapter 12 Success Criteria
[✓] Working memory defined
[✓] Short-term context defined
[✓] Episodic memory defined
[✓] Semantic memory defined
[✓] Knowledge store defined
[✓] Memory extraction defined
[✓] Memory importance defined
[✓] Memory scope defined
[✓] Memory retrieval defined
[✓] Vector retrieval defined
[✓] Hybrid retrieval defined
[✓] Re-ranking defined
[✓] Context compression defined
[✓] Document chunking defined
[✓] Knowledge graph architecture defined
[✓] Memory conflict resolution defined
[✓] Memory versioning defined
[✓] Memory security defined
[✓] User memory controls defined
[✓] Retrieval evaluation defined
12.52 Next Chapter
The next major step is to move from individual components toward production-grade infrastructure.
Chapter 13 will cover:
Distributed Architecture
Service Boundaries
Queues
Workers
Caching
Database Scaling
Observability
Tracing
Rate Limiting
Authentication
Authorization
Secrets
Configuration
Health Checks
Deployment
High Availability
Disaster Recovery
The target transformation is:
Prototype
↓
Reliable Application
↓
Production System
↓
Scalable AI Platform
End of Chapter 12
Comments
Post a Comment