109.1 Introduction
Modern AI platforms depend on many forms of machine-to-machine authentication. Backend services communicate with databases, workers communicate with queues, applications communicate with AI providers, CI/CD pipelines deploy infrastructure, and background jobs access storage and external APIs.
These interactions require credentials.
Examples include:
A credential is effectively a digital capability. Possession of a sufficiently privileged credential may allow a system or attacker to perform actions without needing a user's password.
Therefore, credential security must be treated as a first-class architectural concern.
The objective is not simply to hide secrets.
A secure credential architecture should provide:
strong credential generation
secure storage
minimal permissions
explicit ownership
controlled distribution
rotation
expiration
revocation
monitoring
auditability
isolation
recovery
The platform should also minimize the number of long-lived static credentials it requires.
109.2 Credential Security Principles
A secure machine-identity architecture should follow several fundamental principles.
Principle 1 — Least Privilege
Every credential should receive only the permissions required for its intended task.
For example:
A video-processing worker that needs to read uploaded media should not automatically receive permission to delete every object in storage.
Similarly, a reporting service that reads analytics data should not have write access to authentication tables.
Principle 2 — Short Credential Lifetime
Long-lived credentials increase the impact of compromise.
Where possible, credentials should be:
short-lived
automatically issued
automatically renewed
automatically revoked
Temporary credentials are generally preferable to permanent secrets.
Principle 3 — Explicit Identity
Every machine should have a recognizable identity.
Instead of:
unknown-service
use identities such as:
api-service
media-worker
ai-orchestrator
notification-worker
billing-service
backup-service
deployment-runner
This allows security systems to determine:
Principle 4 — Credential Separation
Different environments must use different credentials.
For example:
development
staging
production
Production credentials must never be reused in development.
Similarly, one service should not automatically reuse another service's credentials.
109.3 Credential Classification
Credentials should be classified before being stored or distributed.
109.3.1 Public Identifiers
Examples:
client IDs
application IDs
public keys
service names
These may sometimes be visible to clients.
However, public identifiers must not be confused with secrets.
109.3.2 Sensitive Secrets
Examples:
API keys
database passwords
private keys
webhook secrets
signing secrets
OAuth client secrets
These must remain confidential.
109.3.3 High-Privilege Credentials
Examples:
production administrator credentials
root cloud credentials
deployment credentials
infrastructure-management credentials
encryption key-management credentials
These require stronger controls.
They should normally have:
restricted access
short lifetime where possible
strong auditing
multi-person approval for sensitive operations
emergency revocation procedures
109.4 API Key Architecture
API keys are commonly used for machine authentication.
A basic architecture is:
Client
|
| API Key
v
API Gateway
|
v
Authentication Layer
|
v
Authorization Layer
|
v
Application Service
The API key should not automatically represent unlimited access.
Instead, each key should have:
109.5 API Key Format
API keys should be generated using a cryptographically secure random source.
A conceptual structure could be:
prefix + random secret
For example:
ax_live_<random-secret>
The prefix can help identify:
platform
environment
credential category
The secret portion must contain sufficient randomness.
The system should never generate secrets using predictable values such as:
username + timestamp
or:
userID + random number
109.6 Storing API Keys
A major security rule is:
Do not store recoverable API secrets in ordinary database columns unless there is a specific architectural requirement and appropriate protection.
Where verification is possible without recovering the original secret, the platform can store a cryptographic hash or derived verification value.
A database record may contain:
key_id
key_prefix
key_hash
owner_id
scopes
created_at
expires_at
revoked_at
last_used_at
The complete secret is shown to the user only when appropriate, typically immediately after creation.
The platform should not display the full secret later.
109.7 API Key Scoping
Keys should have explicit scopes.
Example:
media:read
media:write
generation:create
generation:read
billing:read
admin:read
A media-processing worker might receive:
media:read
media:write
but not:
billing:write
admin:write
user:delete
This reduces blast radius.
109.8 API Key Lifecycle
A secure lifecycle is:
Generate
↓
Register
↓
Store securely
↓
Distribute
↓
Use
↓
Monitor
↓
Rotate
↓
Expire or Revoke
↓
Destroy
Every stage should be controlled.
109.9 Credential Rotation
Credential rotation replaces an existing credential with a new one.
A safe rotation process should support overlap.
Example:
Old Credential
|
| still valid temporarily
v
New Credential Created
|
v
Consumers Updated
|
v
Old Credential Revoked
This avoids unnecessary downtime.
The rotation system should verify that dependent services successfully use the new credential before revoking the old one.
109.10 Credential Revocation
Credentials must be revocable.
Revocation may be triggered by:
Revocation should ideally be fast.
For high-risk credentials, emergency revocation procedures should be tested regularly.
109.11 Service Accounts
A service account represents a machine or application rather than a human user.
Example:
media-worker
could represent all worker instances belonging to a specific application component.
A service account should have:
unique identity
defined purpose
assigned owner
permission policy
environment
lifecycle
audit trail
Service accounts should not be shared unnecessarily between unrelated applications.
109.12 Service-to-Service Authentication
Internal services should authenticate each other.
A secure request may conceptually look like:
API Service
|
| authenticated request
v
AI Orchestrator
|
| authenticated request
v
Media Worker
The receiving service should verify:
caller identity
credential validity
credential expiration
intended audience
authorization
request context
Authentication alone is not sufficient.
The service must also determine whether that identity is authorized to perform the requested operation.
109.13 Internal Service Tokens
Internal tokens can be used for service-to-service communication.
A secure token should contain or represent:
Short-lived tokens are preferable to permanent internal secrets.
109.14 Workload Identity
Modern cloud environments can provide workload identities that allow applications to obtain permissions without storing static cloud credentials.
Conceptually:
Workload
|
v
Workload Identity
|
v
Cloud Identity System
|
v
Temporary Credential
|
v
Cloud Resource
This can reduce secret-management complexity.
For example, an AI worker may receive temporary permission to access an object-storage bucket instead of carrying a permanent storage password.
109.15 Cloud IAM
Cloud IAM should follow least privilege.
Instead of granting:
Administrator
to an application, assign only the required permissions.
For example:
ObjectStorage.Read
Queue.Receive
Database.Connect
rather than:
*
Wildcard permissions should be carefully reviewed because they can create unexpectedly large privileges.
109.16 Database Credentials
Database credentials should be separated by service.
For example:
api-db-user
worker-db-user
analytics-db-user
migration-db-user
These accounts should not all receive the same permissions.
Application services should normally have limited database privileges.
Migration operations may require elevated privileges but should be performed through controlled deployment workflows rather than ordinary application connections.
109.17 Storage Credentials
Object storage access should also be scoped.
A media worker might receive permission to:
read uploads
write processed-media
but not:
delete backups
read private billing exports
modify security logs
Where supported, temporary signed URLs or temporary credentials can reduce the need to distribute permanent storage secrets.
109.18 AI Provider Credentials
AI platforms frequently integrate with external providers.
Examples include:
Provider credentials should remain on trusted backend infrastructure.
They should generally not be embedded directly in browser JavaScript.
A safer architecture is:
Browser
|
| user request
v
Backend
|
| provider credential
v
AI Provider
The browser should authenticate to the platform, while the platform authenticates to the external AI provider.
109.19 Multi-Provider AI Credential Isolation
An AI platform may use several providers.
For example:
OpenAI
Gemini
Groq
Hugging Face
Other AI Provider
Each provider should have independently managed credentials.
The architecture should avoid a single master credential controlling every external provider.
Provider-specific credentials make:
rotation easier
compromise containment easier
billing attribution easier
auditing easier
provider replacement easier
109.20 Environment Separation
Credentials must be separated by environment.
Example:
Development
├── dev database
├── dev storage
└── dev AI credentials
Staging
├── staging database
├── staging storage
└── staging AI credentials
Production
├── production database
├── production storage
└── production AI credentials
A development machine should never require unrestricted production credentials.
109.21 Secrets Management
Production secrets should be managed through a dedicated secrets-management mechanism where appropriate.
Conceptually:
Application
|
| authenticated request
v
Secrets Manager
|
v
Secret
Examples of secret-management systems include:
The important architectural requirement is controlled access rather than a specific product.
109.22 Environment Variables
Environment variables are useful for configuration, but they should not automatically be considered a complete secrets-management solution.
For example:
DATABASE_URL
AI_PROVIDER_KEY
SESSION_SECRET
STORAGE_SECRET
may be injected into an application environment.
However, production systems should carefully control:
who can read environment configuration
where secrets are stored
who can modify them
whether logs can expose them
whether crash reports capture them
whether debugging systems can access them
109.23 Secret Leakage Prevention
Secrets can accidentally leak through:
source code
Git history
logs
error messages
screenshots
terminal output
CI/CD logs
browser bundles
analytics
monitoring systems
support tickets
Therefore, applications should implement secret-scanning controls.
Examples include:
Pre-commit scanning
↓
Repository scanning
↓
CI scanning
↓
Deployment scanning
A detected production secret should be treated as potentially compromised.
Simply deleting the visible line may not remove the secret from historical Git commits or other systems.
109.24 Logging Rules
Credentials must never be logged.
Unsafe:
API key: ax_live_123456...
Safe:
API key authentication succeeded
key_id=key_123
Sensitive headers should be redacted.
For example:
Authorization: [REDACTED]
X-API-Key: [REDACTED]
Logging systems themselves must also be protected.
109.25 Credential Metadata
Although secret values should remain protected, useful metadata should be observable.
Recommended metadata includes:
credential_id
credential_type
owner
service
environment
created_at
expires_at
last_used_at
last_rotated_at
status
scope
This allows security teams to identify:
unused credentials
expired credentials
unexpectedly active credentials
credentials approaching expiration
credentials with excessive permissions
109.26 Machine Identity Registry
Large AI platforms benefit from maintaining an inventory of machine identities.
Example:
| Identity | Purpose | Environment | Privilege |
|---|
| api-service | API requests | Production | Medium |
| media-worker | Media processing | Production | Limited |
| ai-worker | AI inference | Production | Limited |
| billing-service | Billing operations | Production | High |
| backup-service | Backup operations | Production | High |
| deployment-runner | Deployment | Production | High |
This registry helps establish ownership and accountability.
109.27 Credential Ownership
Every credential should have an owner.
Ownership may belong to:
engineering team
platform team
security team
service owner
infrastructure team
Ownership matters because someone must be responsible for:
rotation
permissions
incident response
retirement
documentation
An orphaned credential is a security risk.
109.28 Credential Expiration
Credentials should have expiration policies.
Examples:
temporary token → minutes
deployment token → short duration
service credential → controlled lifecycle
API key → explicit expiration or periodic review
Not every credential can have the same lifetime.
The correct lifetime depends on:
risk
capability
environment
usage pattern
operational constraints
High-privilege credentials should generally have stricter lifecycle controls.
109.29 Credential Abuse Detection
Authentication telemetry can help detect credential misuse.
Potential signals include:
unusual geographic access
unusual IP ranges
unexpected service-to-service calls
abnormal request volume
access outside normal operating patterns
sudden permission changes
repeated authentication failures
impossible service behavior
access to resources outside normal scope
For AI systems, additional signals may include:
abnormal generation volume
unexpected provider usage
sudden token consumption
unusual model selection
abnormal file-processing volume
109.30 Rate Limiting Machine Identities
Machine identities should also be subject to resource controls.
A compromised service credential should not automatically allow unlimited consumption.
Controls can include:
Requests/minute
Tokens/minute
Concurrent jobs
Storage operations
File-processing jobs
Provider requests
Daily quota
This reduces financial and operational damage.
109.31 Credential Compromise Response
When a credential may have been compromised, the response should be systematic.
Example:
Detection
↓
Credential identification
↓
Risk assessment
↓
Immediate restriction
↓
Revocation
↓
Replacement
↓
Dependent-service update
↓
Log investigation
↓
Scope assessment
↓
Recovery
↓
Post-incident review
The goal is to minimize the credential's remaining usefulness to an attacker.
109.32 Emergency Credential Revocation
The platform should have an emergency mechanism capable of disabling high-risk credentials quickly.
Security administrators should know:
where credentials are managed
how to revoke them
which services depend on them
how replacement credentials are issued
how service recovery is performed
Emergency procedures should be tested before an actual incident.
109.33 Credential Rotation Without Downtime
Rotation should ideally support controlled overlap.
Example:
Credential A = active
Credential B = created
Credential A + B = temporarily valid
Services migrate to B
Credential B = verified
Credential A = revoked
This approach reduces the probability that credential rotation becomes an availability incident.
109.34 Service Credential Dependency Mapping
The platform should know which services depend on which credentials.
Example:
AI Provider Credential
|
+── ai-service
+── generation-worker
+── moderation-worker
When the credential changes, every dependent component can be identified.
This dependency map is especially important during emergency rotation.
109.35 CI/CD Credentials
CI/CD pipelines frequently have powerful permissions.
A deployment pipeline may access:
Therefore CI/CD identities require strong controls.
Recommended principles include:
109.36 Deployment Identity
A deployment identity should only perform deployment-related operations.
For example:
build
→ publish artifact
→ deploy approved version
→ verify deployment
It should not automatically have unrestricted database administration privileges.
Separating deployment and database-administration identities reduces blast radius.
109.37 Webhook Signing Secrets
External systems may send webhooks to the platform.
The platform should verify webhook authenticity using a provider-supported signature mechanism where available.
Conceptually:
External Provider
|
| signed webhook
v
Webhook Endpoint
|
v
Signature Verification
|
v
Request Validation
|
v
Application Processing
Webhook secrets must not be exposed to clients.
Replay protection should also be considered using timestamps, unique event IDs, or provider-supported mechanisms.
109.38 Machine Identity and Authorization
Machine authentication answers:
Who is calling?
Authorization answers:
What is that caller allowed to do?
Both are required.
Example:
media-worker
may be authenticated successfully.
However, authentication does not mean it may:
delete user accounts
The authorization layer must enforce the permission boundary.
109.39 Tenant-Aware Machine Identity
In a multi-tenant AI platform, machine identities must also respect tenant boundaries.
A worker may process a task for:
tenant_A
without gaining access to:
tenant_B
The system should propagate trusted tenant context through controlled service boundaries.
Example:
Authenticated Service
↓
Validated Tenant Context
↓
Authorization Policy
↓
Tenant-Scoped Query
↓
Tenant-Scoped Storage
Tenant identifiers should never be trusted simply because a client supplied them.
109.40 Agent Identity and Tool Credentials
AI agents may call tools such as:
search
storage
databases
APIs
messaging systems
workflow systems
An agent should not receive unrestricted credentials.
Instead:
Agent
|
v
Tool Authorization Layer
|
v
Scoped Tool Credential
|
v
External Resource
The tool layer should enforce:
allowed actions
resource scope
tenant scope
rate limits
approval requirements
logging
This prevents an agent from converting a legitimate tool capability into unrestricted system access.
109.41 Human Approval for High-Risk Credentials
Some actions should require human approval even when a machine identity technically has permission.
Examples may include:
production infrastructure changes
large financial transactions
destructive database operations
security-policy changes
credential privilege escalation
bulk deletion
The architecture can implement:
Agent Request
↓
Policy Evaluation
↓
High-Risk Action?
↓
Human Approval
↓
Controlled Execution
109.42 Credential Security Testing
Credential systems should be tested continuously.
Testing should include:
Functional tests
Verify:
valid credentials work
expired credentials fail
revoked credentials fail
invalid credentials fail
insufficient scopes fail
Security tests
Verify:
credentials cannot be enumerated
secrets are not returned in logs
unauthorized services cannot access protected resources
cross-tenant access fails
privilege escalation fails
expired tokens are rejected
Operational tests
Verify:
109.43 Credential Threat Model
Major threats include:
| Threat | Impact | Primary Defense |
|---|
| Credential theft | Unauthorized access | Secret protection |
| Credential leakage | Account/service compromise | Secret scanning |
| Excessive permissions | Large blast radius | Least privilege |
| Long-lived credentials | Extended attacker access | Expiration |
| Credential reuse | Cross-system compromise | Identity separation |
| Missing revocation | Persistent compromise | Revocation system |
| Poor logging | Investigation failure | Security telemetry |
| CI/CD compromise | Production compromise | Workload identity |
| Agent credential abuse | Automated misuse | Tool authorization |
| Cross-tenant credential abuse | Data exposure | Tenant isolation |
109.44 Reference Architecture
A secure machine-identity architecture can be represented as:
┌───────────────────────┐
│ Identity Registry │
└───────────┬───────────┘
│
┌──────────▼──────────┐
│ Secrets / IAM Layer │
└──────────┬──────────┘
│
┌───────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
API Services AI Workers CI/CD System
│ │ │
▼ ▼ ▼
Authorization Tool Boundary Deployment Policy
│ │ │
└───────────────┬───────┴───────────────┬────────┘
│ │
▼ ▼
Internal Services External Providers
│
▼
Databases / Storage
Every connection should have a defined identity and authorization boundary.
109.45 Recommended Credential Data Model
A conceptual credential table may contain:
credentials
------------
id
owner_id
service_id
credential_type
key_prefix
secret_hash
scope
environment
status
created_at
expires_at
last_used_at
last_rotated_at
revoked_at
Additional audit information can be stored separately:
credential_events
-----------------
id
credential_id
event_type
actor_id
service_id
timestamp
ip_address
metadata
Sensitive secret material should not be unnecessarily duplicated in audit records.
109.46 Credential State Machine
A credential lifecycle can be modeled as:
CREATED
↓
ACTIVE
↓
ROTATING
↓
ACTIVE
↓
EXPIRED
│
└──────────────┐
▼
REVOKED
↓
DESTROYED
Possible emergency transition:
ACTIVE
↓
COMPROMISED
↓
REVOKED
↓
REPLACED
State transitions should be auditable.
109.47 Monitoring and Alerts
Security monitoring should detect:
unexpected credential use
expired credential attempts
repeated authentication failures
unusual request volume
privilege changes
new high-privilege credentials
credentials without owners
credentials without expiration policies
secrets detected in repositories
abnormal provider spending
service identity accessing unexpected resources
Example alert:
HIGH SEVERITY
Credential: ai-prod-01
Service: ai-worker
Event: unusual provider usage
Requests: 12,400/min
Expected: <500/min
Action: credential restricted
109.48 Credential Governance
Credential governance should define:
Who can create credentials?
Who can approve high-risk credentials?
Who can rotate credentials?
Who can revoke credentials?
How long can credentials exist?
How are unused credentials detected?
How are emergency credentials handled?
How is credential access audited?
These policies should be documented and enforced.
109.49 Credential Security Checklist
Before production deployment, verify:
109.50 Final Architecture Principles
A secure AI platform should treat machine identities as security principals rather than simply technical configuration.
The architecture should follow these principles:
Every machine should have a clear identity.
Every identity should have a defined owner.
Every credential should have a defined purpose.
Permissions should follow least privilege.
Static credentials should be minimized.
Short-lived credentials should be preferred where practical.
Credentials should be isolated by environment.
AI provider credentials should remain on trusted backend infrastructure.
Service-to-service communication should be authenticated and authorized.
Credential lifecycle management should include creation, rotation, expiration, revocation and destruction.
Secrets should never appear in source code or logs.
CI/CD identities should be treated as high-value security principals.
Agent credentials should be scoped to explicit tools and resources.
Tenant context must be protected across service boundaries.
Credential activity should be monitored.
Emergency credential revocation should be tested.
High-risk operations should support human approval.
Credential architecture should be continuously reviewed.
The most important concept is simple:
A machine credential is not merely a password for software. It is a security capability.
If that capability is excessive, permanent, poorly monitored, or widely shared, compromise can become catastrophic.
A mature AI platform therefore combines:
Machine Identity
+
Authentication
+
Authorization
+
Least Privilege
+
Secret Protection
+
Short-Lived Credentials
+
Rotation
+
Revocation
+
Monitoring
+
Auditability
+
Tenant Isolation
+
Human Approval
Together, these controls create a defensible machine-identity architecture capable of supporting AI services, workers, agents, external providers, databases, storage systems, CI/CD pipelines and production infrastructure without relying on uncontrolled long-lived secrets.
End of Chapter 109
Next Chapter:
Chapter 110 — Secure Authentication & Session Implementation: Password Security, Sessions, MFA, Recovery, Device Trust & Account Protection
Comments
Post a Comment