56.1 Introduction
Technical security controls alone are not enough to operate a trustworthy AI platform.
An AI system may have:
authentication
authorization
encryption
monitoring
secure storage
sandboxing
model safety controls
incident response
Yet the organization can still have serious problems if nobody knows:
which AI models are deployed
who owns each model
what each model is allowed to do
what data each model can process
which risks have been accepted
which controls are mandatory
who approves high-risk changes
how security decisions are documented
This is the purpose of AI governance.
A useful principle is:
Governance converts security intentions into accountable, repeatable, and auditable decisions.
56.2 What Is AI Governance?
AI governance is the framework used to control how AI systems are:
designed
developed
evaluated
deployed
monitored
changed
retired
Governance should cover the entire AI lifecycle.
Idea
↓
Design
↓
Development
↓
Testing
↓
Approval
↓
Deployment
↓
Monitoring
↓
Change
↓
Retirement
Security should exist throughout this lifecycle rather than being added only immediately before production.
56.3 Governance vs Security
Security and governance are closely related but different.
Security asks:
How do we protect the system?
Governance asks:
What are we allowed to build?
Who is responsible?
What risks are acceptable?
Which controls are mandatory?
Who approves the decision?
How do we prove that controls exist?
A mature organization needs both.
56.4 AI Governance Structure
A practical structure may contain:
AI Governance
|
+-- Policies
|
+-- Risk Management
|
+-- Model Inventory
|
+-- Data Governance
|
+-- Security
|
+-- Privacy
|
+-- Compliance
|
+-- Evaluation
|
+-- Monitoring
|
+-- Incident Management
|
+-- Accountability
56.5 AI System Inventory
Every production AI system should have an inventory.
A model registry can track:
model_id
model_name
version
provider
owner
purpose
risk_level
deployment_status
data_classification
approved_use_cases
restricted_use_cases
evaluation_status
security_status
created_at
updated_at
retirement_date
This creates organizational visibility.
56.6 Why Model Inventory Matters
Without an inventory, an organization may unknowingly have:
Model A → Production
Model B → Experimental
Model C → Forgotten API
Model D → Old version
Model E → Third-party provider
The security team cannot properly protect systems it does not know exist.
Therefore:
Unknown AI systems represent governance risk.
56.7 Model Ownership
Each important AI system should have a clearly identified owner.
Possible roles:
BUSINESS_OWNER
TECHNICAL_OWNER
SECURITY_OWNER
PRIVACY_OWNER
MODEL_OWNER
One person may hold multiple roles in a small organization.
The important requirement is that responsibility is explicit.
56.8 AI Risk Classification
Not every AI feature has the same risk.
A useful internal classification might be:
LOW
MEDIUM
HIGH
CRITICAL
Example:
Low
Formatting assistant
Non-sensitive brainstorming
Medium
Internal document summarization
Productivity assistant
High
Sensitive organizational data processing
Automated decisions affecting important workflows
Agentic operations
Critical
AI connected to highly privileged systems
High-impact autonomous actions
Systems where failure could cause significant harm
The exact classification criteria should be defined by the organization's risk framework.
56.9 Risk Management Lifecycle
A practical AI risk process is:
IDENTIFY
↓
ANALYZE
↓
EVALUATE
↓
MITIGATE
↓
MONITOR
↓
REVIEW
This process should repeat as the system changes.
56.10 Risk Identification
Possible AI risks include:
privacy leakage
prompt injection
model misuse
unsafe outputs
hallucination
data poisoning
model theft
credential compromise
agent misuse
supply-chain compromise
availability failure
vendor dependency
regulatory exposure
The risk register should document these systematically.
56.11 Risk Register
A simple risk record might contain:
risk_id
description
asset
threat
likelihood
impact
risk_score
existing_controls
planned_controls
owner
status
review_date
Example:
Risk:
Unauthorized retrieval of tenant documents.
Likelihood:
Medium
Impact:
High
Risk:
High
Controls:
Tenant isolation
Authorization filtering
Audit logging
Security testing
56.12 Likelihood and Impact
Risk is commonly evaluated using:
Risk ≈ Likelihood × Impact
For example:
Likelihood = High
Impact = High
produces a high-priority risk.
However, numerical scoring is only a decision aid.
A low-frequency event can still require strong controls if the potential impact is severe.
56.13 Risk Treatment
A risk can be handled through several strategies.
MITIGATE
AVOID
TRANSFER
ACCEPT
Mitigate
Add controls to reduce the risk.
Avoid
Do not deploy the risky capability.
Transfer
Move part of the risk through contracts, insurance, or service arrangements where appropriate.
Accept
Explicitly accept residual risk through an authorized decision.
Risk acceptance should never be accidental.
56.14 Security Policies
AI platforms should have documented policies.
Examples:
AI Security Policy
AI Data Handling Policy
AI Model Approval Policy
AI Vendor Policy
AI Agent Policy
AI Logging Policy
AI Incident Response Policy
AI Retention Policy
AI Access Control Policy
Policies define expected behavior.
56.15 Standards and Procedures
A policy may say:
Sensitive AI workloads require security review.
A procedure explains:
How the review is performed.
Who performs it.
Which evidence is required.
What approval means.
Therefore:
Policy
↓
Standard
↓
Procedure
↓
Implementation
↓
Evidence
56.16 Security Controls
Controls translate policy into technical or organizational mechanisms.
Examples:
MFA
RBAC
encryption
network segmentation
sandboxing
logging
rate limiting
backup
dependency scanning
model evaluation
human approval
A control should have:
control_id
description
owner
implementation
test_method
evidence
review_frequency
status
56.17 Control Mapping
Organizations often map controls to multiple frameworks.
Conceptually:
Internal Control
|
+---+---+
| |
Security Privacy
Framework Framework
| |
+---+---+
|
Compliance
This avoids implementing completely separate control systems for every requirement.
56.18 Evidence Management
A control is stronger when there is evidence that it actually operates.
Evidence can include:
configuration
audit log
security test
approval record
deployment record
access review
incident report
scan result
evaluation report
For example:
Control:
MFA required for administrators.
Evidence:
Authentication configuration
+
Access-control test
+
Audit record
56.19 Model Approval Process
A new model should not automatically become production-ready simply because it works technically.
A possible workflow is:
Model Proposed
↓
Use-Case Review
↓
Data Review
↓
Security Review
↓
Privacy Review
↓
Safety Evaluation
↓
Performance Evaluation
↓
Approval
↓
Production Deployment
56.20 Model Cards and Documentation
Each important model should have useful documentation.
Possible fields:
purpose
capabilities
limitations
training information
evaluation results
known risks
supported inputs
restricted uses
security considerations
privacy considerations
version
provider
owner
The documentation should reflect the actual system rather than being generic marketing material.
56.21 Third-Party AI Providers
AI applications frequently depend on external providers.
Governance should track:
provider
model
region
data handling
retention
security controls
contractual terms
availability
subprocessors
incident process
Before using a third-party AI service, organizations should understand what data leaves their environment.
56.22 Vendor Risk
A provider dependency can create:
availability risk
privacy risk
security risk
cost risk
vendor lock-in
model behavior changes
service discontinuation
A vendor assessment can reduce surprises.
56.23 Model Version Changes
AI providers may change models.
For example:
Model v1
↓
Provider update
↓
Model v2
Even if the API remains compatible, behavior may change.
Therefore production systems should monitor:
quality
latency
cost
safety
security
output behavior
after significant model changes.
56.24 Change Management
AI systems require controlled changes.
A change may include:
new model
new prompt
new system policy
new tool
new retrieval source
new memory mechanism
new provider
new security control
A production change workflow might be:
Change Request
↓
Risk Assessment
↓
Testing
↓
Approval
↓
Deployment
↓
Monitoring
↓
Rollback if necessary
56.25 Prompt Changes Are Security Changes
A common mistake is treating prompts as harmless configuration.
A system prompt can influence:
model behavior
tool usage
safety behavior
data handling
workflow decisions
Therefore significant prompt changes should be version-controlled and reviewed.
Example:
prompt_version = 42
should be traceable to the deployed application version.
56.26 Agent Governance
AI agents require stronger governance because they can interact with external systems.
An agent inventory should record:
agent_id
purpose
owner
model
tools
permissions
data_access
approval_requirements
risk_level
deployment_status
A particularly important field is:
allowed_tools
The agent should not receive unrestricted access merely because its underlying model is capable of using many tools.
56.27 Human Approval Policies
Certain operations should require human approval.
Examples:
high-impact financial actions
destructive operations
large data exports
privileged configuration changes
external communications
organization-wide changes
A governance policy can define:
action risk
+
user role
+
resource scope
=
approval requirement
56.28 AI Safety Governance
Safety evaluations should be integrated into governance.
Possible evaluations include:
prompt injection testing
jailbreak testing
privacy leakage testing
bias testing
robustness testing
tool-use testing
multilingual testing
multimodal testing
adversarial testing
The evaluation record should include:
test_version
model_version
test_cases
results
failures
mitigations
approval_status
56.29 Security Exceptions
Sometimes a system cannot immediately meet a control requirement.
Instead of silently ignoring the control, create an exception.
Example:
exception_id
control_id
reason
risk
compensating_control
owner
approval
expiration_date
Exceptions should expire.
An exception that never expires becomes an undocumented permanent weakness.
56.30 Separation of Duties
Important decisions should not depend on one uncontrolled individual.
For example:
Developer
↓
Builds system
Security Reviewer
↓
Reviews security
Business Owner
↓
Approves business use
Administrator
↓
Deploys
The exact organizational structure can vary, but critical responsibilities should have appropriate separation.
56.31 Security Accountability
Every major AI capability should have answers to:
Who owns it?
Who can change it?
Who can approve it?
Who can access its data?
Who monitors it?
Who responds to incidents?
Who can retire it?
If the answer is unclear, governance is incomplete.
56.32 AI Governance Database
A governance subsystem could include:
ai_systems
ai_models
ai_agents
ai_risks
ai_controls
ai_policies
ai_assessments
ai_approvals
ai_exceptions
ai_vendors
ai_incidents
ai_changes
Relationships might look like:
AI System
|
+-- Models
+-- Agents
+-- Risks
+-- Controls
+-- Assessments
+-- Vendors
+-- Changes
56.33 Example Model Registry Interface
interface AIModelRecord {
id: string;
name: string;
version: string;
provider: string;
ownerId: string;
purpose: string;
riskLevel:
| "LOW"
| "MEDIUM"
| "HIGH"
| "CRITICAL";
status:
| "EXPERIMENTAL"
| "APPROVED"
| "PRODUCTION"
| "DEPRECATED"
| "RETIRED";
approvedUseCases: string[];
restrictedUseCases: string[];
createdAt: Date;
updatedAt: Date;
}
56.34 Example Risk Record
interface AIRiskRecord {
id: string;
systemId: string;
description: string;
likelihood:
| "LOW"
| "MEDIUM"
| "HIGH";
impact:
| "LOW"
| "MEDIUM"
| "HIGH"
| "CRITICAL";
status:
| "OPEN"
| "MITIGATED"
| "ACCEPTED"
| "CLOSED";
ownerId: string;
reviewDate: Date;
}
56.35 Governance API Boundaries
Governance operations should be protected.
Example:
POST /governance/models
POST /governance/risks
POST /governance/assessments
POST /governance/approvals
GET /governance/models/:id
GET /governance/risks/:id
GET /governance/audit
Only authorized administrative roles should access these endpoints.
56.36 Governance Audit Trail
Important governance actions should generate audit events.
Examples:
MODEL_REGISTERED
MODEL_APPROVED
MODEL_REJECTED
MODEL_DEPLOYED
MODEL_RETIRED
RISK_CREATED
RISK_ACCEPTED
RISK_MITIGATED
CONTROL_CREATED
CONTROL_TESTED
EXCEPTION_CREATED
EXCEPTION_APPROVED
POLICY_UPDATED
These events establish accountability.
56.37 Compliance Management
Compliance should not be treated as a collection of checkboxes.
A mature compliance process connects:
Requirement
↓
Policy
↓
Control
↓
Implementation
↓
Evidence
↓
Testing
↓
Review
This creates traceability.
56.38 Continuous Compliance
AI systems change frequently.
Therefore compliance should be continuously monitored.
A change in:
model
data
provider
prompt
agent
tool
architecture
region
may require a new assessment.
56.39 Governance Metrics
Useful metrics include:
approved AI systems
unregistered AI systems
open high-risk findings
overdue risk reviews
expired exceptions
models without owners
models without recent evaluations
agents without tool reviews
controls failing tests
vendor assessments overdue
These metrics provide leadership visibility.
56.40 AI Governance Dashboard
A governance dashboard could display:
AI SYSTEMS
42 total
35 production
7 experimental
HIGH-RISK SYSTEMS
5
OPEN HIGH-RISK FINDINGS
3
EXPIRED EXCEPTIONS
0
MODELS WITHOUT OWNER
0
OVERDUE REVIEWS
2
The dashboard should help people make decisions rather than merely display numbers.
56.41 Governance Threat Model
Governance itself can be attacked.
Potential threats include:
unauthorized model approval
fake risk acceptance
audit-log manipulation
unapproved model deployment
hidden AI systems
expired security exceptions
privilege abuse
vendor misrepresentation
tampered evaluation results
Therefore governance systems require the same security principles as other sensitive applications.
56.42 Protecting Approval Workflows
An approval should identify:
who approved
what was approved
which version
when
under which conditions
Example:
Approval:
Model X
Version 7
Use case: internal summarization
Risk level: Medium
Approved by: authorized reviewer
Timestamp: recorded
A future model version should not automatically inherit the approval of a materially different version.
56.43 Governance and DevSecOps
Governance should integrate with development pipelines.
Example:
Developer creates model configuration
↓
Security checks
↓
Policy checks
↓
Evaluation
↓
Approval status
↓
CI/CD deployment gate
A production deployment can be blocked if required governance conditions are not satisfied.
56.44 Example Deployment Gate
Conceptually:
IF
model_registered
AND security_evaluation_passed
AND required_approval_exists
AND no_blocking_risk
AND deployment_policy_allows
THEN
deploy
ELSE
block
This makes governance enforceable rather than purely documentary.
56.45 Governance as Code
Some policies can be represented programmatically.
Example:
function canDeploy(model: AIModelRecord): boolean {
return (
model.status === "APPROVED" &&
model.riskLevel !== "CRITICAL"
);
}
Production systems should implement richer policy evaluation than this simplified example.
The important idea is that policy can become machine-enforceable.
56.46 Governance and Incident Response
An incident should update governance.
For example:
Incident
↓
Root Cause Analysis
↓
Risk Register Update
↓
Control Improvement
↓
Policy Update
↓
New Evaluation
This creates a continuous improvement cycle.
56.47 AI Lifecycle Governance
The complete lifecycle can be represented as:
IDEATE
↓
CLASSIFY
↓
ASSESS
↓
DESIGN
↓
BUILD
↓
TEST
↓
APPROVE
↓
DEPLOY
↓
MONITOR
↓
REASSESS
↓
UPDATE
↓
RETIRE
Security and governance should follow the system through every stage.
56.48 Production Governance Checklist
Inventory
Risk
Security
Privacy
Change Management
Accountability
Continuous Improvement
56.49 Reference Governance Architecture
A complete architecture can be represented as:
AI GOVERNANCE
|
+----------------------+----------------------+
| | |
Inventory Risk Policy
| | |
Models Assessments Controls
Agents Risk Register Standards
Providers Exceptions Procedures
| | |
+----------------------+----------------------+
|
Security Review
|
Privacy Review
|
AI Evaluation
|
Approval
|
Deployment
|
Monitoring
|
Incident Response
|
Lessons Learned
|
Reassessment
56.50 Final Architecture Principle
A secure AI platform needs more than secure code.
It needs a system of accountability that makes important decisions:
visible
documented
authorized
reviewable
measurable
auditable
enforceable
The central governance chain is:
IDENTIFY
↓
CLASSIFY
↓
ASSESS
↓
CONTROL
↓
APPROVE
↓
DEPLOY
↓
MONITOR
↓
REVIEW
↓
RETIRE
The most important principle is:
Every important AI capability should have an owner, a defined purpose, a known risk profile, appropriate controls, documented approval, and continuous oversight.
Governance should not exist separately from engineering.
The strongest architecture connects:
Governance
+
Security
+
Privacy
+
Engineering
+
Operations
+
Incident Response
into one continuous lifecycle.
That is what transforms an AI application from a collection of models and APIs into an accountable production system.
Comments
Post a Comment