Architecture Stack: Risk Controls

This is a conversation between Alex (Engineering Manager) and Jordan (Senior Architect) exploring Risk Controls.

Part 1: The Risk Landscape

Alex: Jordan, we've covered CCoE (organization), Policy as Code (automation), FinOps (cost). Is there more to governance?

Jordan: Yes. Risk controls. Governance isn't just about preventing cost overruns. It's about preventing security incidents, compliance violations, and operational disasters.

Alex: We need to control risk?

Jordan: You can't eliminate risk. But you can manage it: identify, assess, mitigate, monitor, respond.

Alex: That's comprehensive?

Jordan: Yes. L4.4 is where governance gets serious.

Part 2: What Is Risk Management?

Alex: Define it.

Jordan: Risk management is identifying potential problems, assessing their likelihood and impact, implementing controls to reduce risk, and monitoring for incidents.

Key components:

• Risk Identification: What could go wrong?
• Risk Assessment: How likely? How bad if happens?
• Risk Mitigation: What controls prevent it?
• Monitoring & Detection: Did it happen? Detect quickly.
• Response & Recovery: When it happens, respond effectively.
• Learning: Post-incident review. Prevent recurrence.

Risk management is continuous cycle. Never 'done'.

Alex: It's systematic?

Jordan: Very. Best organizations treat risk like operations or security. Structured, measurable, continuous.

Part 3: Risk Categories

Alex: What types of risks?

Jordan: Several categories:

Security Risks:

• Unauthorized access (credential compromise)
• Data breaches (PII, intellectual property exposed)
• Malware/ransomware (systems compromised)
• DDoS attacks (service unavailable)
• Insider threats (malicious employee)

Compliance Risks:

• Regulatory violation (HIPAA, PCI-DSS, GDPR)
• Audit failure (can't prove compliance)
• Data residency violation (data in wrong country)
• Retention violation (kept/deleted data incorrectly)
• Privacy violation (PII mishandled)

Operational Risks:

• Infrastructure failure (database unavailable)
• Data loss (no backup, recovery impossible)
• Deployment failure (breaking change deployed)
• Cascading failures (one failure triggers others)
• Human error (misconfiguration, wrong command)

Reputational Risks:

• Data breach (customer trust damaged)
• Public security incident (reputation harm)
• Service outage (perceived unreliability)
• Regulatory fine (public embarrassment)

Financial Risks:

• Unexpected costs (unoptimized cloud spend)
• Compliance fines (regulatory violation)
• Incident costs (breach response, recovery)
• Revenue loss (service unavailable)

Alex: That's a lot?

Jordan: Yes. Organizations face multiple risk categories. Good governance addresses all of them.

Part 4: Risk Assessment Framework

Alex: How do you assess risk?

Jordan: Risk assessment has two dimensions:

Probability (Likelihood):

• Low: <5% annual chance
• Medium: 5-50% annual chance
• High: >50% annual chance

Impact (Severity):

• Low: $0-100K impact
• Medium: $100K-1M impact
• High: >$1M impact

Risk Matrix:

             Low Impact    Medium Impact   High Impact
Low Prob     Accept        Accept          Mitigate
Medium Prob  Accept        Mitigate        Mitigate
High Prob    Mitigate      Mitigate        Eliminate

Example risks:

High probability/high impact risks get priority.

Alex: This is data-driven?

Jordan: Yes. Objective assessment drives prioritization.

Part 5: Risk Mitigation Strategies

Alex: How do you reduce risk?

Jordan: Four strategies:

Avoid: Don't do the risky thing.

• Example: Use managed database (Aurora) instead of self-managed database (EC2 + MySQL)
• Eliminates database operation risk

Mitigate: Reduce probability or impact.

• Example: Use encryption to reduce breach impact
• Use multi-region for availability to reduce downtime
• Use automated testing to reduce deployment failures

Transfer: Move risk to someone else.

• Example: Use AWS (they handle infrastructure risk)
• Use SaaS instead of self-hosted (vendor handles operations)
• Insurance (cyber liability insurance pays for breach)

Accept: Do nothing. Accept the risk.

• Example: Accept that unused instances cost $1K/year
• Not worth the effort to optimize
• Cost to mitigate > cost of risk

Best practice: Avoid high-impact risks. Mitigate medium risks. Accept low risks.

Alex: Requires judgment?

Jordan: Yes. Is it worth the effort to reduce this risk? Sometimes answer is no.

Part 6: Security Controls

Alex: How do you mitigate security risks?

Jordan: Multiple layers:

Preventive Controls (prevent bad things):

• Encryption at rest (database compromised, data is encrypted)
• Encryption in transit (network sniffing doesn't expose data)
• Access control/IAM (least privilege: only necessary permissions)
• Network isolation (security groups, NACLs, private subnets)
• Strong authentication (MFA, no weak passwords)

Detective Controls (detect bad things quickly):

• CloudTrail (log all API calls)
• VPC Flow Logs (log all network traffic)
• GuardDuty (AI detects anomalous behavior)
• Config Rules (detect misconfigurations)
• Security Hub (aggregate findings)
• Intrusion detection systems (detect attacks)

Responsive Controls (react to bad things):

• Incident response plan (know what to do)
• Automated response (block compromised account)
• Investigation tools (forensics to understand what happened)
• Communication plan (notify affected users)
• Recovery procedures (restore from backup)

Best practice: Assume breaches happen. Detect quickly. Respond fast.

Alex: Defense in depth?

Jordan: Yes. Multiple layers so if one fails, others catch it.

Part 7: Compliance Controls

Alex: How do you ensure compliance?

Jordan: Compliance requirements vary (HIPAA, PCI-DSS, SOC2, GDPR). General controls:

Governance:

• Document policies (security, data handling, access)
• Publish policies to teams
• Require acknowledgment

Technical Controls:

• Encryption required by regulation
• Access controls (role-based, audit trails)
• Data retention (keep X years, delete after)
• Segregation of duties (one person can't approve their own changes)

Operational Controls:

• Training (all employees trained on compliance)
• Background checks (for sensitive roles)
• Incident response (breach reporting requirements)
• Audit trails (who did what when)

Audit & Assessment:

• Internal audits (do we comply?)
• Third-party audits (external verification)
• Remediation (fix gaps found in audits)
• Continuous compliance (automated checks)

Key: Document everything. Audits require proof.

Alex: Audit trails matter?

Jordan: Critical. Auditors need to see that controls were in place and working.

Part 8: Compliance as Code

Alex: Can you automate compliance?

Jordan: Partially. Compliance as Code:

Automated Checks:

# Check: CloudTrail must be enabled
policy "cloudtrail_enabled" {
  rule = all aws_cloudtrail.* {
    is_enabled == true
  }
}
 
# Check: S3 bucket versioning (required by HIPAA)
policy "s3_versioning_enabled" {
  rule = all aws_s3_bucket.* {
    versioning.enabled == true
  }
}
 
# Check: MFA required for console access
policy "mfa_required" {
  rule = all aws_iam_user.* {
    mfa_device != null
  }
}

Compliance Dashboard:

Control                           Status    Evidence
CloudTrail enabled               PASS      All trails active
S3 versioning enabled            PASS      Config checked all buckets
MFA required                     FAIL      5 users without MFA
Encryption at rest               PASS      All DBs encrypted
Access logging enabled           PARTIAL   98% of resources

Benefits: Continuous monitoring. Catches drift quickly. Audit evidence automated.

Alex: This reduces manual review?

Jordan: Massively. Auditors just verify the automated system works.

Part 9: Audit & Evidence

Alex: How does auditing work?

Jordan: Audit cycle:

Planning Phase:

1. Define audit scope (which systems, which controls)
2. Risk assessment (which controls are highest risk)
3. Sample selection (how many resources to test)

Testing Phase:

1. Collect evidence (CloudTrail logs, Config snapshots, IAM policies)
2. Verify controls (is policy enforced? Are controls working?)
3. Test exceptions (identify controls that failed)

Reporting Phase:

1. Document findings (control failures)
2. Assess severity (high/medium/low)
3. Recommend remediation (how to fix)
4. Track progress (auditor follows up on fixes)

Continuous Auditing:

• Automated scans (daily or weekly)
• Reports generated automatically
• Exceptions tracked over time
• Trend analysis (improving or deteriorating?)

Alex: Evidence must be traceable?

Jordan: Yes. Auditors need: 'Control X was checked on Y date by Z tool, and result was W.'

Part 10: Incident Response

Alex: What if something goes wrong?

Jordan: Incident response plan. Process for when breach or incident occurs:

Preparation:

• Incident response team (who handles incidents?)
• Playbooks (documented procedures for common incidents)
• Communication plan (who notifies whom?)
• Backup procedures (how to restore from backup?)

Detection:

• Monitoring alerts you
• Team investigates
• Severity assessed (is this serious?)

Containment:

• Limit blast radius (isolate compromised systems)
• Stop attack (block attacker, revoke credentials)
• Gather forensic evidence (preserve logs for investigation)

Eradication:

• Fix vulnerability (patch, reconfigure)
• Remove attacker (change all credentials)
• Verify fix (test that vulnerability is closed)

Recovery:

• Restore systems from clean backup
• Verify data integrity
• Bring systems back online

Post-Incident Review:

• What happened? (root cause analysis)
• Why did it happen? (what controls failed?)
• How do we prevent recurrence? (policy/control improvements)
• What did we learn? (team training)

Goal: Detect in <1 hour. Contain in <4 hours. Recover in <24 hours.

Alex: Time matters?

Jordan: Critically. Every hour of downtime costs money and trust.

Part 11: Incident Response Tools

Alex: What tools help?

Jordan: Incident response infrastructure:

Detection & Alerting:

• CloudWatch Alarms
• GuardDuty (security findings)
• Security Hub (aggregated findings)
• VPC Flow Logs analysis
• Web Application Firewalls (WAF)

Investigation & Forensics:

• CloudTrail (API audit logs)
• VPC Flow Logs (network activity)
• CloudWatch Logs (application logs)
• Systems Manager Session Manager (access running systems)
• AWS X-Ray (request tracing)

Response Automation:

• AWS Lambda (trigger response actions)
• Systems Manager Automation (orchestrate response)
• SNS/SQS (alert and queue tasks)

Communication:

• PagerDuty (on-call alerting)
• Slack integration (team notifications)

Example Automated Response:

Event: GuardDuty detects EC2 instance communicating with malware
Action:
  1. Lambda function triggered
  2. Isolate instance (remove from load balancer)
  3. Snapshot EBS volume (forensic evidence)
  4. Alert security team (Slack + PagerDuty)
  5. Create ticket (document incident)
  6. Notify manager (impact assessment)

Best incident responses are automated.

Alex: Reduces response time?

Jordan: By orders of magnitude. Automated response: seconds. Manual response: hours.

Part 12: Post-Incident Learning

Alex: What happens after incident?

Jordan: Blameless post-incident review. Goal is learning, not blame.

Retrospective Process:

1. Timeline: Document exactly what happened and when
2. Root Cause Analysis: Why did this happen?
3. Contributing Factors: What conditions allowed this?
4. Preventive Controls: What should have prevented this?
5. Detective Controls: What should have detected this sooner?
6. Action Items: Specific changes to prevent recurrence
7. Ownership: Assign owners to action items with deadlines

Example:

Incident: Database credentials leaked in GitHub
 
Timeline:
- 3/15 10AM: Developer committed credentials to public repo
- 3/15 2PM: Credentials discovered by automated scan
- 3/15 3PM: Credentials rotated
- 3/15 4PM: No unauthorized access detected
 
Root Cause: Developer forgot to add database password to .gitignore
 
Contributing Factors:
- No pre-commit hook to scan for secrets
- Training on credential management not recent
 
Preventive Controls:
- Implement pre-commit hook (prevents commit of secrets)
- Use AWS Secrets Manager (no credentials in code)
- Training refresher on security best practices
 
Detective Control:
- GitHub secret scanning already working (caught it)
 
Action Items:
- Install pre-commit hook framework (Dev Lead, 1 week)
- Migrate to AWS Secrets Manager (Infra Team, 2 weeks)
- Run security training (Security Team, 1 month)
 
Timeline to complete: 1 month

Result: Better controls. Team learns. Incident less likely to repeat.

Alex: This is blameless?

Jordan: Yes. Focus on systems and processes, not individual blame. Removes fear. Encourages reporting.

Part 13: Risk Reporting & Dashboards

Alex: How do you track risk?

Jordan: Dashboards and regular reporting:

Risk Dashboard (Monthly):

Risk Category       Risk Level   Trend    Action
Security           MEDIUM       ↓ (improving)    Continue monitoring
Compliance         LOW          → (stable)       Track audit
Operational        HIGH         ↑ (worsening)    Escalate
Financial          MEDIUM       ↓ (improving)    FinOps team

Specific Metrics:

Security:
- Vulnerabilities: 3 (target: 0)
- Incidents: 0 (trend: ↓)
- MTTR (Mean Time To Respond): 30 min (target: <15 min)
- Security findings: 12 (target: <5)
 
Compliance:
- Audit findings: 2 (target: 0)
- Compliance gaps: 1 policy not enforced
- Control effectiveness: 98% (target: 99%)
 
Operational:
- Incident count: 2 (target: 0)
- MTBF (Mean Time Between Failures): 45 days (target: 90 days)
- Deployment failure rate: 2% (target: <0.1%)
- Data loss incidents: 0 (target: 0)

Escalation Process:

• Risk level HIGH: Escalate to CTO immediately
• Risk level MEDIUM: Review with CCoE monthly
• Risk level LOW: Monitor, no action needed

Report distributed: CTO, CFO, Board (quarterly)

Alex: Visibility to leadership?

Jordan: Essential. Executives need to understand organizational risks.

Part 14: Integrating All L4 Components

Alex: We've covered CCoE (org), Policy as Code (automation), FinOps (cost), Risk Controls (compliance/security). How do they work together?

Jordan: Integrated governance:

CCoE sets strategy. 'We will be secure, compliant, cost-optimized.'

Policy as Code enforces it. 'These rules are checked automatically on every deployment.'

FinOps optimizes cost. 'We track spend and eliminate waste.'

Risk Controls manage safety. 'We detect and respond to incidents quickly.'

Example Flow:

Team wants to deploy new application.
 
Step 1 (CCoE): Review architecture. Does it follow standards?
Step 2 (Policy as Code): Run policies. Does it pass security/compliance checks?
Step 3 (FinOps): Estimate cost. Is it reasonable? Reserved instances applied?
Step 4 (Risk Controls): Security review. Incident response ready?
Step 5: Deploy.
Step 6 (Ongoing): Monitor costs, security posture, compliance status.

All layers working together. No gaps.

Alex: This is the complete L4?

Jordan: Yes. Organization, automation, cost, and risk. Together, they govern cloud at scale.

Part 15: The Complete Stack

Alex: We've now covered all four layers.

Jordan: Yes:

L1 (Services & Data): How applications are structured L2 (Platform Delivery): How applications are deployed L3 (Infrastructure as Code): How infrastructure is defined L4 (Governance & Control): How infrastructure is governed

Each layer depends on the others. Together they form complete cloud architecture stack.

Alex: And they're all connected?

Jordan: Completely. L1 informs L2 design. L2 needs L3 infrastructure. L3 must comply with L4 policies. L4 policies optimize based on L3 costs.

Part 16: The Bottom Line

Alex: If I pitch this?

Jordan: Risk controls are the guardrails of cloud governance. We identify risks systematically. Implement preventive controls. Detect incidents quickly. Respond effectively. Learn from incidents. Compliance is continuous. Security is built-in. No surprises.

Alex: And the commitment?

Jordan: We're saying: governance is not optional. It's foundational. Without it, cloud becomes chaos. With it, cloud becomes competitive advantage.

Key Takeaways