Architecture Stack: Policy as Code

This is a conversation between Alex (Engineering Manager) and Jordan (Senior Architect) exploring Policy as Code.

Part 1: The Manual Review Problem

Alex: "Jordan, so CCoE sets policies. But how do they enforce them? Manual review?"

Jordan: "That's the old way. Manual review doesn't scale. As teams move faster, reviews become bottlenecks."

Alex: "So?"

Jordan: "Policy as Code. Policies written as code, enforced automatically. No manual review needed for routine decisions."

Alex: "Policies are code?"

Jordan: "Yes. Like tests. You write assertions. If violated, the pipeline fails. Automatic enforcement."

---

Part 2: What Is Policy as Code?

Alex: "Define it."

Jordan: "Policy as Code is expressing organizational policies as enforceable code rules. Policies evaluated automatically, violations detected, enforcement actions taken.

Key aspects:

• Declarative: 'Describe what should be true'
• Automated: No human review for routine violations
• Auditable: Every policy check is logged
• Composable: Multiple policies work together
• Versionable: Policies tracked in Git

Example policy: 'All S3 buckets must have versioning enabled.'

Policy as code detects: team creates S3 bucket without versioning
Evaluation: FAIL
Action: Block deployment
Message: S3 bucket violates policy. Enable versioning.

No human needed to review."

Alex: "This scales automatically?"

Jordan: "Completely. One policy applies to all teams, all deployments."

---

Part 3: Policy as Code Tools

Alex: "What tools?"

Jordan: "Several categories:

Infrastructure as Code (Terraform):

• Sentinel: HashiCorp's language. For Terraform Cloud/Enterprise. Excellent for Terraform policies.

General Policy Engine:

• OPA/Rego: Open Policy Agent. Language-agnostic. Works with Terraform, Kubernetes, APIs, databases. Most flexible.

Cloud-Native:

• AWS Config: For AWS. Evaluates AWS resources against rules.
• Azure Policy: For Azure. Policy-driven governance.

Container & Kubernetes:

• Kyverno: Kubernetes native. Policies for pods, deployments, services.
• Pod Security Policies: Kubernetes built-in (deprecated but still used).

IaC-Specific:

• CloudFormation Guard: For CloudFormation templates.
• Checkov: For Terraform and CloudFormation. Security-focused.

Most powerful: OPA/Rego. Can enforce policies across entire infrastructure stack."

Alex: "Which one?"

Jordan: "For Terraform: Sentinel (if using Terraform Enterprise). OPA/Rego (most flexible). For multiple platforms: OPA/Rego."

---

Part 4: Policy as Code Languages

Alex: "How do you write policies?"

Jordan: "Depends on tool. Examples:

Sentinel (HashiCorp):

policy \"require_s3_encryption\" {
  rule = all tfplan.resource_changes as rc {
    rc.type is \"aws_s3_bucket\" and
    rc.after.server_side_encryption_configuration is not empty
  }
}

Rego (OPA):

deny[msg] {
    input.resource_type == \"aws_s3_bucket\"
    not input.server_side_encryption_configuration
    msg := \"S3 bucket must have encryption enabled\"
}

Azure Policy (JSON):

{
  \"if\": {
    \"field\": \"type\",
    \"equals\": \"Microsoft.Storage/storageAccounts\"
  },
  \"then\": {
    \"effect\": \"deny\"
  }
}

Languages vary. But concept is same: assert what should be true."

Alex: "Readability?"

Jordan: "Sentinel and Rego are most readable. Azure Policy is JSON (verbose but clear)."

---

Part 5: Types of Policies

Alex: "What policies do you write?"

Jordan: "Several categories:

Security Policies:

• 'All databases must have backups enabled'
• 'All S3 buckets must be private (no public access)'
• 'All EBS volumes must be encrypted'
• 'No secrets hardcoded in Terraform'

Compliance Policies:

• 'All resources must be tagged with cost-center'
• 'All resources must be tagged with owner'
• 'Compliance-sensitive resources (PII) in private subnets only'
• 'All logs must ship to central account'

Operational Policies:

• 'Development instances must auto-terminate after 8 hours'
• 'Production databases must have multi-AZ enabled'
• 'No t2.nano instances in production'
• 'All load balancers must have health checks'

Cost Policies:

• 'No unattached EBS volumes'
• 'EC2 instances without auto-scaling must have tags indicating intent'
• 'Reserved instances required for instances running > 730 hours/year'
• 'S3 lifecycle policies required for buckets > 1GB'

Organizational Policies:

• 'All resources must be created in approved regions only'
• 'No resources created on Friday (change freeze)'
• 'No manual AWS console changes (all via Terraform)'
• 'Pull requests require approval before deployment'"

Alex: "These are comprehensive?"

Jordan: "Yes. If enforced, they prevent most common mistakes."

---

Part 6: Enforcement Strategies

Alex: "How strict should enforcement be?"

Jordan: "Three levels:

Soft (Advisory):

• Policy is checked
• Violation flagged as warning
• Deployment continues
• Team sees warning in logs
• Action: Awareness, learning

Medium (Enforcement with Exception):

• Policy is checked
• Violation blocks deployment
• Team can request exception (tracked)
• Exception reviewed by CCoE
• If approved: exception recorded, deployment proceeds
• Action: Enforce but allow flexibility

Hard (Strict Enforcement):

• Policy is checked
• Violation blocks deployment
• No exceptions allowed
• Action: Zero tolerance

Best practice: Start with soft (advisory). Move to medium (enforcement with exceptions) as teams mature. Hard (strict) only for critical policies (security, compliance)."

Alex: "Adoption is easier with soft start?"

Jordan: "Much easier. Teams don't feel punished. Learn why policies matter."

---

Part 7: Policy as Code in CI/CD Pipeline

Alex: "Where in the pipeline do policies run?"

Jordan: "Multiple stages:

Stage 1: Developer Workstation (Local)

terraform init
terraform plan
checkov -f main.tf  # Policy check locally
opa eval -d policy.rego -i input.json  # OPA policies

Feedback: Immediate. Developer fixes issues before push.

Stage 2: Git Pull Request

Developer pushes code
Pull request created
CI/CD pipeline runs:
  - terraform plan
  - Policy checks (Sentinel, OPA, Checkov)
  - Security scanning
Results shown in PR comment
Developer reviews, makes changes

Feedback: Before merge. Code review includes policy violations.

Stage 3: Pre-Apply (Before terraform apply)

terraform plan reviewed by human
All policies must pass
If violations found:
  - Explain violation
  - Ask for exception request
  - Or ask for correction

Feedback: Last gate before deployment.

Stage 4: Post-Deployment (Audit)

Resources deployed
Continuous compliance monitoring
If drift detected:
  - Alert team
  - Optionally auto-remediate

Feedback: Ongoing. Catches manual changes or new violations."

Alex: "Multiple gates?"

Jordan: "Yes. Early feedback is cheap (fix locally). Late feedback is expensive (revert deployed resources)."

---

Part 8: Exception Management

Alex: "How do exceptions work?"

Jordan: "Formal exception process:

1. Policy Violation Occurs: Terraform plan fails due to policy violation.

2. Team Requests Exception:

   Exception request form:
   - Policy violated: 'S3 bucket must be encrypted'
   - Reason: 'Development test environment, temporary'
   - Duration: '30 days'
   - Risk acceptance: 'Team acknowledges risk'
   - Approver: 'CCoE security lead'
   

3. CCoE Reviews: Is the exception justified? What's the risk? Can it be time-limited?

4. Decision:

   • Approved: Exception recorded. Deployment proceeds. Tracked.
   • Denied: Must comply with policy. Or implement differently.
   • Conditional: Approved with requirements ('Only in dev account', 'Must have VPC Flow Logs enabled')

5. Enforcement:

   • Exception stored in exception registry (Git)
   • Policy engine checks registry
   • If exception matches: Allow violation
   • If exception expires: Re-enforce policy

6. Audit Trail: All exceptions logged. Report on exception count and types. Identify patterns."

Alex: "Prevents policy bypass?"

Jordan: "Completely. Exceptions are tracked, approved, temporary. Not hiding violations."

---

Part 9: Policy Composition

Alex: "Do policies work together?"

Jordan: "Yes. Multiple policies evaluated against single resource.

Example:

Policy 1: All S3 buckets must be encrypted (MUST PASS)
Policy 2: All S3 buckets must have versioning (MUST PASS)
Policy 3: All S3 buckets must have backup (SOFT: warning)

Resource created:

• S3 bucket with encryption and versioning: PASS all policies
• S3 bucket with encryption, no versioning: FAIL Policy 2 (blocked)
• S3 bucket with encryption, versioning, no backup: PASS Policies 1-2, WARNING on Policy 3

Multiple policies applied consistently. Clear feedback to teams."

Alex: "Policy composition is powerful?"

Jordan: "Yes. Complex governance from simple rules."

---

Part 10: Policy Maintenance & Versioning

Alex: "How do you evolve policies?"

Jordan: "Policies change as organization matures and threats evolve.

Version Control:

policies/
├── v1/
│   ├── encryption.rego
│   └── tagging.rego
└── v2/
    ├── encryption.rego  (new rule: must use KMS customer keys)
    ├── tagging.rego
    └── network-isolation.rego  (new)

Migration:

1. New policy released (v2)
2. Audit: Current resources compliant with v2?
3. Remediation: Fix non-compliant resources
4. Cutover: Switch default to v2
5. Legacy: v1 still available for exceptions

Sunset: Remove very old policies after migration complete."

Alex: "Backward compatibility?"

Jordan: "Policies should be backward compatible when possible. Breaking changes require migration plan."

---

Part 11: Testing Policies

Alex: "How do you test policies?"

Jordan: "Like code. Automated tests.

Example (Rego/OPA):

Test: S3 bucket with encryption should pass

input := {
  \"resource_type\": \"aws_s3_bucket\",
  \"server_side_encryption_configuration\": {
    \"rule\": [{\"apply_server_side_encryption_by_default\": \"AES256\"}]
  }
}

# Run policy
deny[msg] should produce no messages (PASS)


Test: S3 bucket without encryption should fail

input := {
  \"resource_type\": \"aws_s3_bucket\"
}

# Run policy
deny[msg] should contain \"S3 bucket must have encryption enabled\" (FAIL as expected)

Test suites ensure policies work as intended. As important as testing code."

Alex: "Policies need testing?"

Jordan: "Absolutely. A broken policy blocks all deployments. Must be reliable."

---

Part 12: Policy as Code Performance

Alex: "Does policy checking slow deployments?"

Jordan: "Slightly. Local checks: seconds. Pipeline checks: 1-2 minutes extra.

Performance considerations:

• Local checks: <5 seconds. Keeps developer feedback fast.
• Pipeline checks: <2 minutes. Acceptable for CI/CD.
• Large deployments: Large Terraform plans can take longer. Optimize policies for speed.

Best practice: Tight policies run fast. Overly complex policies can slow pipeline. Balance."

Alex: "Acceptable overhead?"

Jordan: "Yes. Small price for preventing bad deployments."

---

Part 13: Common Policy Pitfalls

Alex: "What goes wrong?"

Jordan: "Common mistakes:

Too Many Policies: Organization drowns in warnings. Teams tune out. Exceptions everywhere.

Solution: Start with 5-10 critical policies. Add incrementally.

Policies Too Strict: Block legitimate use cases. Teams bypass policies or resist adoption.

Solution: Use soft enforcement initially. Allow exceptions.

Policies Not Documented: Teams don't understand why policies exist.

Solution: Every policy has rationale. Documented clearly.

Policies Not Owned: Who maintains policies? Who reviews exceptions? Unclear.

Solution: Assign policy ownership. CCoE owns policy system. Individual policy owner assigned.

No Feedback Loop: Teams violate policies but don't understand why.

Solution: Policy violations include clear explanation. Link to documentation."

Alex: "Governance is nuanced?"

Jordan: "Very. Policy as Code is tool. Organizational discipline is the real work."

---

Part 14: Integrating with Terraform Modules

Alex: "How do Policy as Code and Terraform Modules work together?"

Jordan: "Modules express policies in code. Policies enforce usage.

Example:

# vpc-module/main.tf enforces encryption
resource \"aws_s3_bucket_server_side_encryption_configuration\" \"logs\" {
  bucket = aws_s3_bucket.vpc_flow_logs.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = \"aws:kms\"
      kms_master_key_id = aws_kms_key.logs.arn
    }
  }
}

# Policy checks: S3 bucket must use KMS encryption
policy \"s3_kms_required\" {
  rule = all tfplan.resource_changes as rc {
    rc.type is \"aws_s3_bucket_server_side_encryption_configuration\" implies
    rc.sse_algorithm is \"aws:kms\"
  }
}

Modules provide default implementations. Policies enforce usage of modules and additions."

Alex: "Layered enforcement?"

Jordan: "Yes. Modules are first layer (by design). Policies are second layer (if bypassed)."

---

Part 15: Bridging to FinOps

Alex: "Policy as Code enforces security and compliance. But what about cost?"

Jordan: "That's L4.3: FinOps. Policies optimize spending. Policies prevent wasteful resource creation."

Alex: "So policies control what gets created?"

Jordan: "Yes. Policy as Code ensures safe, compliant creation. FinOps ensures cost-effective creation."

---

Part 16: The Bottom Line

Alex: "If I pitch this?"

Jordan: "Policy as Code automates governance. Policies written as code, enforced automatically. No human review for routine decisions. Teams get immediate feedback. Violations are blocked or tracked. Exceptions are managed formally. Compliance is continuous. Scalable from 10 to 10,000 deployments."

Alex: "And the commitment?"

Jordan: "We're saying: governance is not optional. We enforce it automatically via Policy as Code. Teams innovate within guardrails, not around them."

---

Key Takeaways

Aspect	Details
Core Concept	Policies expressed as code, enforced automatically
Key Tools	Sentinel (Terraform), OPA/Rego (general), AWS Config, Checkov, Kyverno
Enforcement Points	Local (dev workstation), PR stage, pre-apply, post-deployment
Enforcement Levels	Soft (advisory), Medium (with exceptions), Hard (strict)
Exception Process	Formal request, CCoE review, time-limited, tracked, audited
Policy Types	Security, Compliance, Operational, Cost, Organizational
Maintenance	Version controlled, tested, documented, owned
Performance	<5 seconds local, <2 minutes pipeline. Acceptable overhead
Common Pitfall	Too many policies or too strict. Start small, add incrementally