Terraform Deep-Dive

How Terraform Works

Every Terraform run walks the same loop: read your desired state from HCL, refresh what's actually running, diff the two, then reconcile reality to match the diff.

graph LR
    classDef authored fill:#3498db,stroke:#2980b9,color:#fff
    classDef readonly fill:#9b59b6,stroke:#8e44ad,color:#fff
    classDef planout fill:#e67e22,stroke:#d35400,color:#fff
    classDef mutate fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef state fill:#2c3e50,stroke:#1a252f,color:#fff

    subgraph LOCAL["Local workspace"]
        HCL["*.tf files<br/>desired state, human-authored"]:::authored
        INIT["terraform init<br/>download providers, init backend"]:::authored
        HCL --> INIT
    end

    subgraph PLANPHASE["Read-only planning — safe to run anytime, never touches real infra"]
        REFRESH["Refresh<br/>read real infra via provider API"]:::readonly
        DIFF["Compute diff<br/>desired HCL vs refreshed reality"]:::readonly
        PLANOUT["Execution plan<br/>+ create, ~ update, - destroy"]:::planout
        REFRESH --> DIFF --> PLANOUT
    end

    subgraph MUTATE["Mutating phase"]
        APPLY["terraform apply<br/>real create/update/destroy API calls"]:::mutate
        STATEFILE["terraform.tfstate<br/>rewritten to record new reality"]:::state
        APPLY --> STATEFILE
    end

    INIT --> REFRESH
    PLANOUT -->|"human or CI approval"| APPLY
    STATEFILE -.->|"next run's refresh reads from here"| REFRESH

The plan → apply lifecycle, one phase at a time

Core Commands

Workspaces

Workspaces give you isolated state files within the same backend and same codebase. Each workspace has its own state.

graph TD
    classDef neutral fill:#3498db,stroke:#2980b9,color:#fff
    classDef staging fill:#e67e22,stroke:#d35400,color:#fff
    classDef prod fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef shared fill:#f39c12,stroke:#d68910,color:#000

    subgraph BUCKET["Single S3 bucket + single DynamoDB lock table — shared by every workspace"]
        BACKEND["s3://my-tf-state/<br/>same credentials, same bucket policy for all"]:::shared
    end

    BACKEND --> WS_DEFAULT["default workspace<br/>key: terraform.tfstate"]:::neutral
    BACKEND --> WS_STAGING["staging workspace<br/>key: env:/staging/terraform.tfstate"]:::staging
    BACKEND --> WS_PROD["prod workspace<br/>key: env:/prod/terraform.tfstate"]:::prod

    WS_DEFAULT -.->|"same .tf files,<br/>same provider version"| WS_STAGING
    WS_STAGING -.->|"same .tf files,<br/>same provider version"| WS_PROD

The dotted arrows above are the whole limitation in one picture: every workspace resolves to the same HCL and the same provider version pin. Only the state key changes.

terraform workspace new staging      # create workspace
terraform workspace select prod      # switch to prod
terraform workspace list             # list all workspaces
terraform workspace show             # current workspace

# Use workspace name to vary resources
locals {
  instance_type = terraform.workspace == "prod" ? "t3.large" : "t3.micro"
}

resource "aws_instance" "web" {
  instance_type = local.instance_type
  tags = {
    Environment = terraform.workspace
  }
}

Limitation: Workspaces share the same backend and codebase. For true env isolation (separate AWS accounts, different backend configs), use separate directories or Terragrunt.

When NOT to use workspaces

The workspace-per-env anti-pattern: Many teams use workspaces for prod/staging/dev with the same codebase. This works until prod needs a different module version, different provider config, or different backend credentials. At that point, separate directories win.

Workspaces for ephemeral environments (correct use)

# Create per-PR environment
terraform workspace new "pr-${PR_NUMBER}"
terraform apply -var="suffix=pr-${PR_NUMBER}"

# Destroy after PR merge
terraform workspace select "pr-${PR_NUMBER}"
terraform destroy -auto-approve
terraform workspace select default
terraform workspace delete "pr-${PR_NUMBER}"

Terragrunt — true env isolation

Terragrunt wraps Terraform, giving each environment its own backend config and variable file without duplicating HCL. Where a workspace changes only the state key, Terragrunt lets every environment vary independently — different AWS account, different provider pin, different backend bucket — while still sharing the underlying module source.

infra/
├── modules/vpc/          # shared module (DRY)
├── prod/
│   ├── terragrunt.hcl    # prod backend + inputs
│   └── vpc/terragrunt.hcl
└── staging/
    ├── terragrunt.hcl    # staging backend + inputs
    └── vpc/terragrunt.hcl

# staging/terragrunt.hcl
remote_state {
  backend = "s3"
  config = {
    bucket = "my-tf-state-staging"
    key    = "${path_relative_to_include()}/terraform.tfstate"
    region = "us-east-1"
  }
}

inputs = {
  environment = "staging"
  instance_type = "t3.micro"
}

cd staging/vpc
terragrunt apply   # uses staging backend, staging inputs automatically

Remote Backend (S3 + DynamoDB)

Every production Terraform config must use a remote backend — local state is never safe in teams.

terraform {
  backend "s3" {
    bucket         = "my-company-tf-state"
    key            = "services/api/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true                       # SSE-S3 or SSE-KMS
    kms_key_id     = "arn:aws:kms:..."          # optional: use CMK
    dynamodb_table = "terraform-state-lock"     # lock table
  }
}

Bootstrap the backend (one-time):

# Create the S3 bucket
aws s3api create-bucket --bucket my-company-tf-state --region us-east-1
aws s3api put-bucket-versioning --bucket my-company-tf-state \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket my-company-tf-state \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
aws s3api put-public-access-block --bucket my-company-tf-state \
  --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Create the DynamoDB lock table
aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

State key conventions:

# Per-service, per-environment
services/api/prod/terraform.tfstate
services/api/staging/terraform.tfstate
services/worker/prod/terraform.tfstate
infra/vpc/prod/terraform.tfstate

Migrate from local to remote:

# Add backend config to main.tf, then:
terraform init -migrate-state
# Terraform uploads local .tfstate to S3 automatically

State locking flow

The dynamodb_table entry in the backend block isn't optional decoration — it's what stops two concurrent runs from corrupting the same state file. Here's what actually happens on every plan/apply:

moved Block — Safe Refactoring

When renaming a resource or moving it to a module, use moved to tell Terraform the resource is the same — no destroy + recreate.

# Before: resource "aws_instance" "server"
# After:  resource "aws_instance" "web_server"

moved {
  from = aws_instance.server
  to   = aws_instance.web_server
}

# Moving a resource into a module
moved {
  from = aws_security_group.api
  to   = module.api.aws_security_group.main
}

# Moving a for_each resource
moved {
  from = aws_iam_user.legacy["alice"]
  to   = aws_iam_user.app_users["alice"]
}

Workflow:

# 1. Add moved block to config
# 2. Run plan — should show 0 resources to add/destroy
terraform plan
# Output: # aws_instance.web_server has moved to aws_instance.server
#           No changes. Your infrastructure matches the configuration.

# 3. Apply (updates state file only, no real changes)
terraform apply

# 4. Remove the moved block after apply (it's a one-shot migration)

Why not terraform state mv? state mv modifies state directly without a plan step and leaves no record in code. moved blocks are code-reviewable, repeatable, and self-documenting.

check Block — Continuous Assertions

check blocks validate assumptions about your infrastructure on every plan/apply. Unlike precondition/postcondition, they warn rather than error.

# Assert that the ALB responds with 200 (non-blocking warning)
check "alb_health" {
  data "http" "alb" {
    url = "https://${aws_lb.api.dns_name}/health"
  }

  assert {
    condition     = data.http.alb.status_code == 200
    error_message = "ALB health check returned ${data.http.alb.status_code}"
  }
}

# Assert an S3 bucket is not publicly accessible
check "s3_not_public" {
  assert {
    condition     = aws_s3_bucket_public_access_block.api.block_public_acls == true
    error_message = "S3 bucket must have public access blocked"
  }
}

Checks appear in terraform plan output as warnings — they do not prevent apply. Use them for invariants you want surfaced every run.

terraform test Framework (1.6+)

Write unit/integration tests for Terraform modules in .tftest.hcl files.

modules/
└── s3-bucket/
    ├── main.tf
    ├── variables.tf
    ├── outputs.tf
    └── tests/
        ├── defaults.tftest.hcl
        └── versioning_enabled.tftest.hcl

# modules/s3-bucket/tests/defaults.tftest.hcl

# Variables for this test run
variables {
  bucket_name = "test-bucket-abc123"
  environment = "test"
}

# Test 1: verify default outputs
run "default_configuration" {
  command = plan   # plan-only (no real resources created)

  assert {
    condition     = output.bucket_name == "test-bucket-abc123"
    error_message = "Bucket name output incorrect"
  }

  assert {
    condition     = aws_s3_bucket.this.tags["Environment"] == "test"
    error_message = "Environment tag not set correctly"
  }
}

# Test 2: actually create the bucket and verify
run "apply_and_verify" {
  command = apply   # creates real resources

  assert {
    condition     = aws_s3_bucket.this.bucket == "test-bucket-abc123"
    error_message = "Bucket not created with correct name"
  }
}

# Run all tests in the module
terraform test

# Run a specific test file
terraform test -filter=tests/defaults.tftest.hcl

# Output:
# defaults.tftest.hcl... in progress
#   run "default_configuration"... pass
#   run "apply_and_verify"... pass
# Success! 2 passed, 0 failed.

Test isolation: Each run block that uses apply creates and destroys real resources in a temporary workspace. Add -var="environment=test" or use mock providers to avoid hitting real AWS.

# Mock provider for fast plan-only tests (no AWS calls)
mock_provider "aws" {}

run "plan_only_with_mock" {
  command = plan
  # Uses mock provider — instant, free, no AWS credentials needed
}

terraform import

Bring an existing real resource under Terraform management:

# Classic syntax
terraform import aws_instance.web i-1234567890abcdef0

# Declarative import (Terraform 1.5+) — define in config, then run apply
import {
  to = aws_instance.web
  id = "i-1234567890abcdef0"
}

Use when: resource was created manually, you need to adopt it without recreating it. After import, write matching config to avoid plan showing changes.

terraform state rm

Remove a resource from state without destroying the real resource:

terraform state rm aws_instance.web
terraform state rm 'module.vpc.aws_subnet.public[0]'  # escape brackets in shell

Use when: module refactoring (remove from one module, re-import to another), or stop managing a resource.

terraform apply -replace

Force destroy + recreate of a specific resource:

terraform apply -replace="aws_instance.web"
terraform apply -replace="module.ecs.aws_ecs_service.app"

Use when: resource is subtly broken but Terraform thinks it's healthy. Replaced the deprecated terraform taint command (v0.15.2+).

State file operations

terraform state list                          # list all resources in state
terraform state show aws_instance.web         # show details of one resource
terraform state mv aws_instance.old aws_instance.new  # rename/move resource
terraform state pull > backup.tfstate         # backup state

Drift Detection and Fix

# Detect drift: shows resources that changed outside Terraform
terraform plan   # unexpected changes = drift

# Sync state without making changes (inspect what drifted)
terraform apply -refresh-only

# Fix option 1: let Terraform correct it
terraform apply   # Terraform reverts manual change back to desired state

# Fix option 2: accept the manual change
terraform apply -refresh-only   # update state to match reality
# Then update your .tf config to match

DB User Management Example

Full example: create multiple DB users with for_each, random passwords stored in Secrets Manager.

terraform {
  required_providers {
    mysql = {
      source  = "petoju/mysql"
      version = "~> 3.0"
    }
  }
}

locals {
  # Set is stable — adding/removing one user only affects that user
  db_users = toset(["svc_payments", "svc_reporting", "svc_audit"])
}

resource "random_password" "passwords" {
  for_each = local.db_users
  length   = 24
  special  = false
}

resource "mysql_user" "app_users" {
  for_each = local.db_users

  user               = each.key
  host               = "%"
  plaintext_password = random_password.passwords[each.key].result
}

resource "aws_secretsmanager_secret" "db_passwords" {
  for_each = local.db_users
  name     = "db/${each.key}/password"
}

resource "aws_secretsmanager_secret_version" "db_passwords" {
  for_each      = local.db_users
  secret_id     = aws_secretsmanager_secret.db_passwords[each.key].id
  secret_string = random_password.passwords[each.key].result
}

Key points:

• for_each not count — keyed by username, safe to remove a user without cascading destroy
• random_password IS stored in state — encrypt your state file (S3 SSE + KMS)
• Passwords stored in Secrets Manager — apps use IRSA to fetch at runtime, never in environment variables

Version History

timeline
    title Terraform Major Milestones
    2019 : 0.12 - HCL2, proper type system, first-class expressions. Big syntax break from 0.11.
    2020 : 0.13 - Module source iteration with for_each and count
    2020 : 0.14 - Sensitive values in state, provider lock file .terraform.lock.hcl
    2021 : 0.15 and 1.0 - Stable API guarantee. Deprecated taint replaced by -replace flag.
    2021 : 1.1 - moved_from block for refactoring without state rm and re-import
    2022 : 1.3 - Optional attributes in object types
    2023 : 1.5 - import block in config: declarative import
    2023 : 1.6 - Built-in test framework: terraform test
    2024 : 1.8 - Provider-defined functions

terraform version   # check current version

Current stable: Terraform 1.x (1.6–1.9 range as of 2025–2026). OpenTofu is the OSS fork maintained by the community after HashiCorp's BSL license change in 2023.

HCL Patterns

Dynamic blocks

# Instead of repeating ingress blocks
resource "aws_security_group" "web" {
  dynamic "ingress" {
    for_each = var.allowed_ports
    content {
      from_port   = ingress.value
      to_port     = ingress.value
      protocol    = "tcp"
      cidr_blocks = ["0.0.0.0/0"]
    }
  }
}

Local values

locals {
  common_tags = {
    Project     = var.project
    Environment = terraform.workspace
    ManagedBy   = "terraform"
  }
}

resource "aws_instance" "web" {
  tags = merge(local.common_tags, { Name = "web-server" })
}

Data sources

# Reference existing resources not managed by this config
data "aws_vpc" "main" {
  filter {
    name   = "tag:Name"
    values = ["prod-vpc"]
  }
}

resource "aws_subnet" "app" {
  vpc_id = data.aws_vpc.main.id
  # ...
}

Module outputs and dependencies

module "vpc" {
  source = "./modules/vpc"
  cidr   = "10.0.0.0/16"
}

module "ecs" {
  source    = "./modules/ecs"
  vpc_id    = module.vpc.vpc_id          # implicit dependency
  subnet_ids = module.vpc.private_subnets
}

State Surgery — Advanced State Management

terraform import (legacy) vs import block (1.5+)

Old way — CLI import (not in state, not reviewable):

# Import existing AWS resource into state (one at a time, no plan preview)
terraform import aws_s3_bucket.my_bucket my-existing-bucket-name
terraform import aws_instance.web i-0123456789abcdef0
# Problem: no dry-run, no code generation, easy to get wrong address

New way — import block (Terraform 1.5+, preferred):

# import.tf — commit this, review in PR, run in CI
import {
  id = "my-existing-bucket-name"
  to = aws_s3_bucket.my_bucket
}

import {
  id = "i-0123456789abcdef0"
  to = aws_instance.web
}

# With import block you get a full plan showing what will be imported
terraform plan   # shows: "will import aws_s3_bucket.my_bucket"

# -generate-config-out: auto-generate HCL from the live resource
terraform plan -generate-config-out=generated.tf
# Produces HCL with all attributes filled from AWS — clean up and commit

terraform apply  # import happens as part of normal apply

moved block — safe resource address refactoring

When you rename a resource in HCL or move it into/out of a module, Terraform sees it as destroy+create without a moved block.

# Renamed resource: aws_instance.old_name → aws_instance.new_name
moved {
  from = aws_instance.old_name
  to   = aws_instance.new_name
}

# Moved into a module: aws_s3_bucket.logs → module.storage.aws_s3_bucket.logs
moved {
  from = aws_s3_bucket.logs
  to   = module.storage.aws_s3_bucket.logs
}

# Moved from one module call to another
moved {
  from = module.app["service-a"]
  to   = module.app["service-b"]
}

# Verify with plan — should show "move" not "destroy+create"
terraform plan
# ~ aws_instance.new_name (moved from aws_instance.old_name)
#   # (no changes to resource, just address change)

terraform state commands — surgical operations

# List all resources in state
terraform state list
terraform state list | grep aws_security_group

# Show full details of one resource in state
terraform state show aws_s3_bucket.my_bucket
# Outputs all attributes as they exist in state — useful for debugging diffs

# Remove a resource from state WITHOUT destroying it
# (hand off to another workspace, or stop managing it)
terraform state rm aws_s3_bucket.my_bucket

# Move resource between state files (e.g., splitting monolith into modules)
# In source workspace:
terraform state mv aws_s3_bucket.logs module.storage.aws_s3_bucket.logs
# WARNING: modifies state directly with no plan. Take a backup first.

# Pull remote state to local for inspection
terraform state pull > state_backup_$(date +%Y%m%d).json

# Push modified state back (DANGEROUS — use only for corruption recovery)
terraform state push state_backup.json

# Manually take a state lock (useful for maintenance windows)
terraform force-unlock <lock-id>   # release stuck lock after crash

Module refactoring — splitting a monolith

graph LR
    classDef before fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef after fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef child fill:#3498db,stroke:#2980b9,color:#fff

    subgraph BEFORE["Before"]
        MONO["Single root module<br/>directly manages ~50 resources"]:::before
    end

    subgraph AFTER["After"]
        ROOT["Root module<br/>calls child modules only"]:::after
        NET["module.network<br/>VPC, subnets, routing"]:::child
        COMP["module.compute<br/>ASG, ECS, EC2"]:::child
        DB["module.database<br/>RDS, parameter groups"]:::child
        ROOT --> NET
        ROOT --> COMP
        ROOT --> DB
    end

    BEFORE -.->|"moved blocks re-address<br/>every resource — zero destroy/create"| AFTER

# Step 1: write new module code
# Step 2: add moved blocks for every resource being re-addressed
# Step 3: terraform plan — verify zero destroy/create, only moves
# Step 4: terraform apply — moves are instantaneous (metadata only)
# Step 5: remove moved blocks in a follow-up PR (they're only needed once)

# Example: moving 5 resources into a network module
moved { from = aws_vpc.main              to = module.network.aws_vpc.main }
moved { from = aws_subnet.private_a      to = module.network.aws_subnet.private["a"] }
moved { from = aws_subnet.private_b      to = module.network.aws_subnet.private["b"] }
moved { from = aws_internet_gateway.igw  to = module.network.aws_internet_gateway.main }
moved { from = aws_route_table.public    to = module.network.aws_route_table.public }

Targeted apply — breaking the plan cycle

# Apply only specific resources (bypass unrelated failures)
terraform apply -target=aws_s3_bucket.my_bucket
terraform apply -target=module.network
terraform apply -target=aws_security_group.app -target=aws_security_group.db

# WARNING: targeted apply leaves state inconsistent — dependencies may be stale
# Always follow with a full plan+apply to ensure consistency
# Never use -target in automated pipelines

# Similarly for plan (useful for understanding impact)
terraform plan -target=module.network

replace — force recreation of a single resource

# Taint is deprecated since 1.2. Use -replace instead.
terraform apply -replace=aws_instance.web
# Equivalent to: destroy + create in a single apply
# Use when: resource is corrupted, needs AMI refresh, or is in bad state

State lock debugging

# State is locked when:
# - Another terraform apply is running
# - A previous run crashed without releasing the lock
# - DynamoDB lock table entry is stale

# Check DynamoDB for stuck lock
aws dynamodb get-item \
  --table-name terraform-state-lock \
  --key '{"LockID": {"S": "mybucket/path/to/terraform.tfstate"}}' \
  --region us-east-1

# Release stuck lock (confirm no apply is actually running first)
terraform force-unlock <lock-id>
# Lock ID is in the error message:
# "Error acquiring the state lock: ID: abc-123-def..."