Terraform in Production: What Breaks at 400+ Resource Groups — Tanmay Trivedi

The Monolithic State Trap

When starting with Terraform on Azure, keeping all resource groups in a single root module feels convenient. But as an enterprise footprint scales to 40+ subscriptions and 400+ resource groups, a monolithic state architecture becomes a severe operational liability. Running a single 'terraform plan' can take 45+ minutes, and a state locking conflict can freeze deployments across the entire engineering organization.

Never let a single Terraform state file manage multiple distinct lifecycle boundaries. Separate networking, identity, and application tiers.

Plan Bottlenecks & API Rate Limits

During 'terraform refresh', the AzureRM provider queries the Azure Resource Manager API for every single resource declared in the state. At 400+ resource groups, you will quickly hit Azure Resource Manager read rate limits (12,000 read requests per hour per subscription), triggering HTTP 429 throttling and failing CI/CD pipeline runs.

backend.tfhcl

terraform {
  backend "azurerm" {
    resource_group_name  = "rg-tfstate-prod"
    storage_account_name = "sttfstateprod01"
    container_name       = "tfstate"
    key                  = "networking/hub-vnet.tfstate"
  }
}

State Partitioning & Remote Data Sources

We restructured the repository into isolated, composable workspaces partitioned by lifecycle and blast radius. The central connectivity hub has its own isolated state file. Each spoke workload maintains an independent state file that queries hub outputs via 'terraform_remote_state' data sources or Azure Resource Graph queries.

spoke-networking.tfhcl

data "terraform_remote_state" "hub" {
  backend = "azurerm"
  config = {
    resource_group_name  = "rg-tfstate-prod"
    storage_account_name = "sttfstateprod01"
    container_name       = "tfstate"
    key                  = "networking/hub-vnet.tfstate"
  }
}

resource "azurerm_virtual_network_peering" "spoke_to_hub" {
  name                      = "peer-spoke-to-hub"
  resource_group_name       = azurerm_resource_group.spoke.name
  virtual_network_name      = azurerm_virtual_network.spoke.name
  remote_virtual_network_id = data.terraform_remote_state.hub.outputs.hub_vnet_id
}

Hard-Won Takeaways

1. Blast radius containment is your highest priority: An accidental destructive change in a spoke workspace must never have the ability to impact the central hub firewall or ExpressRoute gateways. 2. Pin provider versions explicitly: Never allow unpinned ~> minor provider upgrades in CI/CD without automated staging validation. 3. Automate drift detection: Run daily read-only scheduled plans to catch out-of-band console changes before they cause deployment conflicts.