Terraform Data Infrastructure for SaaS: How We Manage It

Terraform data infrastructure is the practice of managing data platforms—including warehouses, lakes, and pipelines—using code rather than manual console configurations. In our work with mid-market SaaS companies, we have found that managing cloud resources through a declarative language is the only way to maintain the auditability and reliability required for production AI systems. Using terraform data infrastructure allows our team to treat a BigQuery dataset or an IAM permission with the same rigor as an application feature.

When a data team moves from "clicking buttons in the UI" to managing infrastructure as code (IaC), they solve the problem of environment drift. We have seen many organizations where the "Development" warehouse has different permissions than "Production" because a teammate manually added a service account three months ago and forgot to document it. Terraform eliminates this ambiguity.

What is terraform data infrastructure?

Terraform data infrastructure is a methodology where data-specific cloud resources—such as BigQuery datasets, Snowflake warehouses, AWS S3 buckets, and IAM roles—are defined in HashiCorp Configuration Language (HCL) files. These files act as the single source of truth for the environment's state, allowing teams to version control their data stack, automate deployments, and replicate environments across multiple regions or stages.

In our experience, terraform data infrastructure provides three primary benefits to scaling data teams:

1. Repeatability: You can spin up a "staging" environment that is an exact mirror of "production" in minutes.
2. Auditability: Every change to a database schema or a permission is recorded in a git commit history.
3. Governance: You can enforce naming conventions and security policies globally across all data assets.

Why use Terraform for data infrastructure specifically?

Data infrastructure has unique requirements compared to standard application infrastructure. While a DevOps engineer might use Terraform to manage Kubernetes clusters or VPCs, a data engineer uses it to manage the lifecycle of data. We focus on managing the "container" of the data—the datasets, the tables (if they are external), and the access controls—while leaving the "logic" of the data (the SQL transformations) to tools like dbt.

If you are evaluating your team's current setup, our AI Readiness Diagnostic provides a structured assessment of whether your foundation can support automated infrastructure management.

When we build terraform data infrastructure for clients, we separate the concerns into three distinct layers:

1. The Foundation Layer: VPCs, networking, and core project settings.
2. The Storage Layer: BigQuery datasets, Cloud Storage buckets, and Pub/Sub topics.
3. The Access Layer: Service accounts, IAM roles, and row-level security policies.

How do we structure terraform data infrastructure for SaaS?

For a scaling SaaS company, a flat directory of Terraform files quickly becomes unmanageable. We recommend a modular structure that separates resources by function and environment. This ensures that a change to a development dataset cannot accidentally delete a production table.

The Module Pattern

We build reusable modules for common data patterns. For example, a "BigQuery Dataset Module" would include the dataset definition, default encryption keys, and a standard set of IAM bindings for the data engineering team.

# modules/bigquery_dataset/main.tf

resource "google_bigquery_dataset" "dataset" {
  dataset_id                  = var.dataset_id
  friendly_name               = var.friendly_name
  location                    = "US"
  default_table_expiration_ms = var.is_production ? null : 3600000 # Expire dev data
  
  labels = {
    env      = var.environment
    managed_by = "terraform"
  }
}

resource "google_bigquery_dataset_iam_binding" "reader" {
  dataset_id = google_bigquery_dataset.dataset.dataset_id
  role       = "roles/bigquery.dataViewer"
  members    = var.reader_members
}

By using this module, we ensure that every dataset in the company follows the same labeling and expiration logic. This is a core component of the Data Foundation we implement for our clients, ensuring that infrastructure remains lean and organized.

Environment Separation

We never manage multiple environments (Dev, Staging, Prod) in a single Terraform state file. Instead, we use separate directories or workspaces. For most SaaS data teams, separate directories are preferred because they allow for different versions of modules to be tested in Dev before being promoted to Prod.

infrastructure/
├── modules/
│   ├── bigquery/
│   ├── gcs/
│   └── iam/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   └── terraform.tfvars
│   └── prod/
│       ├── main.tf
│       └── terraform.tfvars

Managing BigQuery resources with Terraform

BigQuery is the heart of the modern data stack for many SaaS companies. Managing it via terraform data infrastructure requires a delicate balance. You want Terraform to manage the datasets and the permissions, but you typically want dbt to manage the actual tables and views within those datasets.

Dataset Ownership

Terraform should own the google_bigquery_dataset resource. This allows you to manage:

• Location: Ensuring all data stays within specific geographic boundaries for GDPR/compliance.
• Access Control: Defining which service accounts (e.g., Fivetran, Airbyte, dbt) have permission to write to which datasets.
• Encryption: Managing Customer-Managed Encryption Keys (CMEK) if required.

Avoiding Resource Conflict

A common mistake we see is trying to manage every individual table in Terraform. For a SaaS company with hundreds of tables, this leads to massive Terraform state files and slow execution times. Instead, we use Terraform to create the "landing zones" (raw datasets) and "transformation zones" (analytics datasets).

We then use IAM roles to give dbt the roles/bigquery.dataEditor role on the analytics datasets. This creates a clean handoff: Terraform builds the warehouse "rooms," and dbt arranges the "furniture" inside them.

Handling secrets and service accounts

Security is often the primary driver for adopting terraform data infrastructure. Manual management of JSON keys for service accounts is a significant security risk. We follow the principle of least privilege, creating specific service accounts for each tool in the pipeline.

For example, a Fivetran service account only needs permission to write to the raw_ datasets. It should never have permission to read from the analytics_ datasets or delete data in other projects.

resource "google_service_account" "fivetran_loader" {
  account_id   = "fivetran-loader"
  display_name = "Fivetran Data Loader"
}

resource "google_project_iam_member" "fivetran_bq_owner" {
  project = var.project_id
  role    = "roles/bigquery.dataEditor"
  member  = "serviceAccount:${google_service_account.fivetran_loader.email}"
}

Instead of downloading JSON keys, we recommend using Workload Identity Federation or short-lived tokens whenever possible. If you must use keys, they should be stored in a secret manager (like Google Secret Manager or AWS Secrets Manager), with the secret itself being managed by Terraform.

Common pitfalls when scaling terraform data infrastructure

Even with a strong start, data teams often run into hurdles as their terraform data infrastructure grows. Here is how we navigate those challenges.

State Drift from "Hotfixes"

In a crisis, a team member might manually change a permission in the console to "unblock" a pipeline. This creates state drift. When Terraform next runs, it will attempt to revert that manual change. The Solution: Implement a CI/CD pipeline (like GitHub Actions or GitLab CI) that runs terraform plan on every pull request. This makes the infrastructure changes visible to the whole team before they are applied.

Circular Dependencies

In complex data stacks, you might have a service account that needs access to a bucket, but the bucket's policy needs to reference the service account. Terraform handles most of these dependencies automatically, but highly nested modules can occasionally cause "cycle" errors. The Solution: Keep modules flat. Avoid nesting modules more than one level deep. Use "data sources" to reference existing resources rather than passing every single resource object between modules.

Managing dbt Cloud with Terraform

Many teams forget that their orchestration and transformation layers are also part of their infrastructure. We use the dbt Cloud Terraform provider to manage projects, environments, and jobs. This ensures that when a new data engineer joins the team, their access to dbt Cloud is provisioned automatically via code.

Why governance belongs in your Terraform code

Governance is often treated as a post-hoc activity—something a compliance officer checks once a quarter. By moving governance into your terraform data infrastructure, it becomes proactive.

We use Terraform to enforce:

• Resource Labeling: Every dataset must have an owner and a cost_center label.
• Deletion Protection: Production datasets have deletion_protection = true to prevent accidental terraform destroy mishaps.
• VPC Service Controls: Ensuring that data cannot be exported to unauthorized external IP addresses.

For teams looking to master these patterns, we cover the intersection of infrastructure and analytics in our Learn AI Bootcamp. We teach practitioners how to build these systems so they can move from being "data cleaners" to "system architects."

Frequently Asked Questions About terraform data infrastructure

Should I manage individual BigQuery tables in Terraform?

No. In most SaaS use cases, tables are dynamic and change frequently based on upstream application schemas or dbt transformations. Managing them in Terraform creates high maintenance overhead. Use Terraform to manage datasets and permissions, and use dbt or your ETL tool to manage the tables within those datasets.

How do I handle existing infrastructure that wasn't built with Terraform?

You can use the terraform import command to bring existing cloud resources under Terraform management. We recommend a "piece-by-piece" approach: start by importing your most critical datasets and service accounts, then gradually move to networking and secondary storage.

What is the best way to store Terraform state for a data team?

Always use a remote backend with state locking. For GCP users, this is a Cloud Storage bucket with versioning enabled. This prevents two team members from running terraform apply at the same time and corrupting the state file.

Does Terraform replace dbt?

No. Terraform and dbt serve different purposes. Terraform manages the "infrastructure" (the warehouse, storage, and permissions), while dbt manages the "data models" (the SQL logic that transforms raw data into insights). They are complementary tools in a modern data stack.

How often should we run Terraform applies?

Infrastructure doesn't change as often as data models. For most teams, running Terraform applies via a CI/CD pipeline whenever a Pull Request is merged to the main branch is sufficient. This ensures that the live environment always matches the code in your repository.

Ready to build a production-grade data foundation?

If your team is struggling with manual configurations, permission errors, or a lack of visibility into your cloud costs, it is time to formalize your infrastructure. We help scaling data teams move from fragile, manual setups to robust, automated systems.

We cover the implementation of these patterns in detail in our Learn AI Bootcamp. Whether you are looking to deploy production AI agents or simply want to clean up your data engineering stack, our team provides the blueprint and the hands-on training to get you there. Book a free consultation to talk through your current architecture and identify the quickest path to a managed, automated data foundation.