Improving Terraform with Advanced Iterative Patterns

Summary:

In this post, we explore how to scale Infrastructure as Code using Terraform and JSON input. You'll learn how to avoid duplication, reduce complexity, and integrate with Splunk using a clean, iterative approach.


💡Why Declarative IaC Breaks at Scale

Declarative IaC (Infrastructure as Code) is great for managing a handful of resources with limited dependencies. But as your infrastructure grows—sometimes into the hundreds or thousands of objects—maintaining a purely declarative approach becomes cumbersome.

What starts as readable and intuitive code can quickly become:

Bloated Codebase

Managing thousands of lines—despite good naming conventions—can lead to cognitive overload and frustrating navigation.

Brittle Architecture

Making frequent edits to a growing declarative codebase invites human error. Even with static analysis tools (linting, syntax checks, etc.), repetitive resource blocks go against DRY (Don't Repeat Yourself) principles.

‍

</> Using JSON to Drive Infrastructure

Iterative, data-driven modules drastically reduce duplication and complexity. Rather than repeating code for every resource, a few well-abstracted modules ingest external data and dynamically generate infrastructure.

This method encourages a separation of concerns:

  • Code lives in a Terraform repo
  • Data (like JSON) resides in a separate repo or config store

This pattern keeps your Terraform clean and modular. The code changes rarely—only the data does.

‍

🧠 Terraform + Splunk Example with Iterative Modules

Let’s dive into an example. Consider the following JSON representing a collection of Splunk saved searches:

{
  "saved_searches": {
    "error_search": {
      "name": "Error Search",
      "index": "main",
      "sourcetype": "error_logs",
      "search_terms": "error OR failure OR exception",
      "cron_schedule": "*/5 * * * *",
      "private_ips": ["10.10.10.1", "10.10.10.2", "10.10.10.3"]
    },
    "auth_search": {
      "name": "Auth Search",
      "index": "security",
      "sourcetype": "auth_logs",
      "search_terms": "login OR logout OR authentication",
      "cron_schedule": "0 * * * *",
      "private_ips": ["10.10.10.10", "10.10.10.20", "10.10.10.30"]
    }
  }
}

‍

✅ Terraform Module (main.tf)

terraform {
  required_providers {
    splunk = {
      source  = "splunk/splunk"
      version = "1.4.30"
    }
  }
}

# ── Provider Configuration ────────────────────────────────────────────────
provider "splunk" {
  url                 = "127.0.0.1:8089"
  username            = "admin"
  password            = "password"   # ← secure in TF Cloud / Vault
}

# ── Variable Mapping the JSON Structure ───────────────────────────────────
variable "saved_searches" {
  description = "Map of Splunk saved searches and attributes"
  type = map(object({
    name          = string
    index         = string
    sourcetype    = string
    search_terms  = string
    cron_schedule = string
    private_ips   = list(string)
  }))
}

# ── Iterative Resource Creation ───────────────────────────────────────────
resource "splunk_saved_searches" "this" {
  for_each = var.saved_searches

  name          = each.value.name
  description   = "Splunk saved search for ${each.value.name}"
  cron_schedule = each.value.cron_schedule
  disabled      = false
  is_scheduled  = true
  alert_track   = true

  # Dynamic SPL query using HEREDOC and template directives
  search = <<-EOT
    search index=${each.value.index} sourcetype=${each.value.sourcetype}
      | search ${each.value.search_terms}
      | where (
          %{for ip in each.value.private_ips~}
            src_ip="${ip}"
          %{endfor}
        )
      | stats count by src_ip
  EOT
}


🔑 Key Takeaways

  • The Splunk provider is configured to reach loopback (local machine) on port 8089
  • The variable "saved_searches" is a map of objects with several input args defined
  • for_each in the resource block iterates over the JSON map, creating one Splunk saved search resource per entry.
  • A HEREDOC with a for loop injects each private_ip into the SPL query, utilizing a string template directive
  • Updating or adding a saved search now means editing JSON, not Terraform code.


🖥️ Run Locally with Docker

Let's get this running locally! To start, run the Splunk docker container via the docker run command:

# Start a single‑node Splunk instance (accepts license, enables ssl)
$ docker run -d \
  -p 8000:8000 -p 8089:8089 \
  -e "SPLUNK_START_ARGS=--accept-license" \
  -e "SPLUNK_PASSWORD<password" \
  -e SPLUNK_HTTP_ENABLESSL=true \
  splunk/splunk

‍

Tail logs until splunk is ready (use image id from container created by run command)

$ docker logs -f ba259c2b6262

PLAY RECAP *********************************************************************
localhost                  : ok=84   changed=11   unreachable=0    failed=0    skipped=82
rescued=0    ignored=0
Friday 02 May 2025  02:19:39 +0000 (0:00:00.051)       0:01:15.556 ************
===============================================================================
splunk_common : Start Splunk via CLI ----------------------------------- 26.94s
splunk_common : Update /opt/splunk/etc --------------------------------- 13.27s
splunk_common : Get Splunk status --------------------------------------- 2.57s
splunk_common : Update Splunk directory owner --------------------------- 2.29s
Gathering Facts --------------------------------------------------------- 1.55s
splunk_common : Check if requests_unixsocket exists --------------------- 1.49s
splunk_common : Test basic https endpoint ------------------------------- 1.41s
splunk_common : Generate user-seed.conf (Linux) ------------------------- 1.11s
splunk_standalone : Get existing HEC token ------------------------------ 1.00s
splunk_common : Cleanup Splunk runtime files ---------------------------- 0.90s
splunk_standalone : Check for required restarts ------------------------- 0.79s
splunk_standalone : Setup global HEC ------------------------------------ 0.77s
splunk_common : Hash the password --------------------------------------- 0.70s
Check for required restarts --------------------------------------------- 0.70s
splunk_common : Check for scloud ---------------------------------------- 0.65s
splunk_common : Check for existing splunk secret ------------------------ 0.63s
splunk_common : Wait for splunkd management port ------------------------ 0.62s
splunk_common : Remove splunktcp-ssl input ------------------------------ 0.55s
splunk_common : Get Splunk status --------------------------------------- 0.54s
splunk_common : Enable Splunkd SSL -------------------------------------- 0.54s
===============================================================================

Ansible playbook complete, will begin streaming splunkd_stderr.log

‍

Run terraform apply

$ terraform init
$ terraform apply -var-file="saved_searches.json"


Expect output similar to:

Plan: 2 to add, 0 to change, 0 to destroy.
...
Apply complete! Resources: 2 added.

‍

Verify in Splunk Web

  1. Browse to https://127.0.0.1:8000 → Search & Reporting → Alerts.
  2. Confirm the Auth Search and Error Search alerts are present with the generated SPL.


You should see a Splunk query similar to the following:

search index=security sourcetype=auth_logs
| search login OR logout OR authentication
| where (
          src_ip="10.10.10.10"
          src_ip="10.10.10.20"
          src_ip="10.10.10.30"
)
| stats count by src_ip

‍
‍
💭 Conclusion & Next Steps

Moving from a purely declarative style to iterative, JSON‑driven Terraform modules keeps your codebase lean, DRY, and ready for scale. Separate logic from data, embrace for_each, and watch maintenance overhead drop.

Ready to level‑up your IaC?

  • ⭐ Star this pattern in your internal playbook.
  • 💬 Reach out to us to implement this pattern in your org.
  • 🔔 Follow the blog for more Terraform iterative modules, Splunk integrations, and scalable DevOps workflows.

Happy Terraforming! 🚀