Signals #4: Mapping the Terraform Module Ecosystem

The Terraform Registry has thousands of AWS modules. Some are maintained. Most are abandoned. A few namespaces dominate. Which resources appear together? Which upstream modules get composed into new ones? Who ships continuously?

This signal crawls the entire registry, extracts structural metadata from source code, and produces datasets that answer these questions.

Datasets: Two JSON files served via CloudFront — one for all AWS modules, one for namespaces ranked by count.

---

The Registry API

Terraform’s registry exposes a paginated JSON API:

https://registry.terraform.io/v1/modules?limit=100&offset=0&provider=aws

Each page returns up to 100 modules with metadata: namespace, name, version, downloads, source URL, publish date.

The Crawler

source scripts/fetch-modules.sh

Fetch All Pages

fetch_modules () {
  local offset="${1:-0}"
  while :; do
    http_code=$(curl -sS -w '%{http_code}' -o "$OUTDIR/modules_${offset}.json" \
      "${BASE}?limit=${limit}&offset=${offset}&provider=aws")

    next=$(jq -r '.meta.next_offset // empty' "$OUTDIR/modules_${offset}.json")
    [[ "$http_code" == "429" ]] && echo "Rate limited, stopping." >&2 && break
    [[ -z "$next" ]] && break
    offset=$next
    sleep "$SLEEP"
  done
}

Handles rate limiting (429), pagination, and stores raw responses. Typically completes in 2-3 minutes for ~5000 modules.

Parse and Enrich

parse_modules () {
  cat "$OUTDIR"/*.json \
    | jq '.modules[]| . + {
      path: [.namespace, .name, .provider]|join("/"),
      registry_url: "https://registry.terraform.io/v1/modules/" + .namespace + "/" + .name + "/" + .provider,
      repo: .source|split("https://github.com/")|last
    }'
}

Adds computed fields: a canonical path, direct registry URL, and extracted GitHub repo path.

Namespace Ranking

parse_namespaces () {
  jq 'group_by(.namespace)[]| {
    namespace: first.namespace,
    url: "https://registry.terraform.io/modules/" + first.namespace,
    size: length
  }' | jq -sc 'sort_by(.size)|reverse'
}

Who publishes the most modules? CloudPosse, HashiCorp, a few others. Then a long tail of single-module namespaces.

Full Pipeline

update_dataset

One command: fetch → parse → rank → upload to S3.

Deep Metadata: Source Code Inspection

The registry index tells you what exists. To understand how modules are built, you need the source.

Filter Recent, Active Modules

fetch_subset /tmp/tf-aws-modules.json > /tmp/subset.json

Filters modules published in the last 30 days with 100-1000 downloads — active but not the mega-popular ones everyone already knows.

Download and Inspect

generate_metadata /tmp/subset.json

For each module:

1. Download the tagged release tarball from GitHub
2. Run terraform-config-inspect --json to extract structure
3. Produce JSONL with module calls, resources, and providers

curl -sL "$source/archive/refs/tags/$tag.tar.gz" | tar xz
terraform-config-inspect --json "$folder" > "$folder.json"

No git clone. No full repo. Just the tagged release, inspected, then deleted.

Output Schema

{
  "id": "schubergphilis/mcaf-cloudfront/aws",
  "published_at": "2025-06-01T...",
  "version": "4.0.1",
  "tag": "v4.0.1",
  "namespace": "schubergphilis",
  "downloads": 450,
  "modules": ["cloudposse/label/null", "..."],
  "resources": ["aws_cloudfront_distribution", "aws_waf_web_acl"],
  "namespaces": ["cloudposse", "hashicorp"],
  "providers": ["aws", "random"]
}

Every record tells you: this module uses these resources, calls these upstream modules, depends on these providers.

Questions This Answers

Who ships continuously?

jq 'group_by(.namespace)[]|{ns: first.namespace, count: length}' \
  | jq -s 'sort_by(.count)|reverse|.[0:20]'

Which resources appear together?

jq '[.resources[]]' /tmp/metadata.jsonl \
  | jq -s 'flatten|group_by(.)|map({resource: first, count: length})|sort_by(.count)|reverse'

Which modules are most reused?

jq '[.modules[]]' /tmp/metadata.jsonl \
  | jq -s 'flatten|group_by(.)|map({module: first, count: length})|sort_by(.count)|reverse'

Answer: cloudposse/label/null is in almost everything. Standard naming/tagging patterns propagate through the ecosystem.

Which providers beyond AWS?

jq '.providers[]' /tmp/metadata.jsonl | sort | uniq -c | sort -rn

random, null, local, archive — utility providers show up everywhere alongside aws.

Grafana Dashboard

Upload datasets to S3, point Grafana Infinity datasource at your CloudFront URL:

• your-domain/data/tf-aws-modules.json
• your-domain/data/tf-aws-namespaces.json

Visualize:

• Module publish frequency over time
• Namespace activity heatmap
• Download distribution (long tail)
• Resource co-occurrence patterns

The Pattern

Registry API → Paginated Crawl → Parse + Enrich → Upload → Grafana
     ↓
Source Tarballs → terraform-config-inspect → Structural Metadata → Analysis

Two layers:

1. Surface — what’s published, who publishes, when
2. Deep — what’s inside, how they’re built, what they compose

Both produce static JSON datasets. Refresh when you’re curious.

Modules Used

Tool	Purpose
curl	Registry API + tarball downloads
jq	JSON transformation at every stage
terraform-config-inspect	HCL → JSON structural extraction
upload_dataset (lib.sh)	S3 upload to datasets bucket
Grafana Infinity	Visualization

---

Part 4 of the Signals series — small systems that observe larger systems.

Previous: Signals #3: GitHub Commit History as a Queryable Dataset | Next: Signals #5: Slicing CloudFront Logs