Signals #1: Turn AWS Documentation Into a Live API

AWS publishes Lambda runtime identifiers on a documentation page. Not as an API. Not as a JSON feed. As an HTML table.

I needed that data as a consumable dataset — for dashboards, for tooling, for automation. So I built an endpoint that scrapes it, extracts the runtime identifiers, and serves them as JSON. Cached at the edge. Refreshed on demand.

Total infrastructure: one shell function, one Lambda, one CloudFront behavior.

Live endpoint: signals.cloudless.sh/runtimes

---

The Problem

You want to know what Lambda runtimes are currently available. AWS gives you this:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

An HTML page with tables. No machine-readable format. No API. If you want this data in your CI pipeline, in a monitoring dashboard, or in a Terraform module — you scrape.

The Handler

runtimes () {
    curl -sS https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html \
        | sed -n '/<table/,/<\/table>/p' \
        | sed -n '1,/<\/table>/p' \
        | grep -o '<code class="code">[a-z0-9.]*</code>' \
        | sed 's/<[^>]*>//g' \
        | jq -R | jq -sc
}

Five pipes:

1. curl — fetch the documentation page
2. sed — extract the first table (the one with current runtimes)
3. grep — pull out <code> elements containing runtime identifiers
4. sed — strip HTML tags
5. jq — wrap lines into a JSON array

Output:

["nodejs22.x","nodejs20.x","python3.13","python3.12","java21","java17","dotnet8","ruby3.4","provided.al2023","provided.al2"]

No dependencies beyond curl, sed, grep, and jq. All available in the Lambda execution environment via layers.

The Infrastructure

module "runtimes" {
  source  = "ql4b/lambda-function/aws"
  version = "1.2.0"

  context      = module.label.context
  attributes   = ["runtimes"]
  source_dir   = "../app"
  runtime      = "provided.al2023"
  handler      = "handler.runtimes"
  architecture = "arm64"
  memory_size  = 1024

  layers = [
    module.runtime.layer_arn,
    module.jq.layer_arn,
  ]
}

resource "aws_lambda_function_url" "runtimes" {
  function_name      = module.runtimes.function_name
  authorization_type = "NONE"

  cors {
    allow_origins = ["*"]
    allow_methods = ["GET"]
    allow_headers = ["*"]
    max_age       = 300
  }
}

Lambda Function URL — free, no API Gateway needed.

Edge Caching

CloudFront sits in front with a path-based behavior:

ordered_cache_behavior {
  path_pattern           = "/runtimes"
  target_origin_id       = local.origin_id
  viewer_protocol_policy = "redirect-to-https"
  allowed_methods        = ["GET", "HEAD"]
  cached_methods         = ["GET", "HEAD"]

  forwarded_values {
    query_string = true
    headers      = ["X-Forwarded-For", "User-Agent"]
    cookies { forward = "none" }
  }

  min_ttl     = 0
  default_ttl = 0
  max_ttl     = 60
}

First request hits Lambda. Subsequent requests within 60 seconds are served from the edge. AWS updates their docs maybe once a quarter — a 60-second TTL means Lambda invocations are near-zero.

The cache handles freshness.

Path Gating

Every other path returns 403 via a CloudFront Function:

function handler(event) {
  return {
    statusCode: 403,
    statusDescription: 'Forbidden',
    headers: { 'content-type': { value: 'application/json' } },
    body: { encoding: 'text', data: '{"error":"not found"}' }
  };
}

Only explicitly configured paths (like /runtimes) route to a Lambda origin. Everything else is blocked at the edge before it ever reaches your account.

The Pattern

This is the simplest instance of the signals pattern:

External Data Source → Shell Scraper → CloudFront Cache → Public JSON Endpoint

The cache acts as the refresh mechanism — stale entries trigger a Lambda invocation, fresh entries are served from the edge.

Applicable whenever you need to:

• Turn an HTML page into an API
• Serve a slowly-changing dataset globally
• Avoid building a “data pipeline” for something that changes quarterly

Cost

Component	Monthly Cost
Lambda	~$0.00 (cached, <100 invocations/day)
CloudFront	~$0.01 (edge cache hits)
Total	~$0.01

Modules Used

Module	Purpose
terraform-aws-lambda-function	Lambda + IAM + CloudWatch
terraform-aws-lambda-shell-runtime-layer	Bash execution environment

---

Part 1 of the Signals series — small systems that observe larger systems.

Next: Signals #2: Browser Events to S3 Data Lake in Bash