Signals #2: Browser Events to S3 Data Lake in Bash

TL;DR: A first-party analytics pipeline — browser fetch sends events to a bash Lambda that uses curl --aws-sigv4 to publish to SNS. Firehose delivers to partitioned S3. Athena queries it. Total cost: under $1/month for a personal site.

Every pageview on cloudless.sh flows through a pipeline that costs under $1/month and lands in a partitioned S3 data lake queryable with Athena. The stack is bash, curl, SNS, and Firehose.

This is the second signal: collecting events from your own infrastructure, enriching them server-side, and storing them in a format that doesn’t require a running service to query.

Live endpoint: POST signals.cloudless.sh/events

---

The Flow

Browser (fetch + keepalive)
    │
    ▼
CloudFront /events (POST, no cache, CORS)
    │
    ▼
Lambda (bash, arm64, 256MB)
    ├── Extract body (handle base64)
    ├── Enrich: source_ip, user_agent, country, received_at
    └── curl --aws-sigv4 → SNS Publish
            │
            ▼
        Firehose (transform + dynamic partitioning)
            │
            ▼
        S3: raw-data/site={site}/type={type}/year={y}/month={m}/day={d}/

The Client: No Library

export function sendEvent(endpoint, type = "pageview") {
  if (!endpoint) return;
  fetch(endpoint, {
    method: "POST",
    body: JSON.stringify({
      schema: "cloudless.web_event.v1",
      type,
      site: "cloudless",
      page: {
        url: location.href,
        path: location.pathname,
        title: document.title,
        referrer: document.referrer
      },
      browser: {
        language: navigator.language,
        timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
        viewport: { width: innerWidth, height: innerHeight }
      },
      event: { id: crypto.randomUUID(), timestamp: new Date().toISOString() }
    }),
    keepalive: true,
    credentials: "omit"
  });
}

keepalive: true survives page navigation. credentials: "omit" avoids cookie overhead.

<script type="module">
  import { sendEvent } from "../scripts/events.js";
  const endpoint = import.meta.env.PUBLIC_EVENTS_ENDPOINT;
  if (endpoint) sendEvent(endpoint);
</script>

Set PUBLIC_EVENTS_ENDPOINT=https://signals.cloudless.sh/events at build time. Omit it in dev — no events, no noise.

The Handler: curl as AWS SDK

events () {
    local event body source_ip user_agent country received_at
    event="$(cat)"

    # Handle base64 encoding from Function URL
    local is_base64
    is_base64="$(echo "$event" | jq -r '.isBase64Encoded // false')"
    body="$(echo "$event" | jq -r '.body // empty')"
    [[ "$is_base64" == "true" ]] && body="$(echo "$body" | base64 -d)"
    [[ -z "$body" ]] && echo '{"error":"no-body"}' && return 0

    # Server-side enrichment from CloudFront headers
    source_ip="$(echo "$event" | jq -r '.headers["x-forwarded-for"] // ""')"
    user_agent="$(echo "$event" | jq -r '.headers["user-agent"] // ""')"
    country="$(echo "$event" | jq -r '.headers["cloudfront-viewer-country"] // ""')"
    received_at="$(date -u +%Y-%m-%dT%H:%M:%S.000Z)"

    message=$(echo "$body" | jq -c \
        --arg received_at "$received_at" \
        --arg source_ip "$source_ip" \
        --arg user_agent "$user_agent" \
        --arg country "$country" \
        '. + { received_at: $received_at, source_ip: $source_ip,
               user_agent: $user_agent, country: $country }')

    # Publish to SNS using curl's native SigV4
    if [[ -n "${EVENTS_TOPIC_ARN:-}" ]]; then
        local region="${AWS_REGION:-us-east-1}"
        curl -sS \
            --aws-sigv4 "aws:amz:${region}:sns" \
            --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
            ${AWS_SESSION_TOKEN:+-H "x-amz-security-token: ${AWS_SESSION_TOKEN}"} \
            -H "Content-Type: application/x-www-form-urlencoded" \
            -d "Action=Publish&TopicArn=${EVENTS_TOPIC_ARN}&Message=$(printf '%s' "$message" | jq -sRr @uri)" \
            "https://sns.${region}.amazonaws.com/" >/dev/null
    fi

    echo '{"ok":true}'
}

Key insight: curl 7.75+ supports --aws-sigv4 natively. Lambda’s execution role credentials are injected as environment variables — curl signs the request directly.

The Pipeline: One Module Call

module "events_topic" {
  source  = "cloudposse/sns-topic/aws"
  version = "1.2.0"
  context = module.label.context
  attributes = ["events"]
}

module "pipeline" {
  source  = "ql4b/analytics-pipeline/aws"
  version = "~> 1.2"
  context = module.label.context

  data_sources = [{
    type = "sns"
    arn  = module.events_topic.sns_topic_arn
  }]

  enable_transform            = true
  transform_template          = "sns-transform.js"
  enable_dynamic_partitioning = true

  prefix = join("/", [
    "raw-data",
    "site=!{partitionKeyFromQuery:site}",
    "type=!{partitionKeyFromQuery:type}",
    "year=!{partitionKeyFromQuery:year}",
    "month=!{partitionKeyFromQuery:month}",
    "day=!{partitionKeyFromQuery:day}",
    ""
  ])

  dynamic_partitioning_keys = <<-EOT
    {site: .site, type: .type,
     year: (.received_at|split("T")|first|split("-")|.[0]),
     month: (.received_at|split("T")|first|split("-")|.[1]),
     day: (.received_at|split("T")|first|split("-")|.[2])}
  EOT
}

The analytics-pipeline module handles:

• SNS subscription to Firehose
• Transform Lambda (unwraps SNS envelope)
• Dynamic partitioning (jq expression extracts partition keys from event payload)
• S3 bucket with lifecycle policies
• IAM roles for all components

What Lands in S3

{
  "schema": "cloudless.web_event.v1",
  "type": "pageview",
  "site": "cloudless",
  "page": {
    "url": "https://cloudless.sh/log/shell-first-json-endpoints/",
    "path": "/log/shell-first-json-endpoints/",
    "title": "Shell-First Lambda Endpoints: $0.53/M Requests",
    "referrer": "https://www.google.com/"
  },
  "browser": {
    "language": "en-GB",
    "timezone": "Europe/Madrid",
    "viewport": { "width": 1512, "height": 827 }
  },
  "event": {
    "id": "0b1cb958-1290-4e33-b907-a59d2cb7cff5",
    "timestamp": "2026-06-22T04:19:24.759Z"
  },
  "received_at": "2026-06-22T04:19:25.000Z",
  "source_ip": "198.51.100.42",
  "user_agent": "Mozilla/5.0 ...",
  "country": "ES"
}

Stored at:

s3://your-signals-bucket/raw-data/site=cloudless/type=pageview/year=2026/month=06/day=22/

Querying: Athena with Partition Projection

CREATE EXTERNAL TABLE signals.web_events (
  `schema` string,
  page struct<url:string, path:string, query:string, title:string, referrer:string>,
  browser struct<language:string, timezone:string, viewport:struct<width:int, height:int>>,
  event struct<id:string, timestamp:string>,
  received_at string,
  source_ip string,
  user_agent string,
  country string
)
PARTITIONED BY (site string, type string, year string, month string, day string)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://your-signals-bucket/raw-data/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.site.type' = 'enum',
  'projection.site.values' = 'cloudless',
  'projection.type.type' = 'enum',
  'projection.type.values' = 'pageview,click,scroll',
  'projection.year.type' = 'integer',
  'projection.year.range' = '2026,2030',
  'projection.month.type' = 'integer',
  'projection.month.range' = '1,12',
  'projection.month.digits' = '2',
  'projection.day.type' = 'integer',
  'projection.day.range' = '1,31',
  'projection.day.digits' = '2'
);

Partition projection means: no MSCK REPAIR TABLE, no partition management. Athena resolves partitions mathematically.

-- Daily unique visitors this month
SELECT day, COUNT(DISTINCT source_ip) as visitors, COUNT(*) as pageviews
FROM signals.web_events
WHERE site = 'cloudless' AND type = 'pageview' AND year = '2026' AND month = '06'
GROUP BY day ORDER BY day;

-- Top pages by country
SELECT country, page.path, COUNT(*) as views
FROM signals.web_events
WHERE year = '2026' AND month = '06'
GROUP BY country, page.path
ORDER BY views DESC LIMIT 20;

CloudFront Configuration

The /events path requires special treatment — POST must pass through, no caching:

ordered_cache_behavior {
  path_pattern           = "/events"
  target_origin_id       = local.events_origin_id
  viewer_protocol_policy = "https-only"
  allowed_methods        = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
  cached_methods         = ["GET", "HEAD"]

  forwarded_values {
    query_string = false
    headers = [
      "Origin", "Access-Control-Request-Method",
      "Access-Control-Request-Headers", "User-Agent",
      "Referer", "Accept-Language", "X-Forwarded-For",
      "CloudFront-Viewer-Country"
    ]
    cookies { forward = "none" }
  }

  min_ttl     = 0
  default_ttl = 0
  max_ttl     = 0
}

Forwarding CloudFront-Viewer-Country gives us geo data without any GeoIP lookup in the handler. CloudFront resolves it at the edge for free.

Cost

For a blog doing ~10K pageviews/month:

Component	Monthly Cost
Lambda (10K invocations × 256MB × ~100ms)	~$0.01
SNS (10K publishes)	~$0.01
Firehose (10K events × ~500 bytes)	~$0.00
S3 (5MB stored)	~$0.00
Athena (queries on demand)	~$0.01/query
Total	< $0.10/month

Compare: Google Analytics ($0 but you lose data ownership), Plausible ($9/month), Fathom ($15/month).

The Pattern

Browser Event → First-Party Endpoint → Enrich → Fan-Out → Batch + Partition → Query on Demand

This generalizes beyond web analytics. Any event source that needs:

• Server-side enrichment
• Append-only storage
• Schema-on-read querying
• Query-time processing instead of a persistent datastore

The same pipeline ingests booking events, API telemetry, and automation outcomes in other parts of the system.

Modules Used

Module	Purpose
terraform-aws-lambda-function	Lambda + IAM + logs
terraform-aws-lambda-shell-runtime-layer	Bash runtime
terraform-aws-analytics-pipeline	SNS → Firehose → S3 with partitioning

---

Part 2 of the Signals series — small systems that observe larger systems.

Previous: Signals #1: Turn AWS Documentation Into a Live API | Next: Signals #3: GitHub Commit History as a Queryable Dataset