Building a JA4 Fingerprint Pipeline on AWS

Part one showed what JA4 fingerprints reveal about traffic — spoofed browsers, residential proxy clusters, the cryptographic proof that a claimed iPhone is running a Go scraper. This post covers how to actually collect that data: the full pipeline from TLS handshake to queryable S3 dataset.

The stack is entirely serverless — a CloudFront Function captures the fingerprint at the edge on every request. A bash Lambda ingests the log stream. SNS and Firehose deliver events to partitioned S3. Athena and Grafana sit on top.

HTTPS Request
    │
    ▼
CloudFront Function          ← captures ja4, ja3, ip, asn, user_agent
    │ console.log(...)
    ▼
CloudWatch Logs
    │ subscription filter
    ▼
Lambda (bash, arm64)         ← decode gzip, parse lines, publish to SNS
    │ curl + SigV4
    ▼
SNS Topic
    │
    ▼
Firehose                     ← unwrap SNS envelope, dynamic partitioning
    │
    ▼
S3 Data Lake
raw-data/site=cloudless/type=ja4/year=2026/month=07/day=22/
    │
    ▼
Glue + Athena                ← external table, ja4_events_v view
    │
    ▼
Grafana

Step 1: Capture at the Edge

CloudFront exposes JA4 and JA3 fingerprints as viewer request headers. A CloudFront Function running on every viewer request reads them and emits a structured log line to CloudWatch — zero latency impact, no Lambda cold start, no backend roundtrip.

function handler(event) {
  var req = event.request;
  var headers = req.headers;

  var ip  = (headers['x-forwarded-for']                    || {value: ''}).value;
  var ja4 = (headers['cloudfront-viewer-ja4-fingerprint']  || {value: ''}).value;
  var ja3 = (headers['cloudfront-viewer-ja3-fingerprint']  || {value: ''}).value;
  var ua  = (headers['user-agent']                         || {value: ''}).value;
  var asn = (headers['cloudfront-viewer-asn']              || {value: ''}).value;

  if (ja4) {
    console.log(ip + ' | ' + ja4 + ' | ' + ja3 + ' | ' + ua + ' | ' + asn);
  }

  return req;
}

CloudFront writes these log lines to /aws/cloudfront/function/<function-name> in CloudWatch Logs automatically. The filter pattern " | " on the subscription filter skips START/END Lambda lifecycle lines and matches only the telemetry lines.

Two things to enable in CloudFront before this works:

• Origin request policy: add CloudFront-Viewer-JA4-Fingerprint and CloudFront-Viewer-JA3-Fingerprint to the headers CloudFront forwards — these are CloudFront-generated headers and must be explicitly allowed
• CloudFront Function: associate the function with the viewer-request event on your distribution

Step 2: Ingest with a Bash Lambda

CloudWatch delivers log batches to the Lambda as a base64-encoded gzip payload. The handler decodes it, parses each pipe-delimited line, and publishes a structured JSON event to SNS via curl with SigV4 signing.

ja4_ingest () {
    local event data logs received_at region
    event="$(cat)"
    received_at="$(date -u +%Y-%m-%dT%H:%M:%S.000Z)"
    region="${AWS_REGION:-us-east-1}"

    # CloudWatch delivers: { "awslogs": { "data": "<base64(gzip(json))>" } }
    data="$(echo "$event" | jq -r '.awslogs.data')"
    logs="$(echo "$data" | base64 -d | gzip -d)"

    echo "$logs" | jq -r '.logEvents[].message' | while read -r line; do
        [[ ! "$line" =~ " | " ]] && continue

        local ip ja4 ja3 ua asn message
        local clean
        clean="$(echo "$line" | sed 's/^[^ ]* //')"
        ip="$(echo  "$clean" | awk -F' [|] ' '{print $1}')"
        ja4="$(echo "$clean" | awk -F' [|] ' '{print $2}')"
        ja3="$(echo "$clean" | awk -F' [|] ' '{print $3}')"
        ua="$(echo  "$clean" | awk -F' [|] ' '{print $4}')"
        asn="$(echo "$clean" | awk -F' [|] ' '{print $5}')"

        message=$(jq -nc \
            --arg ip          "$ip"          \
            --arg ja4         "$ja4"         \
            --arg ja3         "$ja3"         \
            --arg ua          "$ua"          \
            --arg asn         "$asn"         \
            --arg received_at "$received_at" \
            '{
                schema:      "cloudless.ja4_event.v1",
                type:        "ja4",
                site:        "cloudless",
                ip:          $ip,
                ja4:         $ja4,
                ja3:         $ja3,
                user_agent:  $ua,
                asn:         $asn,
                received_at: $received_at,
                event: { id: ("ja4-" + (now|tostring)), timestamp: $received_at }
            }')

        curl -sS \
            --aws-sigv4 "aws:amz:${region}:sns" \
            --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
            ${AWS_SESSION_TOKEN:+-H "x-amz-security-token: ${AWS_SESSION_TOKEN}"} \
            -H "Content-Type: application/x-www-form-urlencoded" \
            -d "Action=Publish&TopicArn=${JA4_TOPIC_ARN}&Message=$(printf '%s' "$message" | jq -sRr @uri)" \
            "https://sns.${region}.amazonaws.com/" >/dev/null
    done

    echo '{"ok":true}'
}

A few things worth noting:

• gzip -d requires the gzip binary, which isn’t in the provided.al2023 base image. It’s provided as a Lambda layer from lambda-shell-layers.
• curl --aws-sigv4 handles SigV4 request signing natively — the only dependency is curl itself.
• jq -sRr @uri URL-encodes the JSON message for the SNS form POST.
• The sed 's/^[^ ]* //' strips the CloudWatch log prefix (timestamp + request ID) before parsing the pipe-delimited fields.

Step 3: Deliver with Firehose

SNS fans out to a Firehose delivery stream. Firehose handles batching, SNS envelope unwrapping via a transform Lambda, and dynamic partitioning into S3. The ql4b/analytics-pipeline/aws module — covered in detail in Serverless Analytics Pipelines with Terraform — wraps all of that into a single module call:

The Terraform module call:

module "ja4_pipeline" {
  source  = "ql4b/analytics-pipeline/aws"
  version = "1.2.3"

  context    = module.label.context
  attributes = ["ja4"]

  data_sources = [{
    type = "sns"
    arn  = module.ja4_topic.sns_topic_arn
  }]

  buffering_interval = 60
  buffering_size     = 1

  enable_transform   = true
  transform_template = "sns-transform.js"

  enable_dynamic_partitioning = true

  prefix = join("/", [
    "raw-data",
    "site=!{partitionKeyFromQuery:site}",
    "type=!{partitionKeyFromQuery:type}",
    "year=!{partitionKeyFromQuery:year}",
    "month=!{partitionKeyFromQuery:month}",
    "day=!{partitionKeyFromQuery:day}",
    ""
  ])

  dynamic_partitioning_keys = "{site: .site, type: .type, year: (.received_at|split(\"T\")|first|split(\"-\")|.[0]), month: (.received_at|split(\"T\")|first|split(\"-\")|.[1]), day: (.received_at|split(\"T\")|first|split(\"-\")|.[2])}"
}

The sns-transform.js template unwraps the SNS envelope — Firehose receives {"Message": "{...}"} from SNS, the transform extracts the inner JSON and passes it downstream. Dynamic partitioning extracts site, type, year, month, day from the event body using jq-style expressions, producing Hive-style S3 paths:

s3://bucket/raw-data/site=cloudless/type=ja4/year=2026/month=07/day=22/

Step 4: Wire the CloudWatch Subscription

The CloudWatch Logs subscription filter connects the log group to the Lambda. The filter pattern " | " matches only the pipe-delimited telemetry lines — START/END lifecycle messages don’t contain | and are skipped automatically.

resource "aws_cloudwatch_log_subscription_filter" "ja4" {
  name            = "${module.label.id}-ja4"
  log_group_name  = "/aws/cloudfront/function/your-function-name"
  filter_pattern  = "\" | \""
  destination_arn = module.ja4_ingest.function_arn

  depends_on = [aws_lambda_permission.ja4_cwlogs]
}

resource "aws_lambda_permission" "ja4_cwlogs" {
  statement_id  = "AllowCloudWatchLogs"
  action        = "lambda:InvokeFunction"
  function_name = module.ja4_ingest.function_name
  principal     = "logs.amazonaws.com"
  source_arn    = "arn:aws:logs:${var.region}:${local.account_id}:log-group:/aws/cloudfront/function/your-function-name:*"
}

One constraint: CloudWatch log subscriptions must be in the same region as the log group. CloudFront Functions log to us-east-1 regardless of your distribution’s origin region. The Lambda and subscription filter need to be deployed in us-east-1.

Step 5: Query with Athena

A Glue external table points at the S3 prefix with partition projection enabled — Athena resolves partitions automatically from the path pattern.

resource "aws_glue_catalog_table" "ja4_events" {
  database_name = "signals"
  name          = "ja4_events"
  table_type    = "EXTERNAL_TABLE"

  parameters = {
    "classification"     = "json"
    "projection.enabled" = "true"

    "projection.year.type"  = "integer"
    "projection.year.range" = "2026,2030"

    "projection.month.type"   = "integer"
    "projection.month.range"  = "1,12"
    "projection.month.digits" = "2"

    "projection.day.type"   = "integer"
    "projection.day.range"  = "1,31"
    "projection.day.digits" = "2"

    "storage.location.template" = "s3://${bucket}/raw-data/site=$${site}/type=$${type}/year=$${year}/month=$${month}/day=$${day}/"
  }

  # partition_keys: site, type, year (int), month (int), day (int)
  # storage_descriptor columns: schema, ip, ja4, ja3, user_agent, asn, received_at, event
}

The Athena view ja4_events_v sits on top of the raw table and adds all the derived columns: parsed JA4 fields, client_predicted_type, asn_holder, asn_network_tier, access_type, crawler. It’s provisioned as a named query in Terraform and run once after deploy:

CREATE OR REPLACE VIEW ja4_events_v AS
SELECT
  cast(from_iso8601_timestamp(received_at) AT TIME ZONE 'UTC' AS timestamp) AS ts,
  ip,
  ja4,
  ja3,
  split_part(ja4, '_', 1)             AS ja4_prefix,
  substr(split_part(ja4, '_', 1), 5, 2) AS ja4_cipher_count,
  substr(split_part(ja4, '_', 1), 7, 2) AS ja4_ext_count,
  substr(split_part(ja4, '_', 1), 4, 1)  AS ja4_sni,
  substr(split_part(ja4, '_', 1), 9, 2)  AS ja4_alpn,
  CASE
    WHEN ja4 = 't13d131000_f57a46bbacb6_e7c285222651' THEN 'Spoofed Browser Cluster'
    WHEN ja4 = 't13d1011h2_61a7ad8aa9b6_3fcd1a44f3e3' THEN 'AI Crawler (Claude/GPT/OAI)'
    WHEN ja4 = 't13d181300_e8a523a41297_43ade6aba3df' THEN 'Googlebot'
    WHEN split_part(ja4, '_', 1) = 't13d1516h2'      THEN 'Desktop Chromium'
    WHEN split_part(ja4, '_', 1) = 't13d1517h2'      THEN 'Desktop Chromium'
    WHEN split_part(ja4, '_', 1) = 't13d2013h2'      THEN 'Chrome on iOS (CriOS)'
    WHEN split_part(ja4, '_', 1) = 't13d1617h2'      THEN 'Firefox'
    WHEN split_part(ja4, '_', 1) = 't13d1912h2'      THEN 'Safari / WebView'
    ELSE 'Other / Custom Stack'
  END AS client_predicted_type,
  user_agent,
  asn,
  CASE asn
    WHEN '132203' THEN 'Tencent Cloud'
    WHEN '141679' THEN 'China Telecom Big Data'
    WHEN '15169'  THEN 'Google'
    WHEN '16509'  THEN 'Amazon'
    -- ... full mapping in ja4.tf
    ELSE NULL
  END AS asn_holder,
  CASE
    WHEN asn IN ('132203', '141679', '9009', '45090') THEN 'Data Center / Hosting'
    WHEN asn IN ('15169', '396982')                   THEN 'Google'
    WHEN asn IN ('16509')                             THEN 'Amazon'
    WHEN asn IN ('3352', '7552', '27699', '28573')    THEN 'Major National Telecom'
    ELSE 'Other Regional ISP / Unknown Pool'
  END AS asn_network_tier,
  received_at,
  site, year, month, day
FROM ja4_events

With this view in place, the detection queries from part one become straightforward:

-- Spoofed browser clusters: claims to be Chrome/Safari but no ALPN
SELECT ja4, asn, asn_network_tier, count(*) AS hits
FROM ja4_events_v
WHERE substr(ja4_prefix, 9, 2) = '00'
  AND (user_agent LIKE '%Chrome%' OR user_agent LIKE '%Safari%')
  AND year = 2026
GROUP BY ja4, asn, asn_network_tier
ORDER BY hits DESC;

-- Same fingerprint across multiple countries
SELECT ja4_prefix, country, asn, count(*) AS hits
FROM ja4_events_v
JOIN cloudfront_requests r ON r.ts BETWEEN ...
WHERE ja4_prefix = 't13d4412h1'
GROUP BY ja4_prefix, country, asn
ORDER BY hits DESC;

The Lambda Layers

The bash Lambda needs three layers:

Layer	Purpose
lambda-shell-runtime	Custom bootstrap that sources handler.sh and routes invocations to named bash functions
jq	JSON processing
gzip	Decompress CloudWatch log payloads

All three are pre-built and published as versioned releases in lambda-shell-layers. The Terraform module references them by URL:

module "gzip" {
  source  = "ql4b/lambda-layer/aws"
  version = "1.2.0"

  source_url               = "https://github.com/ql4b/lambda-shell-layers/releases/download/v0.1.0/gzip-arm64-layer.zip"
  compatible_architectures = ["arm64"]
}

Cost

At blog traffic volumes this pipeline costs effectively nothing:

Component	Cost
CloudFront Function	$0.10 per million invocations
CloudWatch Logs ingestion	$0.50 per GB
Lambda (256MB, arm64)	~$0.00 at low invocation rates
SNS	$0.50 per million publishes
Firehose	$0.029 per GB delivered
S3	$0.023 per GB/month
Athena	$5 per TB scanned

The CloudWatch subscription filter batches log lines before invoking Lambda, so one Lambda invocation processes many fingerprints. Firehose buffers for 60 seconds before writing to S3, keeping object counts manageable.

Deploy

cd infra
terraform init
terraform apply

After apply, run the create_ja4_events_view named query once in the Athena console to materialise the view. From that point, every HTTPS request to your CloudFront distribution produces a fingerprint event in S3 within ~60 seconds.

---