Skip to content
cloudless
by skunxicat
signals serverless bot-detection ja4 enrichment python athena

Building a Trust Layer: Ingestion-Time Enrichment for Web Traffic

enrichment pipeline

In the first article we showed how JA4 fingerprints combined with ASN classification expose spoofed browser clusters. In the second we built the collection pipeline — a CloudFront Function logging fingerprints at the edge, funneled into S3 through Firehose.

Those articles end with a dashboard. The data flows into Grafana — CloudFront logs joined with JA4 fingerprints, bot classification from user-agent CASE expressions. You can see the traffic. But the join is expensive (two full table scans matched by request ID on every dashboard refresh), and the classification can’t be trusted. A user-agent that says “Googlebot” gets labeled as Googlebot regardless of whether the IP actually belongs to Google. The label is there but the confidence is missing.

This article makes that classification trustworthy. Every CloudFront log record arrives in Grafana already enriched — with a JA4 fingerprint, an ASN tier, a bot verification result, and a composite trust score. No joins at query time. No uncertainty about whether a declared bot is real or spoofed.

The Problem With Query-Time Joins

The visibility layer works. CASE expressions in an Athena view classify requests by user-agent into access_type and crawler columns. It’s cheap to change — edit SQL, re-run the view, done. No deploy, no rollback.

But it has limits:

It can’t validate. A user-agent that says “Googlebot” might be Googlebot or might be a scraper from Tencent Cloud. The CASE expression sees the same string either way. Validation requires checking the source IP against published CIDR ranges — logic that doesn’t belong in a SQL view.

It can’t correlate across streams. The JA4 fingerprint lives in a separate table, captured by a different pipeline. Joining them in Athena requires scanning both tables, matching by request_id, for every query. At low volumes the cost is tolerable. At dashboard-refresh cadence it’s wasteful.

It can’t enrich with external data. ASN tier classification, known-bot IP ranges, fingerprint anomaly detection — these require lookups against reference datasets. SQL views can’t call APIs or read JSON files.

The signals series principle is “make the classification cheap to change” — CASE expressions over Lambda enrichment. That principle holds for the interpretation layer (what you call things). But validation (whether to trust what things claim to be) is a different problem. It requires data the SQL view doesn’t have access to.

The Architecture

When a new CloudFront log file lands in S3, an EventBridge rule fires. The event flows through SQS to a Python Lambda that reads the log batch, joins it with JA4 fingerprint data, applies enrichment, and publishes the result to a new analytics pipeline.

CloudFront Log → S3 → EventBridge → SQS → Enrichment Lambda → SNS → Firehose → S3
                                                  ↑
                              JA4 data (S3) ──────┘
                              Bot CIDRs (S3) ─────┘
                              ASN tiers (S3) ─────┘

The enriched records land in their own partitioned S3 prefix, queryable through a Glue table and Athena view — the same pattern as every other pipeline in the system.

Why No Artificial Delay

Both data streams are buffered before landing in S3. CloudFront logs take 1-5 minutes. JA4 events pass through Firehose with a 60-second buffer. By the time the CF log file triggers the EventBridge rule, the corresponding JA4 data for those requests has already been flushed.

The original design proposed an SQS delay to handle a “race condition” between the streams. In practice, the race doesn’t exist — both paths converge through buffered delivery. SQS is there for retry and dead-letter handling, not timing.

If the JA4 data genuinely hasn’t landed yet (a rare Firehose flush misalignment), the Lambda detects a high miss rate (>20% of records without a JA4 match) and raises an exception. SQS retries after the visibility timeout. By then, the data has arrived.

The Join

The correlation key is CloudFront’s x-edge-request-id — present in both the CF log (natively) and the JA4 stream (captured by the edge function from event.request.requestId). No synthetic IDs, no timestamp fuzzy-matching.

The Lambda reads the JA4 Firehose output files for the same day partition, builds an in-memory dictionary keyed by request_id, and does O(1) lookups for each CF log record. At current traffic volumes (~500 JA4 records per day partition), this index fits comfortably in memory.

No DynamoDB. No Redis. Both datasets are already in S3; a point-lookup store would add cost and operational burden for no benefit at this scale.

The Enrichment

Each CF log record gets four things added:

1. Bot CIDR Validation

If the user-agent identifies a known bot (Googlebot, bingbot, GPTBot, etc.), the Lambda checks the source IP against that bot’s published IP ranges.

Google publishes their crawler IPs at a JSON endpoint. Bing does the same. OpenAI provides separate JSON endpoints for each of their bots (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot). A daily scheduled task fetches these lists and stores them in the datasets bucket. The Lambda loads them on cold start.

The result is a verified field:

  • true — IP matches the published range for the declared bot
  • false — IP does not match (spoofed)
  • null — bot uses PTR-based verification (Anthropic, Meta, Amazon) which can’t be done in batch
from ipaddress import ip_address, ip_network

def verify_bot_ip(ip, bot_key, ref_data):
    bot_info = ref_data["bot_cidrs"].get(bot_key)
    if bot_info["verification"] == "ptr":
        return None  # can't verify in batch
    
    addr = ip_address(ip)
    for network in bot_info["networks"]:
        if addr in network:
            return True
    return False

2. ASN Classification

The ASN number arrives from the JA4 stream — the edge function captures it from the cloudfront-viewer-asn request header. No IP-to-ASN database needed; CloudFront resolves it at the edge.

The Lambda maps the ASN number to a tier using a curated JSON file:

{
  "15169": { "name": "Google", "tier": "cloud" },
  "3352":  { "name": "Telefonica de Espana", "tier": "residential" },
  "16276": { "name": "OVH", "tier": "hosting" },
  "33771": { "name": "Safaricom", "tier": "mobile" }
}

About 150 entries, covering the ASNs that actually appear in the traffic. Unknown ASNs get tier: "unknown" and are logged for periodic classification.

The tier tells you where the request originates without checking the user-agent. A “browser” from a cloud ASN is suspicious regardless of what it claims.

3. JA4 Anomaly Detection

With the fingerprint joined to the request, the Lambda can detect inconsistencies:

Protocol mismatch: The user-agent claims to be Chrome/Safari/Firefox, but the JA4 ALPN field shows HTTP/1.1. Every real modern browser negotiates HTTP/2 or HTTP/3. A browser UA with an h1 handshake is a library with a spoofed user-agent string.

Low extension count: Real browsers offer 15-20+ TLS extensions. Minimal TLS stacks (Go’s net/http, Python’s requests) offer 3-5. A browser UA with fewer than 5 extensions is a strong signal of automation.

def detect_anomalies(ja4, user_agent, access_type):
    anomalies = []
    prefix = ja4.split("_")[0]
    ja4_alpn = prefix[8:10]
    ja4_ext_count = prefix[6:8]

    if access_type == "browser" and ja4_alpn not in ("h2", "h3"):
        anomalies.append("protocol_mismatch")

    if access_type == "browser" and int(ja4_ext_count) < 5:
        anomalies.append("low_extension_count")

    return anomalies

4. Trust Score

The four signals combine into a single trust classification:

ValueCondition
verifiedIP matches published CIDR for declared bot
spoofedClaims known bot, IP doesn’t match
legitimateBrowser UA + residential/mobile ASN + no anomalies
suspiciousAny anomaly, or browser UA from cloud/hosting ASN
unverifiedKnown bot, can’t verify (PTR-based or no published ranges)
neutralNot enough signals to classify

The ordering matters — first match wins. A verified Googlebot is verified even if it comes from a cloud ASN. A spoofed Googlebot is spoofed even if it has a perfect fingerprint.

Why Python

The signals series is shell-first. Every Lambda in the system runs bash on provided.al2023. This one is Python 3.12. The deviation is deliberate.

CIDR validation requires ipaddress.ip_network() and membership testing across hundreds of ranges. The existing bash bot-verification script already shells out to Python for this. Batch gzip decompression, JSON parsing at volume, and in-memory dictionary indexing are all operations where Python is natural and bash becomes a liability.

The enrichment Lambda is the only Python function in the stack. Everything else — collection, ingestion, dataset publishing — stays bash. The boundary is clean: shell handles the simple, composable, pipe-friendly operations; Python handles the batch processing that requires data structures and a standard library.

The Output

An enriched record looks like this:

{
  "schema": "cloudless.enriched_request.v1",
  "request_id": "MGKFAfzd_FVrVu80PvKKOWYsqM4sUPOUYtzu5CcdH8qg6u9-LBUG_w==",
  "timestamp": "2026-08-13T19:13:33Z",
  "client_ip": "203.0.113.42",
  "path": "/robots.txt",
  "status": 200,
  "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...",
  "country": "DE",

  "ja4": "t13d191100_9dc949149365_be53661681a4",
  "ja4_match": true,

  "asn": 12816,
  "asn_name": "Leibniz-Rechenzentrum",
  "asn_tier": "academic",

  "access_type": "browser",
  "crawler": null,
  "verified": null,
  "trust": "suspicious",
  "anomalies": ["protocol_mismatch"],

  "site": "cloudless",
  "enriched_at": "2026-08-13T19:14:02.000Z"
}

A browser user-agent requesting /robots.txt from an academic research network. The JA4 fingerprint shows 19 ciphers, 11 extensions, and HTTP/1.1 — the protocol_mismatch anomaly fires. Trust: suspicious. It’s almost certainly a research crawler with a browser UA, not a human with Firefox.

The record flows through the same SNS → Firehose → S3 pattern used by every other pipeline. Dynamic partitioning places it at enriched/site=cloudless/type=enriched/year=2026/month=08/day=13/. A Glue table with partition projection makes it queryable. An Athena view provides the ts timestamp column. Grafana consumes it.

The Infrastructure

The full pipeline is one Terraform file. It instantiates the same modules used everywhere else:

module "enriched_pipeline" {
  source  = "ql4b/analytics-pipeline/aws"
  version = "1.2.3"

  context    = module.label.context
  attributes = ["enriched"]

  data_sources = [{
    type = "sns"
    arn  = module.enriched_topic.sns_topic_arn
  }]

  enable_dynamic_partitioning = true
  prefix = "enriched/site=.../type=.../year=.../month=.../day=.../"
}

One EventBridge rule. One SQS queue with a dead-letter queue. One Lambda. One analytics pipeline module instance. The infrastructure cost is under $1/month at current volumes.

What It Reveals

CloudFront’s built-in Viewers report (Reports & analytics > Viewers) already tells you the headline number: 58% of traffic to cloudless.sh is Bot/Crawler. That’s useful — but it’s a pie chart. You can’t drill into it, filter by path, or verify individual requests.

The visibility layer (the Athena CASE view, built in article #7) breaks the same traffic down by intent:

  • llm_crawler: 23.8%
  • browser: 15.8%
  • retrieval: 15.1%
  • empty: 13.9%
  • seo_crawler: 11.2%
  • unknown: 9.6%
  • seo_tool: 8.4%
  • scan: 1.8%

The numbers align with CloudFront’s aggregate — roughly 60% non-human (llm_crawler + seo_crawler + seo_tool + retrieval + scan). But now you know why they’re here: 24% are training LLMs, 15% are answering user questions, 11% are indexing for search. Same data, more resolution.

The enrichment layer adds a third dimension: trust. The first day of enriched data confirmed patterns, now continuous and automatic:

  • Googlebot from Google ASN: verified. Crawling sitemaps and tag pages.
  • meta-externalagent from Facebook/Meta ASN: unverified (PTR-based). Hitting article pages.
  • AhrefsBot from OVH: unverified (no published CIDRs). Crawling sitemaps.
  • Browser UA from Amazon AWS: suspicious. Cloud ASN + browser claim = automation.
  • Browser UA from Tencent Cloud, China: suspicious. Cloud ASN, protocol_mismatch anomaly.
  • Browser from Telefonica, Spain: legitimate. Residential ISP, no anomalies.

Each layer adds resolution. CloudFront says “58% bots.” The visibility layer says “23% LLM crawlers, 15% retrieval, 11% SEO indexing.” The enrichment layer says “this Googlebot is real, this browser is a scraper from Tencent Cloud, this meta-externalagent can’t be verified.”

And crucially — a portion of what initially falls into the “browser” bucket (15.8%) will shift toward bot territory once scored. A browser user-agent from a cloud ASN with a protocol mismatch anomaly isn’t a human reading your article. It’s automation wearing a disguise. The trust layer makes that visible without waiting for someone to manually investigate.

The trust field becomes a filter in Grafana. Show me only legitimate traffic for real audience analysis. Show me only suspicious for threat investigation. Show me spoofed for active impersonation attempts.

The Series Principle, Revisited

The signals series says: make the classification cheap to change. CASE expressions over Lambda enrichment.

That principle still holds for the semantic layer — what you call a seo_crawler vs an llm_crawler. Those labels change as the ecosystem evolves. New bots appear monthly. The Athena view’s CASE expression remains the right place for that logic.

But trust isn’t a label. It’s a validation result based on external evidence. “Is this IP really Googlebot?” isn’t a question of interpretation — it’s a question of fact, and the fact requires checking a CIDR list. That check belongs at ingestion time, not query time.

The enrichment layer doesn’t replace the visibility layer. It feeds into it. The enriched_requests view has the same access_type and crawler columns as cloudfront_requests. It adds trust, verified, anomalies, and asn_tier. The raw view remains as a fallback, a validation tool, and a simpler entry point.

Two layers. One observes and names. The other validates and scores. Both queryable from the same Grafana dashboard.

Limitations and Honest Assessment

This trust score is a deliberate oversimplification. It’s a broad classification of traffic that, without a commercial WAF (AWS WAF Bot Control, Cloudflare Bot Management), would otherwise be invisible — but it does not pretend to be a serious bot detection system.

What it misses:

A request with a browser-like TLS fingerprint originating from behind a cheap residential proxy will be classified as legitimate. The JA4 handshake looks real. The ASN is residential. There are no anomalies to detect. Our system has no signal to distinguish this from a genuine visitor.

JA4 fingerprints themselves can be spoofed. Modern bot frameworks (Playwright, undetected-chromedriver) use real browser TLS stacks. The fingerprint is identical to a genuine browser because it is a genuine browser — just driven by automation. A serious classifier would also consider JA4H (HTTP/2 fingerprinting), JA4T (TCP fingerprinting), JA4S (server response), and other variants in the JA4+ family.

What a real solution requires:

A single request’s metadata is not enough to reach a verdict. Real bot detection relies on:

  • Traffic pattern analysis — request frequency, temporal distribution, path traversal patterns across sessions
  • Historical fingerprint behavior — how many distinct IPs share the same JA4? How many pages does a given fingerprint visit per hour?
  • Behavioral signals — mouse movement, scroll depth, time-on-page, JavaScript execution patterns (none of which exist in server-side logs)
  • Collective intelligence — threat feeds, IP reputation databases updated in real-time from millions of sites

This pipeline provides none of that. It provides per-request static classification from the metadata available at the edge. That’s enough to separate verified Googlebot from a Tencent Cloud scraper, but not enough to catch a sophisticated residential proxy operation.

Cloudflare’s Web Integrity & Trust team makes this distinction explicit: they separate Risk (how likely a single action is to be harmful — ephemeral, point-in-time) from Trust (built up over time, based on reputation and continuous behavioral analysis). Our trust score is really a Risk assessment dressed up — a point-in-time judgment from one request’s metadata. Real Trust requires watching behavior across an entire session, which is exactly what their client-side systems do and what server-side log enrichment fundamentally cannot.

CloudFront’s own analytics:

CloudFront provides built-in traffic reports (Reports & analytics > Viewers) that already classify requests into Bot/Crawler, Desktop, Mobile, and Empty categories. On this site, CloudFront reports 38% Bot/Crawler traffic — a useful high-level signal. But these are aggregate pie charts. The classification doesn’t appear in the access logs themselves, so you can’t filter individual requests by bot status, correlate with paths or timestamps, or answer “is this specific Googlebot request real or spoofed?” The enrichment layer fills that gap: per-request classification with validation evidence, queryable from the same logs.

The real question isn’t “human or bot” anymore:

The binary human/bot distinction is increasingly irrelevant. A real human’s intent is now routinely routed through a client that isn’t a browser — ChatGPT-User fetches pages on behalf of a person asking a question. A browser extension summarizes content through an API. A mobile app renders web content in a WebView. Cloudflare calls this “hybrid” traffic — sessions that shift from human to agentic and back, like a user browsing a store then handing off checkout to an automated shopping assistant.

What matters is intent: why is this request being made? Is it indexing for search visibility? Training a model? Answering a user’s question? Scraping for competitive intelligence? Probing for vulnerabilities?

Our access_type classification (retrieval, crawler, preview, scan, tool) is an attempt to answer intent rather than machinery. But it’s still derived from user-agent strings — a declaration of intent, not proof of it. The gap between declared intent and actual intent is where real traffic analysis begins.

This system is a starting point. It makes the invisible visible. What you do with that visibility — whether you block, rate-limit, or simply observe — is a separate decision that requires more context than any single request can provide.

Related: Beyond the User-Agent: Catching Spoofed Bots with JA4 and ASN — the investigation that motivated this pipeline.

Related: Building a JA4 Fingerprint Pipeline on AWS — the collection infrastructure this enrichment layer consumes.

Related: From Logs to Signals — the Athena visibility layer that this enrichment layer extends.