Skip to content
cloudless
by skunxicat
signals cloudfront athena grafana web-traffic

Signals #6: Build a Visibility Layer

How can I observe what actually reaches my content without relying on JavaScript analytics?

JavaScript analytics are the default answer. Drop a script tag, get a dashboard. But they have a fundamental blind spot: they only see what executes JavaScript. Crawlers don’t. LLM retrieval agents don’t. Search indexers don’t.

For a content site in 2026, that blind spot is exactly where the interesting traffic lives.

The Problem with JavaScript Analytics

A pageview event fires when a human browser loads a page and runs your script. That’s useful. But it tells you nothing about:

  • Which pages Googlebot is indexing and how often
  • Whether ChatGPT-User is fetching your content when users ask questions
  • How ClaudeBot is crawling your site
  • What security scanners are probing
  • Which pages get link-preview fetches from Slack or WhatsApp

These aren’t edge cases. On cloudless.sh, non-human traffic is the majority. If you’re only measuring JavaScript events, you’re measuring a minority of what’s actually happening.

The Alternative: CloudFront Logs

Every request that hits CloudFront generates a log entry. No JavaScript required. No sampling. No consent banners. Every bot, every crawler, every human — all of it.

CloudFront Standard Logging v2 delivers JSON records to S3 with Hive-style partitioning:

s3://logs-bucket/
  AWSLogs/aws-account-id=.../
    CloudFront/
      DistributionId=EDIST1EXAMPLE1/
        year=2026/month=07/day=02/
          *.json.gz

Each record has: timestamp, path, status, method, user-agent, referrer, country, edge location, cache result, bytes, timing.

The raw data is complete. The challenge is making it queryable and meaningful.

Two Layers of Meaning

Raw log data answers what happened. To answer who did it and why, you need two derived dimensions:

agent — who made the request. Not the raw user-agent string (which is noisy and URL-encoded), but a normalized identity: openai, anthropic, google, perplexity, semrush, scanner, other. Derived from user-agent pattern matching.

access_type — why they accessed. Four categories:

  • retrieval — AI acting on behalf of a user (ChatGPT-User, Claude-User, Perplexity-User). Someone asked a question that referenced your content.
  • crawler — autonomous indexing (GPTBot, ClaudeBot, Googlebot). Building a corpus.
  • preview — social link unfurl (Slackbot, WhatsApp, facebookexternalhit). Someone shared your URL.
  • scan — security/infrastructure probing. Background noise.

The distinction between retrieval and crawler is the most valuable one. A retrieval event means a human is actively consuming your content through an AI interface — right now. A crawler event means a bot is building a future index. Different signals, different implications.

crawler — the specific bot identity, where known. While agent normalizes to a vendor (openai, anthropic, semrush), crawler preserves the exact token: GPTBot, ClaudeBot, AhrefsBot, AhrefsSiteAudit, Amazonbot, Xpanse, Scrapy, etc. Useful when you want to distinguish between, say, OpenAI’s indexing crawler and its real-time retrieval agent, or between Ahrefs’ link crawler and its site audit tool. NULL for human traffic.

The Architecture

CloudFront → S3 (JSON logs, partitioned)
                │
                ▼
         Glue external table
         (schema-on-read, partition projection)
                │
                ▼
         Athena view: cloudfront_requests
         (adds: agent, access_type, site, normalized ts)
                │
                ▼
            Grafana

Three components:

Glue table — defines the schema over the raw S3 data. Partition projection means no manual partition registration — Athena infers partitions from the path structure. Query any date range without maintenance.

Athena view — the semantic layer. Adds agent and access_type via CASE expressions over the user-agent field. Normalizes the timestamp. Maps distribution IDs to site names. This is where raw logs become signals.

Grafana — visualization. Athena datasource queries the view directly. No intermediate storage, no ETL pipeline, no scheduled jobs. Query on demand.

Why Not a Pipeline?

The tempting approach is to build a streaming pipeline: CloudFront → Kinesis → Lambda → enriched events → database. More infrastructure, more moving parts, more cost.

The simpler approach: logs land in S3, Athena queries them in place. The “enrichment” (agent classification, access_type) lives in the view definition — a SQL CASE expression. Change the classification logic by updating the view. No reprocessing, no backfill jobs.

Schema-on-read means the raw data is always there, unmodified. The interpretation layer is separate and cheap to evolve.

What the Dashboard Shows

Once the view exists, the questions become straightforward SQL:

  • Requests per day by agent — who’s visiting over time
  • access_type breakdown — crawler vs retrieval vs human vs scan
  • Top paths by agent — which content attracts which actors
  • retrieval events over time — LLM consumption trend
  • Cache hit rate by path — CDN efficiency
  • Country distribution — geographic reach

The dashboard that emerges isn’t a vanity metrics board. It’s a map of how your content propagates through the modern discovery ecosystem — search engines, AI systems, social previews, and humans.

Part 6 of the Signals series — small systems that observe larger systems.

Previous: Signals #5: Slicing CloudFront Logs | Next: Signals #7: From Logs to Signals