cloudless
by skunxicat
terraform aws api-gateway sns serverless event-driven analytics

From Analytics Pipeline to Event Ingestion

After building our serverless analytics pipeline, we had a powerful system for processing and analyzing data. SNS → SQS → Firehose → S3 → OpenSearch gave us everything we needed to handle high-volume event streams with automatic scaling and cost efficiency.

But then came a seemingly simple request that exposed a gap in our toolkit.

The Browser Automation Challenge

We needed to analyze user behavior patterns on a website by programmatically browsing it and collecting interaction data. The plan was straightforward:

  1. Browser automation script collects events (clicks, page views, form submissions)
  2. Analytics pipeline processes and stores the data
  3. OpenSearch dashboards reveal user behavior insights

We had #2 and #3 covered perfectly. But #1 presented an unexpected problem.

The “Simple” Data Ingestion Problem

Our browser script needed to send events to our analytics pipeline. Since we’d built everything around SNS as the entry point, we needed a way to publish messages to our SNS topic.

Option 1: AWS SDK directly in the browser script

// This felt wrong immediately
const sns = new AWS.SNS();
await sns.publish({
  TopicArn: 'arn:aws:sns:...',
  Message: JSON.stringify(eventData)
}).promise();

Problems:

  • AWS credentials in browser automation scripts
  • Direct SNS access from client-side code
  • No rate limiting or authentication

Option 2: Build a custom Lambda function

Browser → API Gateway → Lambda → SNS → Analytics Pipeline

This would work, but felt like overkill for such a simple use case. We’d need:

  • Lambda function to receive HTTP requests
  • JSON parsing and validation
  • SNS publishing logic
  • Error handling and logging
  • API Gateway configuration
  • IAM roles and permissions

For what? To take an HTTP POST and forward it to SNS.

The “Aha” Moment

Then we remembered: API Gateway can integrate directly with SNS. No Lambda required.

Browser Script → API Gateway → SNS → Analytics Pipeline

This was perfect! But we needed:

  • API key authentication (can’t have open endpoints)
  • Rate limiting (protect against abuse)
  • CORS headers (for browser requests)
  • Multi-stage deployment (staging/prod)
  • Proper error handling

Building this manually every time would be tedious. We needed a reusable Terraform module.

Enter terraform-aws-rest-api-sns-proxy

The module we built solves this exact problem:

module "browser_events" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  context = module.label.context
  
  sns = {
    topic_name = "browser-events"
    topic_arn  = aws_sns_topic.analytics.arn
  }
  
  # Production-ready features
  create_usage_plan = true
  api_key_required  = true
  throttle_rate_limit = 100
  quota_limit = 10000
}

Now our browser script could simply:

// Clean, secure, simple
fetch('https://api.example.com/capture', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-API-Key': process.env.API_KEY
  },
  body: JSON.stringify({
    event: 'page_view',
    url: window.location.href,
    timestamp: new Date().toISOString(),
    userAgent: navigator.userAgent
  })
});

The Results

The browser automation project that started this whole journey worked beautifully:

  • 5-minute setup: Deploy the module, get an API endpoint and key
  • Zero maintenance: No Lambda functions to monitor or debug
  • Perfect integration: Events flowed seamlessly into our existing analytics pipeline
  • Production ready: Rate limiting and authentication built-in

But more importantly, we now had a reusable tool for any event ingestion scenario.

Production-Ready Features

Conditional API Key Authentication

module "event_ingestion" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  # Enable authentication
  create_usage_plan = true
  api_key_required  = true
  
  # Rate limiting
  throttle_rate_limit  = 1000  # requests per second
  throttle_burst_limit = 2000  # burst capacity
  quota_limit          = 100000 # daily quota
}

Multi-Stage Deployment

module "analytics_api" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  stages = ["staging", "prod"]
  
  sns = {
    topic_name = "analytics-events"
    topic_arn  = aws_sns_topic.events.arn
  }
}

CORS Support for Web Clients

Built-in CORS headers enable direct browser integration:

// Frontend JavaScript
fetch('https://api.example.com/capture', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-API-Key': 'your-api-key'
  },
  body: JSON.stringify({
    event: 'user_signup',
    userId: '123',
    timestamp: new Date().toISOString()
  })
});

Beyond Browser Automation

Once we had this module, we started seeing opportunities everywhere:

Mobile App Analytics

# Collect user events from mobile apps
module "mobile_events" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  sns = {
    topic_arn = aws_sns_topic.mobile_analytics.arn
  }
  
  # High-volume mobile traffic
  throttle_rate_limit = 5000
  quota_limit = 1000000
}

Webhook Endpoints

# Third-party services sending us data
module "webhook_ingestion" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  sns = {
    topic_arn = aws_sns_topic.external_events.arn
  }
  
  # Secure webhook endpoint
  api_key_required = true
}

IoT Device Data

# Sensors and devices reporting metrics
module "iot_telemetry" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  sns = {
    topic_arn = aws_sns_topic.device_metrics.arn
  }
  
  # Device authentication required
  create_usage_plan = true
  api_key_required = true
}

The Complete Picture

Now our analytics infrastructure had both ends covered:

# Data ingestion (the missing piece)
module "event_ingestion" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  sns = {
    topic_arn = aws_sns_topic.events.arn
  }
  
  create_usage_plan = true
  api_key_required = true
}

# Analytics pipeline (from our previous work)
resource "aws_sns_topic_subscription" "analytics" {
  topic_arn = aws_sns_topic.events.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.analytics.arn
}

module "analytics_pipeline" {
  source = "github.com/ql4b/terraform-aws-firehose-analytics"
  
  # SNS → SQS → Firehose → S3 → OpenSearch
  data_sources = [
    { type = "sqs", arn = aws_sqs_queue.analytics.arn }
  ]
}

The browser automation script that started this journey? It now sends data through a production-grade API that automatically scales, has built-in security, and feeds into a comprehensive analytics system.

CloudPosse Integration

Built with CloudPosse patterns for consistent labeling:

module "label" {
  source = "cloudposse/label/null"
  
  namespace = "acme"
  name      = "analytics"
  stage     = "prod"
}

module "event_ingestion" {
  source = "github.com/ql4b/terraform-aws-rest-api-sns-proxy"
  
  context = module.label.context
  
  # Automatically tagged with namespace-name-stage
}

Operational Simplicity

Monitoring:

  • Built-in CloudWatch metrics for API Gateway
  • SNS delivery metrics
  • No Lambda function logs to manage

Debugging:

  • API Gateway access logs
  • SNS message attributes
  • Clear error responses

Scaling:

  • API Gateway handles scaling automatically
  • SNS provides unlimited fan-out
  • No Lambda concurrency limits

The Pattern Recognition

This experience taught us something important: the gap between “simple” and “production-ready” is often where the most valuable tools live.

Sending data to SNS is simple. But doing it securely, reliably, and at scale requires:

  • Authentication and authorization
  • Rate limiting and abuse protection
  • Error handling and monitoring
  • Multi-environment deployment
  • CORS support for web clients

The terraform-aws-rest-api-sns-proxy module bridges that gap. It takes a simple need (“I want to POST data to SNS”) and makes it production-ready with a single Terraform module.

When Simple Isn’t Simple Enough

This module is perfect when you need:

  • Event ingestion from any HTTP client
  • Webhook endpoints that fan out to multiple services
  • Analytics data collection from web/mobile apps
  • IoT device telemetry with authentication
  • Microservices communication via events

Skip it when you need complex request transformation or business logic before publishing. But for the 80% case of “take this HTTP request and put it in SNS,” it’s exactly what you need.

The Cloudless Way

This module embodies our approach to infrastructure:

  • Solve real problems: Born from actual project needs
  • Eliminate unnecessary complexity: Direct API Gateway → SNS integration
  • Production-ready by default: Authentication, rate limiting, monitoring
  • Composable: Works with any SNS-based architecture

Sometimes the most valuable tools are the ones that make simple things simple to do right.

The terraform-aws-rest-api-sns-proxy module is available on GitHub. What started as a “simple” browser automation project became a reusable tool that’s now part of our standard toolkit for event-driven architectures.