S3 → Lambda Thumbnail Processing

Shell-first image processing at scale

Thumbnail generation is inherently an OS-level task: a file comes in, a utility processes it, files come out. There’s no application logic, no framework, no state — just a pipeline of tools. Shell scripts are the natural fit.

This guide walks through a production-ready S3 → Lambda → S3 thumbnail pipeline using bash functions, vipsthumbnail, and composable Lambda layers. The function itself is ~40 lines of shell. Everything else — runtime, image processing, JSON parsing — comes from pre-built layers.

The Challenge

Traditional serverless image processing faces several issues:

• Heavy runtimes: Node.js/Python with image libraries consume 300-500MB memory
• Complex dependencies: ImageMagick, Sharp, or Pillow with native bindings
• Slow cold starts: 500ms+ initialization times
• Event parsing complexity: Nested JSON structures require careful handling

The Shell-First Solution

Our approach uses:

• terraform-aws-lambda-shell-runtime-layer — a sub-25ms Go bootstrap published as a Lambda layer
• lambda-shell-layers — pre-built tool layers (jq, vipsthumbnail, etc.)
• terraform-aws-lambda-layer — Terraform module to provision layers from source or URL
• S3 event triggers parsed with jq
• ZIP packages — no container images, no Dockerfile, no ECR

Architecture Overview

S3 Source Bucket → S3 Event → Lambda (Shell + Layers) → vipsthumbnail → S3 Thumbnails Bucket

The complete pipeline processes images automatically when uploaded to the source bucket, generates thumbnails using the lightning-fast vipsthumbnail from a pre-built layer, and stores results in a separate thumbnails bucket.

The Evolution: From Container Images to Composable Layers

The original version of this pipeline used a custom container image (lambda-shell-runtime:full at 417MB) with vips compiled inside it. That approach worked, but it had drawbacks:

• Slow Docker builds when iterating on the handler
• Large ECR images to push on every deploy
• vips compiled per-project instead of shared
• 155ms cold starts (good for containers, but layers do better)

The layer-based approach eliminates all of these:

	Container Image (original)	Composable Layers (current)
Cold start	155ms	~21ms
Deploy artifact	417MB container	~1KB zip (handler only)
vips	Compiled per-project	Shared pre-built layer
Build time	Minutes (Docker build + push)	Seconds (zip + upload)
Dockerfile	Required	None
ECR	Required	Not needed

Infrastructure Setup

Terraform Configuration

The entire infrastructure is declarative — runtime, tools, function, buckets, and triggers:

# Shell runtime layer (Go bootstrap, sub-25ms cold starts)
module "shell_runtime" {
  source  = "ql4b/lambda-shell-runtime-layer/aws"
  version = "~> 1.0"

  name         = "shell-runtime"
  architecture = "arm64"
}

# vipsthumbnail layer (pre-built from lambda-shell-layers releases)
module "vipsthumbnail" {
  source  = "ql4b/lambda-layer/aws"
  version = "~> 1.0"

  name       = "vipsthumbnail"
  source_url = "https://github.com/ql4b/lambda-shell-layers/releases/download/v0.2.0/vipsthumbnail-arm64-layer.zip"

  compatible_architectures = ["arm64"]
  enable_ssm_parameters    = false
}

# jq layer for JSON parsing
module "jq" {
  source  = "ql4b/lambda-layer/aws"
  version = "~> 1.0"

  name       = "jq"
  source_url = "https://github.com/ql4b/lambda-shell-layers/releases/download/v0.2.0/jq-arm64-layer.zip"

  compatible_architectures = ["arm64"]
  enable_ssm_parameters    = false
}

# The Lambda function — just shell scripts
module "thumbnail_function" {
  source  = "ql4b/lambda-function/aws"
  version = "~> 1.0"

  source_dir   = "./app"
  name         = "thumbnail-processor"
  runtime      = "provided.al2023"
  handler      = "handler.thumb"
  architecture = "arm64"
  memory_size  = 512
  timeout      = 60

  layers = [
    module.shell_runtime.layer_arn,
    module.vipsthumbnail.layer_arn,
    module.jq.layer_arn,
  ]
}

S3 Buckets and Event Triggers

module "source_bucket" {
  source  = "cloudposse/s3-bucket/aws"
  version = "~> 4.0"

  context    = module.label.context
  attributes = ["source"]

  versioning_enabled = false
  force_destroy      = true
}

module "thumbnails_bucket" {
  source  = "cloudposse/s3-bucket/aws"
  version = "~> 4.0"

  context    = module.label.context
  attributes = ["thumbnails"]

  versioning_enabled = false
  force_destroy      = true
}

# S3 trigger for Lambda
resource "aws_s3_bucket_notification" "image_upload" {
  bucket = module.source_bucket.bucket_id

  lambda_function {
    lambda_function_arn = module.thumbnail_function.function_arn
    events              = ["s3:ObjectCreated:*"]
    filter_suffix       = ".jpg"
  }

  lambda_function {
    lambda_function_arn = module.thumbnail_function.function_arn
    events              = ["s3:ObjectCreated:*"]
    filter_suffix       = ".png"
  }

  lambda_function {
    lambda_function_arn = module.thumbnail_function.function_arn
    events              = ["s3:ObjectCreated:*"]
    filter_suffix       = ".webp"
  }
}

IAM Permissions

# Function needs read access to source bucket and write access to thumbnails bucket
{
  Effect = "Allow"
  Action = ["s3:GetObject"]
  Resource = "${module.source_bucket.bucket_arn}/*"
},
{
  Effect = "Allow"
  Action = ["s3:ListBucket"]
  Resource = module.source_bucket.bucket_arn
},
{
  Effect = "Allow"
  Action = ["s3:PutObject"]
  Resource = "${module.thumbnails_bucket.bucket_arn}/*"
}

Key insight: AWS CLI’s s3 cp requires s3:ListBucket on the bucket ARN (without /*) to verify object existence before downloading. This commonly overlooked permission causes mysterious timeouts that look like network issues.

Shell Handler Implementation

The complete image processing logic — this is the entire function deployment package:

#!/bin/bash
# handler.sh

set -euo pipefail

# Parse S3 event and extract bucket/key
parse_s3_event() {
    local event="$1"
    echo "$event" | jq -r '.Records[0].s3.bucket.name + "|" + (.Records[0].s3.object.key | @uri | gsub("%2F"; "/"))'
}

# Generate thumbnail using vipsthumbnail from the layer
generate_thumbnail() {
    local input_file="$1"
    local output_file="$2"
    local size="${3:-200x200}"

    vipsthumbnail "$input_file" -s "$size" -o "${output_file}[Q=80,strip]"
}

# Main handler function
thumb() {
    local event="$1"
    echo "Processing S3 event..." >&2

    # Parse S3 event
    local bucket_key
    bucket_key=$(parse_s3_event "$event")
    local source_bucket="${bucket_key%%|*}"
    local object_key="${bucket_key#*|}"

    echo "Processing: s3://$source_bucket/$object_key" >&2

    # Download image from S3
    local input_file="/tmp/input_$(basename "$object_key")"
    curl --aws-sigv4 "aws:amz:${AWS_REGION}:s3" \
        --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
        -H "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
        -sSf "https://${source_bucket}.s3.${AWS_REGION}.amazonaws.com/${object_key}" \
        -o "$input_file"

    echo "Downloaded $(wc -c < "$input_file") bytes" >&2

    # Generate thumbnail
    local thumbnail_file="/tmp/thumb_$(basename "$object_key")"
    generate_thumbnail "$input_file" "$thumbnail_file" "200x200"
    echo "Thumbnail: $(wc -c < "$thumbnail_file") bytes" >&2

    # Upload thumbnail to destination bucket
    local thumbnail_key="thumbnails/${object_key}"
    local content_type="image/jpeg"
    [[ "$object_key" == *.png ]] && content_type="image/png"
    [[ "$object_key" == *.webp ]] && content_type="image/webp"

    curl --aws-sigv4 "aws:amz:${AWS_REGION}:s3" \
        --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
        -H "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
        -H "Content-Type: ${content_type}" \
        -sSf -X PUT \
        --data-binary "@${thumbnail_file}" \
        "https://${THUMBNAILS_BUCKET}.s3.${AWS_REGION}.amazonaws.com/${thumbnail_key}"

    echo "Uploaded to: s3://${THUMBNAILS_BUCKET}/$thumbnail_key" >&2

    # Cleanup
    rm -f "$input_file" "$thumbnail_file"

    echo "{\"source\": \"$source_bucket/$object_key\", \"thumbnail\": \"${THUMBNAILS_BUCKET}/$thumbnail_key\"}"
}

Note: this uses curl --aws-sigv4 which is available in provided.al2023 out of the box — no AWS CLI needed. This eliminates the need for the heavy full runtime variant and keeps the function dependency-free beyond its layers.

Performance

With the layer-based architecture on arm64:

Cold Start:    ~21ms init
Memory:        ~80MB / 512MB (15%)
Processing:    ~1.5s (download + thumbnail + upload for a 3MB JPEG)
Deploy size:   ~1KB (handler.sh only)

Compared to the original container image approach:

Metric	Container Image	Layers	Improvement
Cold start	155ms	~21ms	86% faster
Memory	137MB	~80MB	42% less
Deploy artifact	417MB	~1KB	99.99% smaller
Build time	Minutes	Seconds	Orders of magnitude

Why Layers Beat Container Images Here

The complete journey showed that the runtime-as-a-layer architecture delivers sub-25ms cold starts with zero overhead from layer separation. Adding tool layers (jq, vipsthumbnail) doesn’t measurably increase init time — Lambda loads layers efficiently regardless of count.

Container images make sense when you need:

• Complex system-level configuration
• Tightly coupled dependencies
• Full control over the OS

For image thumbnails, you need exactly three things beyond the OS: a bootstrap, a JSON parser, and an image resizer. Three layers. No Dockerfile.

vipsthumbnail Layer

The vipsthumbnail layer is built from lambda-shell-layers and ships:

• vipsthumbnail binary (libvips 8.16.1)
• Shared libraries (libjpeg-turbo, libpng, libwebp, libtiff, glib2)
• Supports JPEG, PNG, WebP, TIFF input; JPEG, PNG, WebP output

# Fit within 200x200, preserving aspect ratio
vipsthumbnail input.jpg -s 200x200 -o output.jpg[Q=80,strip]

# Output as WebP
vipsthumbnail input.jpg -s 300x300 -o output.webp[Q=75]

# Smart crop to exact dimensions
vipsthumbnail input.jpg -s 200x200 --smartcrop attention -o thumb.jpg

vipsthumbnail uses a streaming pixel pipeline — it doesn’t load the full image into memory. This is why memory usage stays low even for large source images.

Testing the Pipeline

# Upload test image
aws s3 cp test-image.jpg s3://your-source-bucket/photos/test.jpg

# Watch logs
aws logs tail /aws/lambda/thumbnail-processor --follow

# Verify thumbnail
aws s3 ls s3://your-thumbnails-bucket/thumbnails/photos/

Project Structure

The function deployment is minimal:

app/
└── handler.sh    # ~40 lines of shell

That’s it. The runtime, jq, and vipsthumbnail all come from layers. Your deployment package is just your business logic.

Key Learnings

1. curl —aws-sigv4 Replaces AWS CLI

The provided.al2023 runtime ships curl with SigV4 support. For simple S3 get/put operations, this eliminates the need for the AWS CLI entirely — and avoids the binary data corruption issues we found with awscurl.

2. IAM Permission Precision

s3:ListBucket on the bucket ARN (not /*) is required for AWS CLI’s s3 cp. With curl you technically don’t need it, but include it if you plan to use AWS CLI for debugging or if your function does HEAD checks.

3. Layer Count Doesn’t Affect Cold Starts

The runtime layer benchmarks show that going from 1 to 3+ layers adds no measurable init overhead. Lambda loads all layers during environment creation before the bootstrap runs.

4. Memory Efficiency

vipsthumbnail’s streaming architecture means a 10MB source image doesn’t need 10MB of heap. The function processes images well within 512MB regardless of input size (up to Lambda’s /tmp limits).

Conclusion

The layer-based approach to shell thumbnail processing delivers:

• Sub-25ms cold starts — 86% faster than the container image approach
• ~1KB deploy packages — just your shell script
• Composable architecture — swap vipsthumbnail for ffmpeg, add more tools without rebuilding
• No Docker in the deploy path — no builds, no ECR, no image tags to manage
• Production-ready Terraform — three modules, fully declarative

The original container image version proved shell-based image processing works. The layer evolution proves it can be fast, composable, and operationally simple.

---

This example uses the cloudless shell-first ecosystem:

• terraform-aws-lambda-shell-runtime-layer — Go bootstrap as a layer
• lambda-shell-layers — pre-built tool layers (jq, vipsthumbnail, htmlq, etc.)
• terraform-aws-lambda-layer — Terraform module for provisioning layers

Related

• Shell on Lambda: The Journey to Sub-25ms Cold Starts — the complete optimization story
• Lambda Layers Breakthrough — the composable layer philosophy
• Test Lambda Functions Locally — run functions on your laptop with the RIE