Signals #3: GitHub Commit History as a Queryable Dataset

GitHub knows your commit history. But you can’t query it across orgs. You can’t visualize cadence across private repos. You can’t answer “how much did I ship this quarter?” without clicking through dozens of repositories.

This signal collects all your commits — from orgs you own, user accounts, and repos where you’re a collaborator — merges them into a single JSON file, uploads to S3, and visualizes in Grafana.

Result: A scattered time series showing every commit across every org, every private repo, every collaboration — filterable by date, owner, and visibility.

---

The Problem

Your work is spread across:

• Orgs you own (my-org, my-other-org)
• Your personal user account (my-user)
• Repos you collaborate on but don’t own (collaborator-org/*)

GitHub’s contribution graph only shows public repos. The API gives you per-repo data but no cross-org view. You need the full picture.

The Script

source scripts/git-history.sh

Five composable functions, one pipeline:

1. Fetch Branches

git_history_branches () {
  local org="${1:-my-org}"
  local type="${TYPE:-orgs}"
  local outdir="/tmp/$org"
  mkdir -p "$outdir"
  local outfile="$outdir/$org.repo.branches.jsonl"

  gh api "$type/$org/repos" --paginate \
  | jq -r '.[]|select(.fork == false).name' \
  | while read -r repo; do
      gh api "repos/$org/$repo/branches" \
      | jq --arg repo "$repo" -c '{ repo: $repo, branches: . }' \
      >> "$outfile"
      sleep 0.25
    done

  echo "$outfile"
}

For orgs: git_history_branches my-org For users: TYPE=users git_history_branches my-user

2. Filter Multi-Branch Repos

git_history_multibranch () {
  local infile="$1"
  jq -c '{
    repo,
    branches: [.branches[].name],
    size: .branches|length
  } | select(.branches|length > 0)' "$infile"
}

Only repos with at least one branch (filters out empty repos).

3. Fetch All Commits

git_history_commits () {
  local infile="$1"
  local org="${2:-my-org}"

  jq -r '.repo + " " + .branches[]' "$infile" \
  | while read -r repo sha; do
      gh api "repos/$org/$repo/commits?sha=$sha" \
      | jq -c --arg repo "$repo" --arg sha "$sha" \
        '.[]| { repo: $repo, sha: $sha } + .'
    done
}

Iterates every branch of every repo. Produces JSONL with full commit metadata.

4. Filter by Author

git_history_filter () {
  local author="${1:-my-user}"
  local infile="$2"

  jq -c --arg author "$author" '
    { repo, committer: .committer.login, message: .commit.message, date: .commit.author.date }
    | select(.committer == $author)' "$infile"
}

5. Run the Pipeline

# Org you own
git_history_all my-org

# Your user account
TYPE=users git_history_all my-user

# Repos you collaborate on (filtered by owner)
git_history_all_collab collaborator-org

# Only repos starting with "project-"
git_history_all_collab collaborator-org "project-"

Collaborator Repos

You can’t use orgs/ or users/ endpoints for repos you don’t own. Instead, list all repos you have access to and filter by owner:

git_history_collab () {
  local owner="${1:-collaborator-org}"
  local prefix="${2:-}"

  gh api "user/repos?per_page=100" --paginate \
  | jq -r --arg owner "$owner" --arg prefix "$prefix" '
    .[]|select(.owner.login == $owner and .fork == false)
    | select($prefix == "" or (.name|startswith($prefix)))
    | .name'
}

gh api user/repos returns every repo you have access to — owned, organization, collaborator. Filter by .owner.login to get repos from a specific owner.

Merging Datasets

After running all pipelines, merge into a single sorted dataset:

jq -s 'sort_by(.date)' \
  /tmp/my-user/my-user.commits.my-user.jsonl \
  /tmp/my-other-org/my-other-org.commits.my-user.jsonl \
  /tmp/my-org/my-org.commits.my-user.jsonl \
  /tmp/collaborator-org/collaborator-org.commits.my-user.jsonl \
| jq -c | tee /tmp/commits.my-user.json

upload_dataset /tmp/commits.my-user.json

One JSON array. Every commit. Sorted by date. Uploaded to S3, served via CloudFront.

Record Shape

{
  "repo": "airbridge",
  "committer": "my-user",
  "message": "feat: add card pool selection to fulfillment payload",
  "date": "2026-06-24T14:32:11Z"
}

Minimal. Enough for time series visualization and cadence analysis.

Grafana Visualization

Connect Grafana’s Infinity datasource to the uploaded JSON. Use JQ to parse with time range filtering:

map(. + { day: .date[0:10] } )
 | (${__from:date:seconds}) as $from
 | (${__to:date:seconds}) as $to
 | map(. + { ts: .date|fromdate })
 | map(select(.ts <= $to and .ts >= $from))
 | group_by(.org, .day)
 | [.[] | {
     owner: first.org,
     date: first.day,
     count: length,
     visibility: "private",
     from: $from,
     $to
   }]

Result: a scattered time series showing daily commit counts per org/owner across your selected time range.

Questions This Answers

• Engineering cadence — Am I shipping consistently or in bursts?
• Cross-org activity — Where am I spending my time?
• Repository evolution — When did repos go active/dormant?
• Collaboration patterns — Which external repos get my attention?
• Historical reconstruction — What was I building 6 months ago?

The Pattern

GitHub API → gh + jq → JSONL per org → Merge → S3 → Grafana

Run when you want an updated view. The dataset is append-only in the sense that history doesn’t change — you’re collecting facts.

This is signal collection in its simplest form: shell scripts producing datasets from APIs, uploaded as static JSON, consumed by visualization tools.

Modules Used

Tool	Purpose
gh	GitHub CLI with authentication
jq	JSON filtering and transformation
upload_dataset (lib.sh)	S3 upload to datasets bucket
Grafana Infinity	JSON datasource with JQ parsing

---

Part 3 of the Signals series — small systems that observe larger systems.

Previous: Signals #2: Browser Events to S3 Data Lake in Bash | Next: Signals #4: Mapping the Terraform Module Ecosystem