AI/ML Job Market Intelligence Pipeline

Overview

An end-to-end ML data pipeline that monitors the AI/ML job market in real time. A single POST /run-pipeline triggers the full chain: async scraping → PostgreSQL storage → TF-IDF clustering → seniority classification → Groq LLaMA 3.3 70B market briefing → Google Sheets report with a dated tab per run.

Built as a portfolio project to demonstrate every layer of an ML engineering stack — scraping, databases, unsupervised and supervised ML, LLM integration, external API automation, and containerised deployment. Live run: 97 jobs scraped, 8 clusters found, 36/36 tests passing.

Pipeline Architecture

Key Engineering Decisions

• Silhouette k-selection: find_optimal_k() tries k=2–7 and picks the highest silhouette score automatically — no hardcoded cluster count, adapts to each day's data distribution
• Classifier comparison: All three models are evaluated via 5-fold cross-validation F1 macro on every run — the winning model and score are returned in the API response
• RemoteOK JSON API over Playwright: Switched from headless browser scraping to the public JSON API via aiohttp — more reliable, faster, no browser install
• Deduplication by URL: Jobs are keyed on source_url — re-running never double-counts; second run on the same day shows jobs_scraped: 0 by design
• SQLite in tests: Integration tests use SQLite in-memory with mocked LLM and Sheets — no external services needed in CI
• Groq free tier: LLaMA 3.3 70B via Groq — no credit card required; rate-limit retries built in with 10-second backoff

Live Run Results (2026-04-23)

Google Sheets Output

Each pipeline run writes a new dated tab (e.g. 2026-04-23) containing:

• LLM-generated 2–3 paragraph market intelligence briefing
• Top 20 skills by mention rate across all job listings
• Cluster breakdown — job count and percentage per cluster
• Seniority distribution — junior / mid / senior counts