Back to Projects
ML Pipelines
AI/ML Job Market Intelligence Pipeline
End-to-end ML pipeline that scrapes RemoteOK, clusters jobs with K-Means + TF-IDF, classifies seniority across 3 models, generates LLM market insights via Groq LLaMA 3.3 70B, and pushes automated reports to Google Sheets — all from one API call.
Last updated April 2026
Overview
An end-to-end ML data pipeline that monitors the AI/ML job market in real time. A single POST /run-pipeline triggers the full chain: async scraping → PostgreSQL storage → TF-IDF clustering → seniority classification → Groq LLaMA 3.3 70B market briefing → Google Sheets report with a dated tab per run.
Built as a portfolio project to demonstrate every layer of an ML engineering stack — scraping, databases, unsupervised and supervised ML, LLM integration, external API automation, and containerised deployment. Live run: 97 jobs scraped, 8 clusters found, 36/36 tests passing.
Pipeline Architecture
| Step | Component | Technology |
|---|---|---|
| 1 | Async Scraper | aiohttp · asyncio · BeautifulSoup4 · RemoteOK JSON API |
| 2 | Storage | PostgreSQL · SQLAlchemy 2 · Alembic migrations |
| 3 | Feature Extraction | TF-IDF vectorisation (5,000 features) · skill frequency analysis |
| 4 | Clustering | K-Means · silhouette k-selection (k=2–7) · PCA visualisation |
| 5 | Seniority Classifier | Logistic Regression · LinearSVC · Random Forest · 5-fold CV F1 |
| 6 | LLM Insights | Groq API · LLaMA 3.3 70B · structured prompt template |
| 7 | Reporter | gspread · Google Service Account · dated tab per run |
| 8 | API | FastAPI · uvicorn · POST /run-pipeline |
Key Engineering Decisions
- Silhouette k-selection:
find_optimal_k()tries k=2–7 and picks the highest silhouette score automatically — no hardcoded cluster count, adapts to each day's data distribution - Classifier comparison: All three models are evaluated via 5-fold cross-validation F1 macro on every run — the winning model and score are returned in the API response
- RemoteOK JSON API over Playwright: Switched from headless browser scraping to the public JSON API via aiohttp — more reliable, faster, no browser install
- Deduplication by URL: Jobs are keyed on
source_url— re-running never double-counts; second run on the same day showsjobs_scraped: 0by design - SQLite in tests: Integration tests use SQLite in-memory with mocked LLM and Sheets — no external services needed in CI
- Groq free tier: LLaMA 3.3 70B via Groq — no credit card required; rate-limit retries built in with 10-second backoff
Live Run Results (2026-04-23)
| Metric | Result |
|---|---|
| Jobs scraped | 97 |
| Clusters found | 8 |
| Best classifier (F1) | LogisticRegression — 0.417 |
| LinearSVC F1 | 0.404 |
| Random Forest F1 | 0.351 |
| Tests passing | 36 / 36 |
Google Sheets Output
Each pipeline run writes a new dated tab (e.g. 2026-04-23) containing:
- LLM-generated 2–3 paragraph market intelligence briefing
- Top 20 skills by mention rate across all job listings
- Cluster breakdown — job count and percentage per cluster
- Seniority distribution — junior / mid / senior counts
Interested in similar work?
I build AI agents, full-stack applications, and cloud-native systems. Let's discuss your next project.
Get in TouchOther Projects
Platinum TierAI Agents
Personal AI Employee
A 24/7 autonomous AI agent that monitors Gmail, WhatsApp, and filesystem — drafts responses on a cloud VM while you're offline, then executes with your approval when you reconnect.
PythonFastMCPClaude CodeGmail OAuth2Playwright
Platinum TierFull Stack
AHF Auto Parts
Production-grade e-commerce platform for a Japan-based auto parts business — vehicle compatibility lookup, Stripe checkout, admin dashboard, and full order management.
Next.jsTypeScriptPostgreSQLPrismaNextAuth.js
Full Stack
Campaign Manager
A CRM prototype with contacts CRUD, audience segmentation rule builder, multi-step campaign wizard, and analytics dashboard — powered by Prisma and Turso.
Next.js 16TypeScriptTailwind CSSShadCN UIPrisma