Architecture
System overview
The stack, the layers, and what calls what.
One laptop, one SQLite file, one Python package, one dashboard. ZoNuLy is deliberately small: a deterministic pipeline that calls models only for narrow tasks, and background jobs that keep every company filling.
The stack#
| Layer | Technology | Why |
|---|---|---|
| Pipeline | Python 3.12 · httpx · BeautifulSoup · lxml · feedparser | Scraping and parsing are Python’s home ground (D1). |
| Storage | SQLite (WAL) via SQLModel | One file, no server, survives a laptop (D9). Columns are added in place by ensure_columns — no migration tool. |
| API | FastAPI + uvicorn, ~100 routes | Long work runs on background threads and is polled as a task. |
| Scheduling | macOS launchd agents + in-process APScheduler | launchd survives sleep and restarts; the API’s scheduler handles the daily scrape and reply checks. |
| Public copy | Vercel (dashboard) · Catalyst AppSail (API) · demo.db.gz | A read-only window onto a masked snapshot — see Deployment. |
| Models | OpenRouter (aliases cheap / judge / writer) | No local model on an 8 GB machine (D18). Every call is budget-checked and costed. |
| Graph | SQLite tables + FTS5 + NetworkX | The project’s memory in the same file as the data (D10). |
| Dashboard | Next.js 16 · React 19 · Tailwind 4 · GSAP · d3-force | A fast, dense, black-and-white instrument over the API. |
| Gmail over SMTP / IMAP with an app password | Sends from your own address; reads replies back. |
The layering rule#
Imports flow one way. A lower layer never imports a higher one, which keeps the scrapers testable in isolation and lets the API change without touching the pipeline.
dashboardNext.js — talks to the API over HTTP only
app/(app)/*app/docs/*lib/api.ts
apiFastAPI routes, background tasks, access guard
api.pyoutreach/routes.pyresearch/routes.pykg/routes_warm.py
pipeline · matcher · kgOrchestration, scoring, deciding and the graph
tick.pypipeline.pymatcher.pytargeting.pyconfidence.pydecide.pymixer.pyupskill.pyfill.pyreferrals.pykg/
scrapers · contacts · outreach · research · actorsTalking to the outside world
scrapers/contacts/outreach/research/actors/
db · llm · normalizeThe foundation
db.pyllm.pyopenrouter.pynormalize.pyguardrails.pyskills.py
Note
One documented exception: outreach/researcher.py reaches up into kg/warm.py through a lazy, guarded import, to use warm paths when writing.Package map#
Core#
tick.py- The background pipeline: bounded slices of work run by launchd, each with a budget and a lock.
pipeline.py- Runs the scraper fleet, filters, de-duplicates and stores roles, admitting each company through the gate.
admission.py- The one gate for new companies: name clean-up, bogus-site rejection, de-duplication, the overrated rule.
fill.py- Completes one company without a model: description, LinkedIn page, hiring proof, funding — recording why anything is missing.
referrals.py- “The main thing”: the people step, the referral cycle, addresses, Hunter and Apify steps, re-verification.
guardrails.py- The validation layer every extracted value passes before it is written, with provenance.
matcher.py · fit.py- A free lexical gate, then a model rubric on the survivors, then hard caps.
targeting.py · confidence.py- Tiers, pay power, priority — and per-role selection odds.
decide.py- The decision brain: a chance of an offer per company from five questions, the shortlists, one company a day, calibration.
mixer.py- Decide first, then buy: free layers, then Apify job ads only where data is missing. Manual only.
upskill.py- The profile’s gaps across the shortlist, ranked by expected offers gained; proof-gated additions to the profile.
careers.py · hiring_verify.py- Find any careers page and read its roles; corroborate a hiring claim.
india.py · india_sources.py · india_jobs.py- Bengaluru / Hyderabad discovery and the India pay floor.
harvest.py · enrich.py · levels.py- Bulk company discovery; facts and stories; levels.fyi pay.
audit.py · watchdog.py · backoff.py · channels.py- Self-monitoring: hourly audit, stuck-job alarms, per-company retry backoff, channel health.
Subpackages#
| Package | Responsibility |
|---|---|
| scrapers/ | One module per source, each exposing fetch(http, companies) → RawJob list: Greenhouse, Lever, Ashby, HN, RemoteOK, WWR, YC, Wellfound, careers pages, funding RSS, India boards. |
| contacts/ | The people and address waterfall: GitHub commits, team pages, public LinkedIn, public mentions, address patterns, MX checks, Hunter, role classes, affiliation. |
| outreach/ | Research → compose → gates → review → ledger → send → track replies → follow up → schedule → learn. |
| research/ | Data acquisition only (search, page read, GitHub, Reddit, YouTube), routed across ordered backends with a 24 h cache. Never writes pipeline tables. |
| actors/ | “Our own Apify”: LinkedIn guest-page actors with a run ledger, datasets, a page cache and a per-host circuit breaker; plus the Apify People-tab source. |
| kg/ | The knowledge graph: store, context loader, data sync, brief writer, analysis, warm paths. |
External services#
| Service | Used for | Limit |
|---|---|---|
| OpenRouter | Rubric scoring, role classes, drafting polish, reply classification | 18 req/min · ₹100/day · ₹2,500/month · free models first |
| Jina Reader | Reading pages; keyless search via DuckDuckGo / Startpage | Paced; cools down on errors |
| Exa | Company discovery, funding facts, one people search per company | 100 searches a day |
| GitHub API | Commit emails, org members, recent committers | 5,000 / hour with a token, reserves kept |
| Apify | A company’s LinkedIn People tab; the mixer’s LinkedIn and Naukri job ads | $0.60 / day — the $19 Starter credit over a month; the mixer may use $0.05 of it |
| Hunter.io | Learning address patterns | 25 lookups / month (currently off: no key) |
| scrape.do | levels.fyi pay pages | 120 / day · 1,000 / month |
| Gmail | Sending and reading replies | 25 / day by our own cap |
Continue with the Data model and Scheduling.