Problem
A crawler that fetches everything reachable from a seed set drowns in irrelevant pages long before it finds the ones that matter. For a research project on focused web exploration, the naive approach — breadth-first traversal plus an LLM call per candidate link — was both too slow to iterate on and too expensive to run at any real scale: every link, relevant or not, cost the same LLM round-trip.
Constraints
This was a research software engineering project (PFE at ISI), built and defended solo against a fixed academic timeline, with no budget for unlimited LLM calls and no production traffic to learn from — every architectural bet had to be validated against benchmark workloads I built and measured myself.
Approach & architecture
The core idea: don’t ask an LLM whether a link is worth following until a cheap local signal has already ruled out the obvious no’s.
flowchart LR
A[Seed URLs] --> B[Fetch & extract links/content]
B --> C[Local embedding similarity<br/>sentence-transformers]
C -->|low relevance| D[Discard]
C -->|candidate| E[Selective LLM judgment<br/>+ topic expansion]
E --> F[Priority-based frontier]
F --> B
E --> G[(SQLite state)]
G --> H[FastAPI + WebSockets]
H --> I[React/D3 live graph]The system runs as an event-driven pipeline of roughly 16 decoupled stages — acquisition, extraction, filtering, scoring, frontier prioritization, retries, transformation, storage, and telemetry — communicating through explicit state rather than a monolithic loop. That decoupling is what made the two-stage relevance cascade possible: local embedding similarity narrows the field first, and only the surviving candidates get a selective LLM call for relevance judgment and topic expansion. The network-heavy stages run on asyncio/aiohttp instead of a threaded or synchronous fetcher, which is most of where the wall-clock win comes from once the LLM-call volume is already down.
Every run is traceable and replayable: state is explicit and persisted to SQLite, frontier decisions and LLM judgments are recorded, and a FastAPI + WebSocket layer streams live state to a React/D3 frontend so a crawl can be watched — and later replayed — as a graph, not just a log.
What I’d change now
The system depends on an external LLM API for the judgment stage, which means latency and availability are someone else’s problem on your critical path. A local, smaller model for the first-pass judgment (not just the embedding stage) would remove that dependency for the common case and reserve the hosted LLM for genuinely ambiguous candidates. HTML structure sensitivity is the other honest gap — the extraction stage assumes reasonably well-formed markup, and a hierarchical, more tolerant extractor would generalize further.
Impact
On a fixed 560-node benchmark, wall-clock time dropped from roughly 600 seconds to roughly 15 — about 40x — by combining the async I/O rewrite with the relevance cascade. A reference crawl surfaced 50,828 candidate links and explored 539 nodes, with roughly 1% of candidates retained for downstream exploration — the two-stage cascade is what keeps that filtering cheap: local embeddings do the bulk rejection, and the LLM is reserved for the ~1% that make it through.