🔒 Signed an NDA on this one — the numbers below are real, but the client, the dataset specifics, and the repo stay private. You’ll have to use your imagination for the rest.
Problem
A client needed to monitor a public French legal-announcement feed (BODACC) for signals worth acting on — but a raw feed of legal filings isn’t a lead list. Turning it into one meant reliable ingestion at daily scale, deterministic classification of what each filing actually means, and a scoring model the client could trust and audit, not a black box.
Constraints
Freelance solo engagement, fixed budget, no dedicated ops team on the client side — which meant the pipeline had to be operationally boring: scheduled, self-recovering, and legible to a non-engineer reading the output, because I wasn’t going to be the one watching it run every day.
Approach & architecture
In general terms: the pipeline downloads the daily public feed, classifies each announcement against externally configured event rules, optionally enriches company identity data, and scores/prioritizes the result — all through a deterministic, config-driven decision engine rather than a model, so every score traces back to an inspectable rule.
flowchart LR
A[Daily feed download] --> B[Idempotent ingestion<br/>+ dedup]
B --> C[Deterministic classification<br/>YAML event rules]
C --> D[Optional enrichment]
D --> E[Scoring & priority bands]
E --> F[(SQLite, versioned decisions)]
F --> G[CSV / XLSX output]Business logic — event classes, scoring weights, keyword/jurisdiction rules — lives entirely in versioned YAML, not code, so the client’s rules can evolve without a redeploy, and every historical decision stays reproducible against the configuration snapshot it was made under. Validation, retry/backoff, and crash recovery are load-bearing, not decorative: the pipeline has real OS-level concurrency tests (separate processes racing the actual lock file) and real crash-recovery tests (an actual SIGKILL mid-run), not just happy-path coverage.
The one architectural decision I’d point to specifically: the original design paginated through the feed’s /records API. Analysis of a full year of historical volume showed the 10,000-record pagination ceiling actually binds on a large share of days. That finding drove a full redesign to streaming the feed’s JSONL export instead of paging — a case where the fix came from measuring the data, not from a code review.
What I’d change now
A production run once stalled at a fraction of a record per second, and the instinct was to blame the enrichment API. Benchmarking instead of guessing found the real bottleneck: committing to the database once per record. Batching commits fixed it — 17x wall-clock on an identical workload re-run — which is the kind of bug that looks like a network problem until you actually measure it. I’d build the commit-batching pattern in from day one next time, instead of discovering it under load.
Impact
A full specification-compliance audit traced 161 distinct requirements against the shipped system: 97.5% compliant, with the remaining gap documented and authorized rather than silently dropped. Independent verification hand-reconstructed 12 real records against pipeline output — 12/12 exact matches. The system runs under a suite of roughly 1,000 automated tests, including real concurrency and crash-recovery scenarios, not just unit coverage over the happy path.