CarBuyerAssistant

A staged pipeline — six continuous workers, four timer-driven oneshots, one Postgres. The ingester walks all five sources with one lot-first strategy per source; the original discover-the-auction-then-scrape-its-catalog worker pair was retired once a single cross-auction query became strictly better. An enricher normalizes and classifies each offer with an LLM; a valuator runs comps → fair value → landed cost → want-relative score; a notifier fires Discord embeds; a bid_poller watches active lots; nightly timers run a vision pass on the shortlist and distill closed lots into historical sales; a morning digest delivers whatever quiet hours deferred overnight. Each stage NOTIFY/LISTEN-signals the next. Failure of any worker doesn't cascade because Postgres holds the queue state.

Redis or RabbitMQ would have been the textbook queue choice. Both add a moving part to operate, monitor, and back up. Postgres NOTIFY hands all of that to the database I'm already running, and SELECT ... FOR UPDATE SKIP LOCKED makes claim/process/commit safe under concurrent consumers. The cost is that NOTIFY payloads are limited to ~8KB, so the messages only carry row IDs — claiming requires a follow-up query. For a system whose queue depth never exceeds a few hundred, that's a no-brainer trade.

The pivot that mattered: the first build decided for itself what a deal was — a rarity score, a flip margin, a recommended max bid. In practice I don't want a flip-finder; I want to know when the thing I'm already hunting shows up cheap. So the rebuild made a want a first-class row: explicit make/model/year/price/mileage/province criteria, matched against every incoming offer, scored as percent-and-dollars below fair value for that want. Fuzzy wants — archetypes — fan out to concrete models before matching, and a fire-once ledger keyed on (want, offer) means a lot seen twice never pings twice.

Adding classifieds forced the schema to tell the truth: an auction lot and a Kijiji listing are both offers, but only one has a hammer time. The old auction_lots monolith split into a vehicle_offer parent with auction_lot and private_listing children — joined-table polymorphism — so the enrich/value/score path is shared and only bid polling stays auction-specific.

A US lot at $9,000 is not a $9,000 lot. The landed-cost model prices the driveway number: the 25% surtax keyed off the VIN's build origin, the RIV fee, GST at the border, air-conditioning excise, a transport estimate — converted at the Bank of Canada rate captured at ingest. Age gates handle the antique and RIV exemptions. Deal scores and alerts rank on landed CAD, so a Montana pickup competes honestly against an Alberta one.

Each queue-claiming worker opens a dedicated psycopg connection at startup and runs pg_try_advisory_lock(hashtext("notifier")). The lock is held for the process lifetime; closing the connection releases it. If a second instance starts — typically because I ran a worker from a shell while systemd was also alive — it exits non-zero. systemd's Restart=always cycles it; if the real peer is still alive, the next retry also fails. That's the desired symptom: a noisy loop, not a corrupted database.

Without this guard, the SKIP LOCKED claim plus the orphan-recovery sweep both assume no concurrent claimer exists. Duplicates would mean duplicate Discord posts and double-charged LLM calls.

HiBid migrated its catalog to a SPA in May 2026 — the auction pages are empty shells, and lot data arrives via GraphQL POSTs to /graphql. The ingester rebuilds those queries directly, riding a Cloudflare-issued __cf_bm cookie bootstrapped on first contact. When Cloudflare later began 403ing plain httpx on that endpoint, the transport moved to curl_cffi, which impersonates a real browser's TLS fingerprint; the cookie ritual stayed the same. The alternative would have been driving the SPA with Playwright — but the GraphQL contract changes less often than the HTML rendering does, and it's an order of magnitude faster.

HiBid auctions soft-close: a bid in the last 60 seconds extends the lot. Polling at a fixed cadence either burns requests on lots that aren't close to closing or misses the action when one is. The bid_poller keeps a priority queue keyed on next-poll-time, tiered from 60 minutes when a lot is days away down to 30 seconds inside the soft-close window. Every bid change re-runs valuation, so the deal score tracks the live price. Polling continues past the nominal end time until the source reports the lot closed — that's how the system reconstructs final sale prices, which then feed the next valuation pass.

The install script copies systemd units into /etc/systemd/system rather than symlinking, then restorecons the repo to bin_t. Symlinked units on a Fedora host pick up user_home_t and systemd refuses to start them; copying + relabel is the only path that works without disabling SELinux. Every worker unit also ships with the hardening baseline — NoNewPrivileges, ProtectSystem=strict, ProtectHome=read-only, PrivateTmp, RestrictAddressFamilies, MemoryDenyWriteExecute. Process compromise inside any worker still can't write outside the repo path.