independent · recommendation · real-time search · distributed systems
NewsLens began by asking how much recommendation quality survives a chronological evaluation. New articles raised a second problem: they had to become searchable without coupling event delivery to the model server, so the project now has a separate Go, Kafka, PostgreSQL, and FastAPI path with measured load and recovery behavior.
All fitting happens on the chronological training partition: popularity counts, TF-IDF vocabulary, and engagement features. The split explicitly verifies that the latest training timestamp precedes the earliest validation timestamp.
The content model handles 30,466 validation impressions. The remaining 927 — mostly cold-start users plus zero-signal histories — are routed to a popularity model fitted only on training data. The score types are not blended because cosine similarity and click counts are on incompatible scales.
The selected content + fallback system reaches NDCG@10 0.3664, Recall@10 0.5955, and Hit Rate@10 0.6762. Paired 95% bootstrap intervals for the gains over content-only exclude zero under the fixed offline protocol.
A batch model artifact should not own a growing queue of article updates. A Go API validates each event and waits for Kafka acknowledgement, while three keyed partitions absorb bursts and preserve order for each article. Two consumers write through a PostgreSQL event ledger, retry bounded failures, and move invalid or exhausted records to a dead-letter topic.
FastAPI reads a bounded candidate set and detects category, entity, and freshness intent. Ranking exposes lexical relevance, exponential freshness, and popularity components instead of hiding them behind one unexplained score. A six-query contract fixture matched all 18 labeled intent fields; its small size is reported as a limit, not a product-quality claim.
In the final local burst, all 500 events were accepted at 1,225 events/s. Publish p99 was 44 ms, sampled produced-to-indexed p95 was 79 ms, and all 25 duplicate replays were recognized. A probe targeted a partition owned by a stopped consumer and became searchable after Kafka reassigned it in 5.7 seconds; a backlog event recovered 2.4 seconds after one consumer restarted.
The live tests caught two issues before the documentation was written: Kafka's default producer batching held acknowledgements near one second, and synchronous offset commits pushed burst freshness above 18 seconds. Shorter batching and periodic commits fixed both. A malformed record also reached the DLQ with its original bytes, and an older article update arriving second could not replace the current version.
The frozen hybrid's positive holdout change crossed zero, so the serving recommendation baseline was not replaced. The real-time numbers come from one Docker Desktop host with one Kafka broker and one PostgreSQL instance. They prove this event path and its local process recovery, not replicated production capacity, multi-zone availability, or live-user impact.