Market Pulse

Financial-news streams are noisy, duplicated, and bursty. Running model inference directly on every item wastes compute and makes backpressure difficult to see.

I wanted to understand the boundary between high-throughput stream processing and CPU-bound model inference, then make that boundary observable under load.

• Go ingestion fans out concurrent fetches (GDELT, RSS, replay) into a channel and produces to the news.raw Kafka topic.
• A Go processing consumer group deduplicates via content hash, normalizes, and filters before publishing to news.processed — so the model never runs on junk.
• Python worker processes (not threads) join an inference consumer group, run INT8-quantized FinBERT, and feed a rolling z-score shock detector observable in Grafana.

• Go services ingest, normalize, deduplicate, and publish versioned events through Kafka.
• A Python consumer group runs quantized ONNX FinBERT inference across worker processes.
• Prometheus metrics, Grafana dashboards, and an offline replay mode make the full path inspectable without external API keys.

• Replay mode exercises the same pipeline as live ingestion so throughput and failure behavior can be reproduced.
• A stress generator varies arrival rate and consumer count to expose queue growth and backpressure.

• A cheap filtering stage before inference reduces both cost and operational noise.
• Kafka works well as a language boundary when schemas are treated as the shared API.

• How should inference workers be autoscaled from lag without oscillating?
• How should model-version changes be replayed and compared against historical traffic?