SignalOps

§ 01 · problem

The first version of SignalOps dropped messages under load and page-fatigued its users with flapping alerts. 2,000 logs/sec was a soft ceiling and the alert pipeline fired on every noisy minute.

§ 02 · constraints

• A single small container has to swallow a burst of a million logs without losing any.
• Alerts must stop crying wolf: zero false positives on a replay set of 1,200 ticks.
• p95 search over 1M documents has to stay under 20 ms, even past the OpenSearch 10,000-hit cap.
• Everything runs in CI on real containers — no mocking Redis or OpenSearch.

§ 04 · decisions

1. Why Redis Streams and not Kafka?

   This runs on one small box. Kafka is a cluster and an operational tax; Streams give me consumer groups, XCLAIM for stalled messages, and durable logs in one Redis process. If I outgrow it, the ingest contract is the same.

2. Why cursor paging past OpenSearch's 10,000 limit?

   `from + size` dies past 10k. `search_after` with a stable sort (timestamp, _id) is O(1) per page and plays nicely with the UI's infinite scroll.

3. Why integration tests over mocks?

   Mocking Redis Streams hides exactly the bug I'd care about (stalled-message recovery). 92 tests run against live containers in CI. Slow is fine; wrong is not.

§ 05 · results

• 2,000 → 9,750 logs/sec sustained throughput (4.8×).
• 19,810 docs/sec ingest at the 1M-doc load-test peak.
• p95 search under 15 ms past the 10k OpenSearch cap.
• 0 false positives across 1,200 evaluation ticks.