The queue said 0. 500 orders were stuck.
A poison order made its batch retry forever, held in worker memory, so the queue stayed empty. Weir showed 500 items in flight at the worker, the oldest over 10 minutes old and climbing: stuck, not lost.
Find the data your pipelines lose silently.
Weir follows every item through queues, batches, crons and fan-outs, and shows what was lost, what is stuck, and where the time goes. Standard tools watch lag and errors. Weir counts items, so the problems that raise neither still show up, with exact numbers.
What standard tools missed
Three faults injected into a demo pipeline (Kafka, Celery, Postgres, a cron job), each checked against the pipeline's own ground truth.
2% of orders vanished, with no errors.
A stage dropped orders without logging a thing. Weir named the stage and the share: 2.1% vanished at enrich. The true figure was 2.2%.
The job wasn't slow. The schedule was.
End-to-end delay tripled while every dashboard stayed green. Weir split it: 412 s of 416 s was waiting for a cron, and work time hadn't changed. The true wait was 412 s.
How it works
Every item that enters a stage leaves it exactly once, with one outcome. When that doesn't add up, something was lost, and Weir shows where.
An SDK in each stage
A few lines per service. Items carry their birth time between stages, so every age counts from the start, queue time included.
A scraper between stages
Reads Kafka offsets and staging tables with no code changes, so items waiting between stages show up while they wait, not after.
Counts, not samples
Every item is counted, and only totals leave the process: about 4 µs per message and under 1 KB/s of telemetry at any volume. Built on OpenTelemetry semantic conventions.
- passed
- done
- dropped, with a reason
- errored
- vanished: silent loss
import weir
weir.init(pipeline="orders")
enrich = weir.stage("enrich")
with enrich.receive(weir.extract(msg.headers())) as item:
for line in split(msg):
producer.produce("lines", line, headers=weir.inject(item.emit()))
# no emit and no outcome? the item vanished, and Weir counts itLooking for design partners
We're onboarding a few teams that run Kafka, Celery or cron pipelines, to run Weir on one real pipeline and tell us what it finds.