Three months of silence in an IoT pipeline

In 2025 I was reviewing an air-quality platform built around ESP8266 sensor nodes, AWS IoT Core, DynamoDB, Lambda processing and a hosted dashboard. The system served a live public network connected to a climate programme across eight universities.

At first glance, the dashboard looked plausible. There were maps. There were colours. Previous readings were still available. Nothing on the page said that the newest observation was old.

That was the first problem.

Production path

The intended path was short enough to draw on one line. The DynamoDB stream trigger had been disabled. Data could reach storage without reaching the processing path that made it useful downstream. No visible crash marked the moment. The pipeline became quiet.

Discovery

I found the incident by comparing the freshness implied by the dashboard with timestamps and counts at each boundary of the ingestion path.

1. Latest reading visible to a user
2. Latest item stored in DynamoDB
3. Latest invocation of the processing Lambda
4. Latest successful map update

The timestamps stopped agreeing. That disagreement was more useful than the health indicators we had at the time.

Silent failure

IoT systems often fail at boundaries. A device can be alive while publishing malformed readings. A broker can accept messages while a downstream consumer is disconnected. Storage can continue filling while processing has stopped. A dashboard can render yesterday forever.

How long has it been since evidence crossed this boundary?

That question became the basis of the repair.

Repair

The trigger was restored, but restoring it was the small part. The larger change was to make silence observable.

I added freshness checks at the ingestion and processing boundaries, compared expected and observed update intervals, and made gaps visible to the dashboard. The interpolation path also became gap-aware. A heatmap should never create the impression of continuous measurement when observations are stale or absent.

• Last message received by sensor and location
• Last stored record
• Last downstream processing event
• Processing delay between storage and Lambda
• Missing intervals long enough to affect interpolation
• Dashboard freshness visible to the reader

Operating rule

Every stage that consumes time-series data should expose two clocks: the timestamp carried by the data and the timestamp at which the stage last processed it. Compare both across the pipeline.

Errors tell you that something complained. Freshness tells you whether the system is still telling the truth.