Two Integrations That Had Quietly Stopped Are Working Again
Two of the sources behind press mentions had stopped returning results. Both are working again. This is a maintenance note, and it is public because the interesting part is why nobody noticed.
What happened
Both sources changed the shape of what they return. Neither returned an error. One started delivering its results inside a different structure, and one began serving a challenge page in place of data.
Our code asked for a list, received something that was not a list, found nothing in it, and reported zero results. Zero results is a completely normal outcome for a scan, which is exactly why it went unremarked for weeks.
The general shape of this bug
A silent failure that produces a plausible answer is the worst kind, because monitoring that watches for errors sees nothing wrong. Everything succeeded. It simply succeeded at finding nothing, forever.
The same category of fault produced the SoundCloud problem written up in longer SoundCloud sessions, where a truncated value looked valid until it was used.
What we changed beyond the fix
- Every source is checked against a known query that must return results. Zero is now treated as a fault rather than an answer.
- The response shape is validated before it is parsed, so a changed structure raises an error instead of yielding an empty list.
- Per-source health is visible internally, with a last-successful timestamp, so a source going quiet is obvious rather than invisible.
- An alert fires if any source returns nothing for longer than its normal quiet period.
What this means for you
If you had mentions enabled during the affected period, the backfill has run and anything missed should now be in your panel. Some of it will be several months old, which is not ideal but is better than not having it. You can add anything worth keeping to your press page in one click, as covered in turn a mention into a press item.
We would rather write this up than not. A tool that quietly stops working is a tool you should be told about.
Related: logging and alerting on broken links, which came out of the same thinking.
Create your free actor website and switch mentions on.
