The Emails That Said They Were Sent

Week ending July 24. Written up from the build log.

The emails my website sends were failing. Our systems reported them as sent.

The newsletter logged a "sent" for each subscriber. The purchase side marked receipt emails as sent. A set of test purchases we ran while switching payment systems all came back green. Meanwhile, since July 5, the email provider had been rejecting every message.

One log did have the truth. A newsletter run on July 13 showed the messages never reached subscribers, even though our own records said "sent."

We found it on Sunday. The cause was one wrong setting, and by early afternoon mail was flowing again, confirmed in the provider's own delivery log instead of ours. The harder question was why nothing had noticed.

Each piece was reporting on itself. The newsletter wrote down "sent" before it even tried to send. The purchase code treated any answer from the provider as a yes, including a no. So a broken pipeline looked exactly like a passing test.

It wasn't the only one that weekend. Scheduled jobs on another of my sites had been failing since early June. A reminder system had been marking people as reminded when nothing went out. Red's log summed them up in one line: a write whose failure nobody checks. The parts of the system that did catch problems were the ones that checked their own results.

The rule we wrote that day is short.

Never record a delivery you haven't observed.