Back to Blog

AI

The Emails That Said They Were Sent

By Khai Tran · · 2 min read

For two weeks, the emails my website sends were failing, and four different parts of the system said they went out. One quiet log had the truth.

Featured image for The Emails That Said They Were Sent

Week ending July 24. Written up from the build log.

For fourteen days, the emails my website sends were failing. Four different parts of the system reported them as sent.

The newsletter logged a "sent" for each subscriber. The purchase side marked receipt emails as sent, in three separate places. A set of test purchases we ran while switching payment systems all came back green. Meanwhile, since July 5, the email provider had been rejecting every message.

One log did have the truth. A newsletter run on July 13 recorded zero subscribers reached, and it sat beside twelve rows that said "sent."

We found it on Sunday. The cause was one wrong setting, and by early afternoon mail was flowing again, confirmed in the provider's own delivery log instead of ours. The harder question was why nothing had noticed for two weeks.

Each piece was reporting on itself. The newsletter wrote down "sent" before it even tried to send. The purchase code treated any answer from the provider as a yes, including a no. So a broken pipeline looked exactly like a passing test.

It wasn't the only one that weekend. Scheduled jobs on another of my sites had been failing since early June. A reminder system had been marking people as reminded when nothing went out. Red's log summed them up in one line: a write whose failure nobody checks. The parts of the system that did catch problems were the ones that checked their own results.

The rule we wrote that day is short.

Never record a delivery you haven't observed.