Engineering blog
The 4 AM Bug: How a Wrong Mental Model Wasted Three Days
I was so sure I knew what was wrong that I spent three days debugging the wrong thing. Here's what finally fixed it — and what I learned about diagnosing before solving.
The 4 AM Bug: How a Wrong Mental Model Wasted Three Days
It was 4 AM on a Thursday. I had a leave request going through PeoplePulse's approval workflow and the manager's inbox simply wasn't updating. The frontend was getting a 200 OK. The database write was confirming. But the UI showed nothing.
I was certain it was a WebSocket reconnection issue. So I spent three days on WebSocket reconnection.
It was not a WebSocket reconnection issue.
What I actually missed
The approval endpoint was writing to the database correctly. But I'd added optimistic UI updates in an earlier sprint, and those optimistic updates were patching state in a way that bypassed the usual refetch. The state after the 200 OK was already populated with the pre-approval version of the leave request — and the component had no reason to refetch because it thought it had fresh data.
The WebSocket was working fine the whole time. I just never got to see its updates because React state was being patched before the socket event arrived.
The lesson I keep having to relearn
When something breaks, most engineers — myself included — immediately start narrowing down to the most *interesting* possible cause. WebSockets are interesting. Optimistic UI state patches are boring. So the brain skips past the boring answer and starts digging tunnels.
The fix I now force myself to use: before touching any code, write down exactly what data you expect to exist at each layer (network, database, in-memory state, rendered UI) and then actually check each one. Not guess — check. Open the Redux DevTools. Log the React state. Look at the actual DB row.
In this case, 20 minutes of proper state inspection would have replaced three days of incorrect assumptions.
What changed in the codebase after this
After fixing the actual bug, I went through the entire optimistic update pattern and added explicit invalidation logic so that server responses always win over local state patches. The pattern is cleaner now and hasn't caused a recurrence.
The WebSocket code is also better as a side effect — but that's incidental.
The debugging process is embarrassing in retrospect. I'm writing it down because I'll probably do it again in six months, and maybe reading this first will save me a day.