Follow-up: Delayed Incoming Emails

• Edit
• Delete
• Tags
• Autopost

It's standard operating procedure when we have blips or downtime that we look at what happened and how we can get better. Early this week, our incoming email queue was not processing emails for several hours. 

What Happened

This week, Amazon Web Services went about rebooting their cloud servers as part of scheduled maintenance. After the reboot on Monday, our incoming message queue stopped communicating with the server. Between roughly 3am EST - 11am EST, all emails were being received by Help Scout, but not processed in the database.

There was no data lost during this window and Help Scout was never down. It just took a while for us to get incoming emails processed and everything in sync again. 

How We're Getting Better

When Help Scout launched back in April, there wasn't a "high availability" version of the message queue we use, meaning it could be a single point of failure. More recently, a high availability version of the software has entered beta and we're going to start testing it. 

Assuming we can move this new version of the message queue into production eventually, this blip would be completely avoided in the future. 

Any additional questions or concerns can be addressed to us via email. Thank you!