Tonight (10pm EST) we have to take down the Help Scout web app for 5-15 minutes to deploy several updates. Our incoming and outgoing queues will continue to function normally, so no emails will be lost.
Take a break to enjoy some time with your valentine and we'll be back before you finish your second heart-shaped chocolate. :-)
Yesterday morning we ran into some rogue emails that temporarily clogged our outgoing email queue. No data was lost, only delayed for up to a couple of hours (some weren't delayed at all).
So sorry for the interruption! Tonight we're deploying a fix to prevent this moving forward.
Help Scout has been experiencing sporadic delays with incoming email delivery. It's been hard to diagnose as some cases were caused by external email servers getting the email to us in time, while other cases may have in fact been caused by our own mail server. Therefore, tonight we will be making changes to our mail server to address the issues we can from our side.
Starting at 9pm CST/10pm EST, the mail server will go offline for approximately 10 minutes (or less) so that we can make some configuration changes. Emails will most likely bounce during this time but we will work as quickly as possible to prevent the loss of any emails.We are so sorry for the inconvenience, but once this change is made, we feel our mail server will no longer play a role in the random slow deliveries.UPDATE: 10:20 pm EST -
The mail server changes were executed this evening and it was a success. We were able to do everything ahead of time, so the originally planned offline time of 10 minutes was shortened to 15 seconds (long enough for us to perform a server restart). No emails should have been lost during this short window.
Once again, we feel this change will solve the email slowness from the Help Scout side. Please continue to watch for slow deliveries, and if you spot any, don't hesitate to let us know.It's standard operating procedure when we have blips or downtime that we look at what happened and how we can get better. Early this week, our incoming email queue was not processing emails for several hours.
What Happened
This week, Amazon Web Services went about rebooting their cloud servers as part of scheduled maintenance. After the reboot on Monday, our incoming message queue stopped communicating with the server. Between roughly 3am EST - 11am EST, all emails were being received by Help Scout, but not processed in the database.
There was no data lost during this window and Help Scout was never down. It just took a while for us to get incoming emails processed and everything in sync again.
How We're Getting Better
When Help Scout launched back in April, there wasn't a "high availability" version of the message queue we use, meaning it could be a single point of failure. More recently, a high availability version of the software has entered beta and we're going to start testing it.
Assuming we can move this new version of the message queue into production eventually, this blip would be completely avoided in the future.
Any additional questions or concerns can be addressed to us via email. Thank you!
Incoming email processing has been down for a couple of hours and we're working on restoring it. No messages have been lost, they just aren't showing up in Help Scout yet because the message queue is down.
The rest of Help Scout continues to operate normally, we'll update this post as soon as incoming mail is restored. So sorry for the problem!
Update @ 11:30am EST: All emails are now processing and will be loaded in your accounts soon. So sorry for the disruption!
At some point between 2:00-2:30 am EST time on Tuesday, December 13, Amazon Web Services will be performing maintenance on our database instances. This could impact availability of Help Scout for a short time. Thank you!
Amazon Web Services (AWS), where most of Help Scout is hosted, announced upcoming maintenance that will impact our services at some point between August 1-7, from 3-3:30am EST. We don't have an exact date from Amazon, but the upgrade will take place during that 30-minute window. Availability should only be effected for a few minutes.
For more on the work being done, here's a link to the official announcement from Amazon.
This morning around 6:24am EST, FeedMyInbox.com, Brightwurks.com and our mail server were experiencing issues for less than an hour.
The issues were due to an outage with one of our hosting partners, Voxel.net. We've been happy customers for several years and Voxel's network is extremely reliable, but something went wrong this morning. Once we hear from them about what happened, we'll update this post.
As with every outage on any of our sites, we're working on how we can handle this better in the future and not interrupt our services. So sorry for the inconvenience.
On Friday, July 15 from 8-9pm EST we will be performing scheduled maintenance on Help Scout.
The application and marketing site will continue running normally. However, no incoming emails will be processed and no outgoing emails will be sent during the one-hour window. Everything you send from Help Scout during the window will be sent once the maintenance is completed. Thanks everyone!
This afternoon we are doing some maintenance on Feed My Inbox. Only emails will be offline. The website will continue working properly.
The maintenance is expected to take 3 hours or less. We won't be sending emails during this time, but once we're back up all posts during the maintenance will be sent. No data will be lost.
Hit us up at help (at) feedmyinbox.com if you have any questions.
