2 Issues: Evening of May 8

We've been working on a number of infrastructure changes behind-the-scenes of Help Scout to further increase redundancy, so that your account works as expected no matter what. Last night we had a couple of growing pains and need to share the details:

Issue #1 + How We're Fixing it

Last week we brought a new mailserver online to handle more volume and give us additional redundancy in the case of a failure somewhere. We now have two mailservers running with two different hosting providers so that there's never a single point of failure.

Last night, between 7:30pm - 12:30am the new mailserver ran into some problems. The server thought it was out of space when it actually had plenty available. During this window, sporadic emails forwarded to Help Scout were bounced. Some emails were processed, others were bounced and we don't really know what made the difference.

Since nothing was actually "down" during this time, none of our monitoring alerted us to the problem. That's why it took longer to find and fix the issue. Our development team has already come up with a solution for storing messages off should this ever happen again, avoiding any bounces.

Issue #2 + How We're Fixing it

From 10:00-10:15pm last night, we had a scheduled maintenance window for outgoing email. Any emails sent from Help Scout were put on hold while we did the maintenance, while everything else operated as normal. 

For a period of less than 10 minutes, incoming emails could have possibly bounced. Our hosting provider had to clear a cache we were unaware of, which caused a slight delay. This particular maintenance most likely won't happen again. It's part of the learning process in dealing with different cloud providers and understanding how they operate. Should we ever need to perform this maintenance again, we'll know the cache on their end must be cleared.

Oddly enough, these two issues were not related even though they happened in the same evening. We sincerely apologize for the inconvenience caused. Another ironic twist is that this maintenance is all designed to bring more stability to Help Scout in the long term and we still believe it will.

A great deal of work is being done on our end to continue scaling the Help Scout infrastructure and making sure email is always stored and processed even if everything else goes down. Last night was a blip we're confident will not happen again. Staying up is and always be the most critical priority. 

Have any specific questions? You can email us here.

2 Issues: Evening of May 8

We've been working on a number of infrastructure changes behind-the-scenes of Help Scout to further increase redundancy, so that your account works as expected no matter what. Last night we had a couple of growing pains and need to share the details:

Issue #1 + How We're Fixing it

Last week we brought a new mailserver online to handle more volume and give us additional redundancy in the case of a failure somewhere. We now have two mailservers running with two different hosting providers so that there's never a single point of failure.

Last night, between 7:30pm - 12:30am the new mailserver ran into some problems. The server thought it was out of space when it actually had plenty available. During this window, sporadic emails forwarded to Help Scout were bounced. Some emails were processed, others were bounced and we don't really know what made the difference.

Since nothing was actually "down" during this time, none of our monitoring alerted us to the problem. That's why it took longer to find and fix the issue. Our development team has already come up with a solution for storing messages off should this ever happen again, avoiding any bounces.

Issue #2 + How We're Fixing it

From 10:00-10:15pm last night, we had a scheduled maintenance window for outgoing email. Any emails sent from Help Scout were put on hold while we did the maintenance, while everything else operated as normal. 

For a period of less than 10 minutes, incoming emails could have possibly bounced. Our hosting provider had to clear a cache we were unaware of, which caused a slight delay. This particular maintenance most likely won't happen again. It's part of the learning process in dealing with different cloud providers and understanding how they operate. Should we ever need to perform this maintenance again, we'll know the cache on their end must be cleared.

Oddly enough, these two issues were not related even though they happened in the same evening. We sincerely apologize for the inconvenience caused. Another ironic twist is that this maintenance is all designed to bring more stability to Help Scout in the long term and we still believe it will.

A great deal of work is being done on our end to continue scaling the Help Scout infrastructure and making sure email is always stored and processed even if everything else goes down. Last night was a blip we're confident will not happen again. Staying up is and always be the most critical priority. 

Have any specific questions? You can email us here.

Short Scheduled Outgoing Email Maintenance this Evening

Tonight at 10pm EST, outgoing email will be offline for about 15 minutes while we make some changes. Emails sent during this time may be delayed, but everything else will continue working as normal. Anything out of the ordinary will be posted as an update here ... otherwise you can assume the changes went according to plan.

Help Scout Scheduled Downtime to Launch Traffic Cop

This is one of those status blog posts we like to write. We're taking Help Scout offline tonight from 10-10:30pm EST so we can launch a new feature called Traffic Cop (more info here- http://hlp.sc/HkivDE).

All emails will continue to be processed. Only the web app will be offline while we make a number of database changes. We will post an update here when finished.

Small Help Scout Outage Today

Today Help Scout was down for 3-5 minutes at about 1:45pm EST. We were doing a database export and temporarily ran out of storage on our server. We've already changed how we do database exports to prevent this from happening in the future. Sorry for the issue, folks!

Feed My Inbox: Twitter Integration Temporarily Broken

The Feed My Inbox integration with Twitter feeds was broken for roughly 12 hours over the last day. The issue has been resolved and we're back on track. So sorry for the inconvenience!

Help Scout Outage this Morning

This morning between 2:22am and 2:43am PST, Help Scout was down. It was a result of a brief Amazon Web Services (our cloud provider) outage. Below is the message that was posted on the AWS status blog:

Screen20shot202012-03-1520at2010

Our apologies for the inconvenience.

Scheduled downtime for tonight at 10pm EST

Tonight (10pm EST) we have to take down the Help Scout web app for 5-15 minutes to deploy several updates. Our incoming and outgoing queues will continue to function normally, so no emails will be lost.

Take a break to enjoy some time with your valentine and we'll be back before you finish your second heart-shaped chocolate. :-)

Outgoing Email Queue Delayed

Yesterday morning we ran into some rogue emails that temporarily clogged our outgoing email queue. No data was lost, only delayed for up to a couple of hours (some weren't delayed at all).

So sorry for the interruption! Tonight we're deploying a fix to prevent this moving forward.

Help Scout - Slow Email Delivery

Help Scout has been experiencing sporadic delays with incoming email delivery. It's been hard to diagnose as some cases were caused by external email servers getting the email to us in time, while other cases may have in fact been caused by our own mail server.  Therefore, tonight we will be making changes to our mail server to address the issues we can from our side.  

Starting at 9pm CST/10pm EST, the mail server will go offline for approximately 10 minutes (or less) so that we can make some configuration changes. Emails will most likely bounce during this time but we will work as quickly as possible to prevent the loss of any emails.

We are so sorry for the inconvenience, but once this change is made, we feel our mail server will no longer play a role in the random slow deliveries.

UPDATE: 10:20 pm EST -

The mail server changes were executed this evening and it was a success.  We were able to do everything ahead of time, so the originally planned offline time of 10 minutes was shortened to 15 seconds (long enough for us to perform a server restart).  No emails should have been lost during this short window. 

Once again, we feel this change will solve the email slowness from the Help Scout side.  Please continue to watch for slow deliveries, and if you spot any, don't hesitate to let us know.