So, who do we yell at?

It has been a very trying afternoon.  We appreciate your patience with today's large scale network outage.

So, what happened?  The truth is, the details are still coming in and as yet we don't know the root cause, but we do know the results.

We first noticed that connections to the server were disrupted at 12:52pm EST.

 

Everything at the data center was online, there were no hardware or software issues with any of the servers or networking hardware, though, the fiber coming into the building went totally dark.

 

Calls to the fiber providers and other providers in the Chicago area (where our data center is located) reported that a very large fiber interconnect on the Southside of Chicago, one of the absolute worst place to have a problem, had a problem.

 

We have yet to be notified of what the exact nature of the problem was, either a hardware outage at the provider’s level, a physical disruption in the fiber lines, or what.

 

We do know that we were not the only data center affected by the outage, and that also a number of hospitals in the area relying on the same fiber lines were also completely down.

 

We are currently awaiting a report from the providers of the fiber lines as to what was the cause of the outage.

 

Connectivity was restored at approximately 3:39pm EST, so, ultimately, we were down for less than 3 hour(s), and we are continuing to monitor things on our end (though, again, there was no issue with your data or your servers).

 

We advertise 99.9% uptime, and across the 8760 hour(s) in a year, we are still keeping good on that promise.  We continue to review our policies and procedures and look for areas of improvement.  We do have the resources to bring another data center online and with offsite backups, we have the data to do so. However, in this instance, the time to update DNS records to point at the second data center would have taken more time to communicate, make, and then filter out than it took for the fiber providers to restore the damaged connections.

 

Though, had the outage persisted into the evening hours, we would have put this plan into action to minimize downtime to our customers.

 

Again, we will be following up with additional information as to the exact cause of the fiber outage in Chicago as soon as we receive the information ourselves, and we look forward to continuing to service you for all of your hosting needs, and if you ever have any questions about anything, please let us know.

Sincerely,

Brad Watson

President 

Halo Applications