Blackout created big problems
Schmalkalden, published on August 18, 2010 at 01:25 by Markus.
A large-area blackout in the whole centre of Erfurt has created big problems for us as well on Monday, 16th August. A defective UPS-equipment in the data center of Keyweb AG was blamed for the fact that all our servers weren’t available.
The area of the inner city of Erfurt was suddenly without electricity on Monday 1.18 p.m.. Lights died down, tramway stood still and all servers of Erfurt’s data center went out shortly after the breakdown of the electricity supply. Construction works - which apparently went wrong - were the reason for the total failure.
The emergency current system of the third data center of Keyweb AG - where all our servers are situated - couldn’t come into action due to a defect of one of two available UPS-equipments. A total failure for hours resulted from that. The mains adapter of our main server was affected through the abrupt stop of the machines. It could be repared not until the late afternoon of 17th August because of the amount of servers which were affected by damages. A backup server limitedly assumed the services since 10.10 p.m..
The interconnection of our data base servers was interrupted through an unfortunate circumstance. That led to the fact that the data status of both master servers dispersed (inf. Split Brain). The texture of data wasn’t warranted anymore. The situation couldn’t promptly be recognized because of the monitoring server which failed due to the damage. The repair of the replication was very time-consuming. It has proved to be very difficult until some minutes ago. A self-developped tool will identify the data status in the following hours and - if necessary - adjust once more.
We will promptly inform about the current status of our equipment as well as about general information around picload.org through our Twitter account. It’s worth following us through Twitter.
We’d like to make a formal apology for the occured trouble and we hope you have understood the failure for hours.
So, that’s from us and the blackout.
With kind regards,
Markus & Tim