We’re experiencing a service outage due to a widespread power failure in Vancouver
July 14th, 2008 10:20 AM by TobyWe are experiencing a service outage due to a power failure at the data center which hosts our web servers.
We expect service to be fully restored once the power issue has been resolved, and will continue to post updates to this website as they become available.
Update 11:38 AM PDT: A city-wide power-failure in Vancouver has disrupted the operation of our web servers.
Update 12:25 PM PDT: We have been notified that power should be restored to our data center by 3:00 PM PDT. Read coverage on CBC.ca and NowPublic.
Update 2:15 PM PDT: The following update is from Peer1, the operator of the facility where our servers are hosted:
“Our Generator is back online and we are testing it for stability for the next 15 minutes. We’ll know more in 20 minutes and if all goes well, we’ll fire up the UPSs and start powering server again.
This is the update as of 2:10 PST from our datacenter engineer.”
Thank you for your patience while this is resolved. If the generator does not remain stable during the Peer1 testing period, service might not be restored by 3:00 PM PDT as we had hoped. We will post another update by 3:05 PM PDT.
Update 2:41 PM PDT: Peer1 has provided us with the following updates:
“The genset continues to perform as expected. We are going to let it run for another 15 minutes and then power up the UPSs and customer servers. All testing is going as planned so far.
With any luck we’ll have you back online within the next 30 minutes.”
and
“Further details on why the genset failed.
The genset cooling is dependant on water cooling provided by the city of Vancouver. However, the water pressure was severely diminished when the fire broke out nearby earlier this morning. This is what was preventing us from keeping the genset cool earlier. As of now, the water pressure is much more stabilized, which means the genset is working as expected.
Again, we will be putting full load on the genset in the next few minutes. Now some of this is starting to make more sense. More details to come.
Thank you”
Update 3:15 PM PDT: Peer1 has indicated that the generator is operational but not yet as stable as they need it to be in order to restore service. More information will be posted here as soon as it is available.
Update 4:10 PM PDT: Peer1 has provided the following update:
We have begun powering up servers in our Harbor Centre datacenter. Please stand by and/or begin testing your servers for reliability.
Hushmail.com is now online and we’re in the process of testing and restoring all other services.