6 Minute Site Wide Outage Today, Sept 6

There was a six minute outage of all of WordPress.com about an hour ago, starting at approximately 16:19 EDT till 16:25 EDT today, Sept 6th, 2011.

With the front end caching some visitors would not have experienced it.

It appears to have been a programming error introduced into our lower level MySQL database interaction code, which explains why the symptom was a blank page.

status.automattic.com now shows:

At 4:19 EDT this afternoon, one of our developers accidentally deployed some code to WordPress.com that caused pages to stop loading and instead return a 500 error. This changed was reverted at 4:25 and everything returned to normal. Sorry for the inconvenience!

Technical operations is investigating with the specific programming group to see where our processes and measures broke down.

Texas Power Issues

Below is the email sent out by our Operations Lead Barry Abrahamson to the emergency vip-notify email list 9:15am Pacific Time, and the update that he just sent out.

All WordPress.com VIP sites are distributed and load balanced across 3 data centers.

If you or any of your colleagues did not receive the vip-notify email, please see “Emergency VIP Notification” to subscribe.

As some of you may have heard, starting this morning (Central Standard Time), ERCOT, the operator of the power grid in Texas has ordered utility companies to implement rolling blackouts across the state. 2 of the 3 WordPress.com data centers are located in Texas – 1 in San Antonio and 1 in Dallas. The Dallas facility has been operating on generator power since yesterday evening without issue. This morning, the generator in the San Antonio facility suffered a failure that has left us without a long-term backup power supply in that facility. There have been 2 utility power blackouts in the San Antonio facility this morning, but so far UPS power has kept the servers running. Technicians are en route to repair the broken generator, but it is possible that there may be another blackout before the repair is complete. If that happens, it is very likely that the 350 servers running WordPress.com in San Antonio will power off. We are currently preparing for that scenario and do
ing everything we can to reduce the impact of such a failure, but wanted to let everyone know the current situation. Here is the link to Reuters story about the blackouts: http://www.reuters.com/article/2011/02/02/ercot-rollingblackots-idUSN0221428420110202 Please feel free to contact us at vip-support@wordpress.com if you have any questions.

And here is the encouraging update sent out a few minutes ago by Barry:

A quick update on the power issues. The generator issues in our San Antonio facility have been mostly resolved. Technicians are still on-site, but we hope to have the “all clear” from them shortly. The facility is currently operating on utility power, but our hosting partner in San Antonio is confident that in the case of another utility power blackout, the transfer to generator power can be made successfully. Once the generator issues are fully resolved, a controlled failover to generator power will be initiated. We will continue to run the facility on generator power until the utility company lets us know the blackouts are over at which point we will switch back to utility power. We don’t expect the blackouts to end until tomorrow at the earliest.

ERCOT has now published information on their own site.

Feb 5th update from Barry:

The power issues in Texas have stabilized. There were no performance or service interruptions as a result of the numerous blackouts in our San Antonio facility. As someone living in Houston, I hope it warms up soon!

Winter Holidays 2010

Hello VIP Partners,

For the last two weeks of December our systems, engineering, and support coverage will be light as people take some time to relax with families and friends. If you are planning any major deployments during the last weeks of this year, please share your schedules with us now.

Like previous years, we will close shop for some winter holidays around Dec 25th, and let our fancy automated systems call us if there are any severe issues. There will be no code reviews or deploys from Thursday, Dec 23rd through Monday, Dec 27th. We’ll review and deploy all SVN commits Tuesday, Dec 28th. If during those days you need an emergency deploy due to an existing code issue, please include “[urgent]” in the subject line to vip-support@wordpress.com .

The week of January 17th through January 21st the VIP Team will be coming together in London. This will also take us away from the queue to work on making our service all around better. Here as well it’d help us if urgent matters were flagged during that week.

Thank you for being a big part of a great 2010 for us! We’re excited for what is ahead in 2011.

Season’s best to you,
Your VIP Team

WordPress 3.1 Deploying Backend

We’ve been working towards merging much of the backend changes of WordPress 3.1 into WordPress.com. On the current trajectory this will land during the first part of next week.

This release is fairly incremental over 3.0, which was a heftier release. The technical overview can be found at http://codex.wordpress.org/Version_3.1 and what looks to be a more general overview at http://www.wpbeginner.com/news/whats-coming-in-wordpress-3-1-features/.

If you are using, or working on using, taxonomies, this area has refinement and improvements in this release.

The 3.1 release will soon be going beta, so now is a good time to update your development environments to trunk.
http://codex.wordpress.org/Installing/Updating_WordPress_with_Subversion

Also, be sure to subscribe to http://wpdevel.wordpress.com/ for the core development play-by-play.

Upgraded to TinyMCE 3.3.9

About 24 hours ago, the WordPress.com team updated TinyMCE to version 3.3.9.2 to resolve issues with Internet Explorer 9 Beta.

The TinyMCE editor is what powers the Visual Editor for Posts and Pages (the text area where I’m writing this post).

This version of TinyMCE is the same one that is now in WordPress.org trunk. The previous version was 3.2.7.