RESOLVED: This maintenance is complete.
UPDATE: This has been rescheduled for February 9, 2017 at the same time (300 – 0700 UTC Feb 10th)
We will perform maintenance on our Subversion infrastructure on Wednesday, February 8, 2017 between 8PM and midnight Pacific time (0300 – 0700 UTC Feb 9th). During this 4 hour window, there will be about 30 minutes where no commits or deploys will be possible.
As you may have read, today we enabled HTTPS support for all domains on WordPress.com. We feel very strongly that encryption is quickly becoming an expected (and soon required) part of the web browsing experience and as such, are enabling HTTP to HTTPS redirects for the over one million custom domains on WordPress.com.
We have excluded VIP sites from this initial deployment for a few reasons:
If you have already established HTTPS for your VIP site (on either WordPress.com VIP or VIP Go), no action is needed on your part. Note that HTTPS is enabled for all primary domains on VIP Go as a part of the site setup process.
If you would like to enable HTTPS support for your WordPress.com VIP site using one of the above options, please open a support ticket and include the following information:
If you have any questions about HTTPS support on VIP, please contact us.
Earlier today, Google researchers disclosed a new vulnerability in the relatively old SSLv3 protocol. To protect WordPress.com visitors, we disabled SSLv3 support across all of WordPress.com (effective 2014-10-14 12:20 AM UTC). A very small percentage, somewhere less than 0.1% of WordPress.com total traffic was previously transmitted over SSLv3. Most of that traffic came from users using Internet Explorer 6 on Windows XP. In order to access any WordPress.com site over SSL, those users will need to either upgrade to and use a modern browser or a modern operating system. Of course, any unencrypted traffic (over HTTP) is unaffected by these changes. We are also looking into the possibility of mitigating the vulnerability and re-enabling SSLv3 support using Fallback SCSV, a solution proposed by Google researchers.
Our new data center is almost complete and it has been serving production traffic for the past two weeks. Initial data shows that it is about 20% faster than our existing facilities, which can be attributed to state-of-the-art networking and server hardware as well as some design and architectural changes.
If you are using the WordPress.com DNS servers or your site is a CNAME to WordPress.com, no action is required. IP changes are only required if you are not using our name servers.
To find out which name servers your site is using, you can run this in Terminal on OS X, replacing $DOMAIN with your actual domain
dig +short NS $DOMAIN
On Windows, you will want to use this command:
nslookup -type=NS $DOMAIN
If you see any mention of WordPress.com in the result, then there is no need to do anything – we have already made the changes for you. If you don’t see WordPress.com mentioned, you are using 3rd party name servers and must make the following changes before Oct 31st:
On October 31st, we will decommission the old facility and if you still have references to the old IP addresses in your DNS zone file, your site may become unavailable.
We highly recommend that if you are hosting on WordPress.com VIP that you use our DNS services. They offer the best balance of performance, flexibility, and availability. If you want to switch your domain to use our name servers and need help or have any questions about the required IP changes, please get in touch.
P.S. Those photos are from the actual data center.
As we mentioned earlier this week, WordPress.com experienced a partial outage and service degregation when one of our three data centers was taken completely offline by a fiber cut. I wanted to provide some more information about how this occurred, what the impact was, and what we are doing to prevent this from happening in the future.
BACKGROUND
WordPress.com is served in real-time out of 3 data centers.
Currently, we use DNS to distribute traffic between locations and have a few ways we manage these DNS entries during an outage. Most of WordPress.com uses wildcard DNS (*.wordpress.com), some other records have systems in place which automatically remove the DNS record of any location which is not responding, and other records are managed semi-automatically, meaning a human has to run the commands to remove the affected IP addresses. All of our DNS records have a low TTL to ensure that clients receive the most recent records.
Additionally, for domains hosted on WordPress.com, we return multiple IP addresses. This allows us to take advantage of DNS failover, sometimes called client retry. Modern browsers will automatically try a different IP address. Relying on browsers to behave in a certain way is not ideal, and thus we only rely on this to work in certain cases which a broken IP is not removed from DNS as well as during the time between when our systems remove the broken addressees and clients refresh their list.
TIMELINE
February 12th 2013, all times UTC –
0600 – Complete loss of connectivity to San Antonio datacenter.
0605 – San Antonio datacenter removed from production for 90% of traffic (everything but the semi automatically DNS records mentioned above).
0605 – 0700 – Troubleshooting network connectivity, trying to determine ETA when service will be restored.
0700 – No ETA for service restoration, decision made to remove San Antonio IP addressees from VIP DNS records.
0700 – 0742 – Testing and verification of DNS updates.
0745 – DNS updates for all VIPs complete.
0930 – Connectivity restored to San Antonio (one link back online).
1113 – Redundant connectivity restored.
1130 – DNS changes reverted, traffic back to San Antonio.
CAUSE
Our San Antonio facility is connected to the Internet via two fiber links which go between San Antonio and Dallas. From Dallas, traffic is sent to the rest of the world. To ensure that links were redundant, our data center provider contracted with two different companies and verified that the fiber ran along different physical routes. Unfortunately, in the past 24 months, both companies were acquired and without our knowledge, the lines were relocated into the same physical fiber bundle. We are still not 100% sure when this change was made, but we think it was in the past few months. A maintenance by the fiber owner on February 12th caused both circuits to be taken offline and a complete loss of network connectivity to the data center. Unfortunately, the maintenance notification never made it to our provider or to us, otherwise we would have taken proactive steps to remove the data center from production.
IMPACT
For subdomains like photos.digitalize.ca, which use a DNS CNAME record to point to WordPress.com, things returned to normal after 5-10 minutes. For top level domains, like raanan.com, our stats show about 5% of traffic was impacted between 0600 and 0745. No data was lost.
REMEDIATION
Our provider has been in the process of replacing the existing fiber links with new ones. The end result will be a redundant circuit ring through one provider and separate redundant circuit through another. We are taking the appropriate precautions to ensure that these new circuits will run on completely separate paths. We are also going to obtain confirmation from both fiber providers that there will not be any work on our circuits for the next 8 weeks. This is the ETA for when the new circuits will be fully operational.
We are exploring the possibility of bringing an additional network provider into our San Antonio facility. This would mean even if both redundant fiber connections to Dallas were impacted, the location would still be able to communicate with the Internet.
We have some tentative plans to switch away from DNS-level distribution between locations and instead use a network routing topology called anycast. This has the advantage of providing faster failover and less manual intervention will be required. Our current anycast implementation was tested during this outage and worked as expected. Until this is complete, we will be working on some things to make the current failover faster. In general DNS updates would be made much faster, but unfortunately, our scripts required some re-verification after our recent data center migration in December/January. We wanted to make sure we didn’t break anything more severely during the DNS update process.
For those not using our DNS servers, it impacts our ability to protect your sites from exposure to outages like this. We urge you to use our DNS servers if at all possible.
We apologize for the service disruption, and as always, use these opportunities to make WordPress.com VIP the best we can.
Barry Abrahamson
Systems and Infrastructure Engineering
We have created a new robot – his name is optimizerbot and he will look through your themes daily to find any PNG or JPEG images that can be losslessly compressed and automatically commit the compressed version to your theme’s subversion repository. We have found that about 80% of existing images in VIP themes can be compressed this way, saving precious bytes and providing a better user experience, especially on slower networks, like 3G.
Here is an example of what the commit message will look like
The first column is the file name, second column is the old file size in bytes, third column is the new file size in bytes, and the last column is the % saved.
Technical details for those who are interested:
|| separator. This creates a table layout in Trac, which we use internally for reviewing changes in some cases.Enjoy and please leave your feedback/questions in the comments!
UPDATE: This maintenance is now complete.
This evening between 9:59 PM and 11:59 PM Pacific we will be performing some network maintenance in our Dallas data center. This maintenance will necessitate disabling publishing only for some sites for 10-15 minutes during the 2 hour window. Not all sites will be affected. We will have extra personnel available during the maintenance to deal with any issues that may arise. If you have any questions, please feel free to contact your VIP Hosting Support team.
UPDATE: This maintenance was completed as scheduled.
This evening, beginning at 9PM Pacific time, we will be conducting network maintenance in our San Antonio, Texas data center. During this maintenance we will replace some legacy 1 Gigabit network equipment with new, 10 Gigabit, switches and routers. Prior to the maintenance, we will route most traffic out of San Antonio to limit the impact of any outage that might result. The maintenance is scheduled to end at 3AM Pacific. I will update this post once the maintenance is complete. While we don’t expect any noticeable interruption, as with any maintenance there is a risk of things going wrong. We will have additional staff on hand during the maintenance window to quickly deal with any issues that may arise. If you have any questions, please feel free to contact your VIP Hosting Support team.
As posted on our public status page we are currently investigating some network issues affecting our Dallas datacenter. This could cause some slow page loads or timeouts for some users. Our graphs show about 20% of the traffic in Dallas is affected. While we investigate, we have moved most VIP traffic out of Dallas to lessen the impact for those who might be affected. As soon as we have more information, we will let you know.
Just a quick note to let you know that we have some scheduled network maintenance in our San Antonio, Texas data center between midnight and 6AM Eastern time on Thursday, April 14th. During this maintenance, our hosting partner will be replacing some network switches with new ones capable of 10Gbit connectivity (that’s a lot). Prior to the maintenance window, we will move all traffic out of this data center, so there should be no disruption to your sites. As with any maintenance, things may not go exactly as planned, so we will have extra personnel available to deal with any issues that may arise.
Please let us know if you have any questions by emailing us
WordPress VIP Lobby 
You must be logged in to post a comment.