Speeding up deployments with preprods

This notice relates to the following platforms: WordPress.com VIP

A number of you use preprod environments to test your code on a VIP environment before pushing to production.

For a little while now we’ve been running an experiment with selected repositories where we can quickly check for differences between a production and pre-production repository, and deploy production changes without review if preprod has already been reviewed.

Let’s look at an example. Say you have your two repositories:

  • vip-enterprise – the production repository
  • vip-enterprise-preprod – the pre-production repository

You commit changes to vip-enterprise-preprod where they drop into the Deploy Queue and are reviewed by our engineers, and deployed after any feedback is addressed.

Following deployment, your team runs any appropriate tests to ensure the code is behaving as expected in pre-production.

Once you’re satisfied,  you commit those same changes to vip-enterprise and they drop into our Deploy Queue. At this point, an engineer needs to ensure those changes are safe to deploy, which means reviewing the code – a highly inefficient practice given the code was already reviewed in vip-enterprise-preprod.

Historically, we didn’t have a good way to avoid that inefficiency but our recent experiment has changed that. Now, our engineers can run a simple command:

$ diffpreprod vip-enterprise
Updating 'wp-content/themes/vip/vip-enterprise':
U wp-content/themes/vip/vip-enterprise/functions.php
Updated to revision 330618.
Updating 'wp-content/themes/vip/vip-enterprise-preprod':
U wp-content/themes/vip/vip-enterprise-preprod/style.css
Updated to revision 329133.
vip-enterprise and vip-enterprise-preprod match!

Good to deploy vip-enterprise

In less than a minute, the engineer has seen r330618 drop in, ran diffpreprod and found it safe to deploy.

We’re running this for select clients already, and will start rolling it out to all preprods soon, so if you want to take advantage of these speedier deployments, please check that your production and pre-production environments are in sync.

If you’d like more information, or assistance, please open a ticket.

Lean Commits === Swift Deploys

One of the most common actions of a developer working on a WordPress.com VIP-hosted project is committing code through our review workflow. We want to get your code deployed quickly while also making sure it is secure and performant, and this process involves human review. (Read more about why we do code reviews.)

A few months ago, we implemented automated PHP CodeSniffer checks on incoming commits. This allows us to provide some kinds of feedback on your commits right away, speeding up the overall review feedback process. As always, we encourage you to set up PHPCS in your local development environment too.

We also ask you to keep your commits small. Large commits take longer to review, can be more complex and can hold up other commits. If the changeset is larger than about 1,000 lines of code it may need more time than our traditional review/deploy workflow facilitates, to give it the proper attention it deserves. In these cases we may revert the commit and ask you to consider smaller, more atomic commits, or push the changes to a scheduled review and follow up in a support ticket about planning a timeline for that.

When we consider the size of a changeset we’re looking at the pending commits sitting in the queue. For example, if there are 3 commits around 900 lines each all committed fairly close together it’s likely we’ll end up looking at all of these as a single changeset. That would push over the limit and we may need to take a bit longer to perform the review, or review those outside of the review/deploy queue.

There are several ways commits can have a more efficient journey through the queue;

Whenever you’re in doubt, please open a ticket with us and check first. We’ll always be happy to help you think about ways to create smaller commits so they have a smooth journey through to deployment.

Happy committing!

Inspecting history and rolling back your theme using subversion

On WordPress.com VIP, we use Subversion to version all of your theme and plugin code. As you may know, one of the primary benefits of using a version control tool like svn is the availability of a complete history of your code, from your very first commit to the latest. Here are some quick tips that should help you access this history and, when necessary, quickly roll back your deployed theme to a previous state:

The Time Machine

Every time you commit a change to your theme, Subversion remembers by creating a version. Each of these versions is assigned a unique number and certain other meta, namely: the username of the person who made the version, the date and time of the version, the number of lines of code changed in the version relative to the previous one and a commit message indicating the nature of the changes in the version. You can inspect your theme’s version history on the command line using the svn log command (documentation: cli, tortoiseSVN.) Here is some sample svn log output for Automattic’s test theme on VIP:

vip/test-theme$ svn log .
------------------------------------------------------------------------
r69251 | viper007bond | 2012-06-20 21:09:52 +0000 (Wed, 20 Jun 2012) | 1 line

Remove testing code
------------------------------------------------------------------------
r69209 | batmoo | 2012-06-20 13:59:49 +0000 (Wed, 20 Jun 2012) | 2 lines

auto-deploy: rm image in a subfolder
------------------------------------------------------------------------
r69197 | batmoo | 2012-06-20 13:40:55 +0000 (Wed, 20 Jun 2012) | 2 lines

Auto-deploy: testing a php + css commit
------------------------------------------------------------------------
r66930 | tottdev | 2012-05-27 11:19:52 +0000 (Sun, 27 May 2012) | 1 line

reverting error code

Now that you can see a list of your previous versions, you may want to inspect one in more detail. What changes did a previous commit introduce? To help you find out, svn provides the svn diff command (docs: cli, tortoiseSVN.) This utility is quite flexible, but for the present purposes, a fairly typical case would be:

vip/test-theme$ svn diff -c 17546 .

Index: header.php
===================================================================
--- header.php	(revision 17545)
+++ header.php	(revision 17546)
@@ -6,7 +6,6 @@
 
  
 
-<meta name="generator" content="WordPress "/> 
 
 <link rel="stylesheet" href="?2" type="text/css" media="screen" />
 

The -c flag shows a change set resulting from a single revision, but there are many other options including -r n:m (show differences between revision n and m) and more — see the svn diff documentation for full details.

Revert! Revert!

Sometimes you many need to roll back code once you’ve committed it to WordPress.com VIP’s theme repository. This might happen for a number of reasons, including the removal of temporary code, a change in business needs, or a performance problem or error. WordPress.com VIP engineers are here to help you in any of these situations, and it’s possible for us to roll back your code for you in an emergency. However, the quickest roll-backs happen when you revert and commit the code yourself. Here’s how to prepare and push a roll-back:

1. Update your theme’s working copy to the latest revision (use

svn up your-theme

or tortoiseSVN.)

2. Use svn log to find out how far you’d like to roll back. Suppose your log reveals something like:

r4 -- change paint color
r3 -- introduce clunk engine
r2 -- modify blee tag
r1 -- add snorkle

In retrospect, you realize that something went wrong sometime after r3. You therefore decide to roll back r3 and r4.

3. Use svn merge to roll back the code. A typical cli command for this would be:

vip/my-theme$ svn merge -c -4 -c -3 .

This “un-does” the commits with version numbers 3 and 4, and rolls the code back to its previous state as of revision 2. As with svn diff, the merge command admits lots of different syntax combinations. You can read about some of those here and here. You can also roll back easily using clients like tortoiseSVN.

4. The merge you just performed has produced a changeset in your working copy. Commit your roll-back like any other change using svn commit. Make sure that you include an explanatory commit message such as “roll-back of r3 and r4 due to JS error”. Your message should indicate which versions are being rolled back, and for what reason. This will help WordPress.com VIP deploy engineers quickly deploy your changeset. Pure roll-backs usually require little or no review, and can be deployed almost immediately, so make sure your change only contains the roll-back, and nothing else. In an emergency situation, you can open an urgent ticket requesting a deploy of your roll-back. Generally this is not necessary, however, and we request that you reserve urgent tickets for when your site is down or for other serious end-user issues.

Notes:

1. We encourage you to think in terms of “rolling back” code instead of reverting individual commits. It is possible to revert things out of sequence. For instance in the previous example we could have reverted r3 but not r4. However, to do so ignores the possibility that something in r4 may depend on something in r3. In almost all cases, it’s safer to roll back all commits in sequence, especially if doing so quickly.

2. Data dependencies. Before a major roll-back it’s good to take a second to consider whether the underlying data has changed in a way that would cause problems with the old code. Have the content or options been modified in any way, including by CLI scripts? This is not usually an issue, but can be in certain cases.

3. Confusingly, subversion also features a command called svn revert whose purpose is not to take your code to a previous state, but only to un-do local un-committed modifications. Therefore we use the term “revert” both in the sense of “roll-back” and the sense of svn revert.

4. Finally, There are an extremely large number of Subversion clients which make all of this very easy. Cornerstone is a a great (though somewhat expensive) client for OSX.

New Feature: Deploy Webhook

We’ve added webhook support for deploy notifications. Now you can get pings via a simple webhook whenever a deploy for your theme or plugin code is pushed live.

You can specify your webhook endpoints in file called wpcom-meta.php in the root of your theme:

<?php
/**
 * Deploy Webhook: http://example.com/path/to/endpoint.php
 */

The value of the endpoints in the “Deploy Webhook” line of the comment can be a comma-separated list of URLs.

Deploy Webhook: http://example.com/path/to/endpoint.php,http://example.com/path/to/alternate.php 

The endpoints should not involve redirects as those may not be followed during the notification process.

Endpoints will receive something like the following as POST data:

array (
    'repo' => 'vip',
    'theme' => 'test-theme',
    'previous_revision' => '1234',
    'deployed_revision' => '5678',
    'deployer' => 'batmoo',
    'revision_log' => 'r5678 Added new stuff that does things',
)

Thank you to the VIPs and partners who suggested this feature and helped us test things out. We’ve already seen or heard about implementations utilizing this like sending notifications to a group HipChat channel. We’d love to hear what you end up doing with this too; let us know!

Automatic Deploys For CSS and Images

For all themes running on WordPress.com VIP, we review changes for performance, security, and best practices to make sure your sites are the best they can be. We have an internal goal to deploy all commits within two hours and our track record is pretty good. Roughly 90% of commits are deployed within that two-hour time frame and we think we can do even better.

Last Wednesday, as one of our meetup projects, we enabled auto-deploy for commits that consist entirely of static CSS or image files. If there are no other pending deploys for your theme, your CSS design changes and image commits will now be deployed almost immediately.   No more waiting for us to review and deploy for you.

We hope that you and your designers enjoy these speedier deploys.  We are looking into ways of expanding this “auto-deploy” functionality to other file types in the future without compromising security, performance, or functionality of your site. Let us know what you think about this feature by leaving a comment on this post.

Deploy Notifications

Deploy Notification Email

Today, we enabled deploy notification emails for all production WP.com VIP themes. It’s a feature that many of you have requested and is a nice complement to the SVN Commit Notifications we enabled a few months ago.

The emails are short and sweet, and sent instantly so you’re notified as soon as your changes are live. Any users that have access to your theme’s SVN repo will receive the email.

As always, please do let us know if you have any comments or suggestions.