New Relic Management in the VIP Dashboard

We’re excited to announce an enhancement to our VIP Dashboard: Management capabilities of your applications APM from New Relic. This integration simplifies managing New Relic APM for both production and non-production environments, offering a seamless experience in monitoring and optimizing your application’s performance.

New Features at Your Fingertips

  • Easy Activation: Activate New Relic APM for your Production and Non-Production environments directly from the VIP Dashboard.
  • User Access Management: Effortlessly manage who has access to New Relic tools within your team.
  • Flexible Testing in Non-Production: Enable New Relic for continuous 7-day periods in non-production environments, perfect for thorough testing and analysis.
  • Uninterrupted Monitoring in Production: In Production environments, New Relic can be either permanently on or off, offering consistent monitoring without time constraints.

What to Expect

Upon activation, a New Relic entity is created for your application, and New Relic begins sampling HTTP requests. Please note, this may have a minimal impact on application performance, but the insights gained are invaluable for optimizing your user experience.

Ready to Enhance Your New Relic experience?

Get started by navigating to the New Relic section, located under “Performance” in your VIP Dashboard. For more detailed information, please check our documentation. If you’re activating New Relic for your Node.JS application please see our documentation for more advice on how to start reporting data from your application. 

Questions or Need Support?

If you have any questions or require assistance, our support team is always ready to help. Open a support ticket, and we’ll ensure your experience with New Relic APM is smooth and beneficial.

Preparing your sites for the busy retail season

The holiday season is just around the corner, and for ecommerce businesses, that means a surge in online shopping activity. While shopping days like Black Friday and Cyber Monday have historically been peak days for ecommerce activity, we know that marketing campaigns are releasing earlier and spurring consumer purchasing over the entire season.

With an increase in customers visiting your sites, it’s essential to ensure that your application is well-prepared to handle the increased traffic and provide a seamless experience.

WordPress VIP has timely, expert advice to help your teams prepare for the rush. Below, you’ll find our Professional Services team’s top recommendations—from identifying performance bottlenecks to optimizing object caching—to set your sites up for holiday success. 

Identify performance bottlenecks

The first step in preparing an application for the heightened demands of the peak ecommerce season is identifying performance bottlenecks. Many can be pinpointed with performance analysis tools such as New Relic and Query Monitor

New Relic slow transaction traces offer a great starting point to identify which parts of a transaction are responsible for latency. Sorting by “Most Dissatisfying” is an effective way of surfacing the overall slowest pain points.

Query Monitor is useful for pinpointing exactly what’s happening on a given page, surfacing specific query-related details, and highlighting slow database queries that warrant further attention. Make sure to leverage Query Monitor when logged out for the most reliable results, as code pathways used when logged in can surface false positives or false negatives.

Leverage full-page caching

Full-page caching is the primary scaling mechanism for sites on the VIP Platform, protecting a site’s origin server from increased load (where a high cache hit rate exists) by retaining valuable backend resources to handle sudden changes in traffic.

While e-commerce sites require cookies on some pages, many do not, and setting cookies that mistakenly bust the cache on all pages is a common observation of e-commerce sites. To better position your application for scale:

Fix slow database queries

Slow database queries are often a quick path to poor application performance. Our new Slow Query Logs page in the VIP Dashboard can help identify issues bogging down application performance, setting you on a course to debug connections to your database. The following are effective and easily adopted methods to improve slow database queries:

  • Offload non-performant queries with Enterprise Search (ES). Not all queries should be offloaded to Enterprise Search, but those that demonstrate evidence of performance issues are often better served via ES. For guidance, check our documentation.
  • Add extra object caching to queries, particularly around custom queries.
  • Reduce the use of JOINs, CALC_FOUND_ROWS, and limit results returned. Review our documentation to modify WP_Query requests for better performance.
  • Build queries with indexes in mind, such as the difference between post_date vs. post_date_gmt. View information about the core WordPress database schema on WordPress.org and WPVIP’s custom post meta index in our documentation.

We have more information on optimizing database queries in our documentation.

Avoid database writes

Applications hosted on the VIP Platform leverage multiple databases for optimum performance, with writes sent to an environment’s primary database and reads sent to database replicas. 

Frontend requests that result in database writes can cause subsequent queries to read only from the primary database, which can negatively affect site performance and stability under load.

  • Use Query Monitor to identify if queries are writing to the database.
  • Avoid direct database queries in code wherever possible. Instead, use WordPress API functions for fetching and manipulating data.
  • If direct database queries are required, follow these best practices.
  • When required, frontend database queries should always be related to a user-requested interaction—not automated (such as a “hit counter”). Processing AJAX or REST API requests through Javascript can also increase the full-page cache hit ratio.

Optimize object caching

Each WordPress environment is provisioned with Memcached, offering an opportunity to cache repetitive or expensive computations in memory to improve performance. Optimizing the use of object caching can be an effective way to further improve application performance, via methods such as those below:

  • Assess object cache statistics with Query Monitor, including request time, volume of GETs, and the total size of all objects stored in memory
  • Leverage wp_cache_get_multiple() to group high volumes of Memcached GET requests and minimise the overhead that occurs when retrieving data from the object cache via high volumes of lookups/sets.
  • Upgrade to WordPress 6.2 or greater to take advantage of multi-get functionality built into WordPress core
  • Keep individual cache objects under 1 MB in size (compressed), or they may fail to be stored–this is particularly important for the alloptions option, as mentioned in our documentation. Keeping object size down also reduces the amount of data being sent back and forth over the network and makes the request much faster. For example, instead of caching the entire result of a WP_Query request, which could mean many full WP_Post objects, consider caching the Post IDs and performing a much faster database query via IDs.
  • Try to cache many small requests together in a larger request. For example, generating a page that will gather data from the post object, then collect thumbnail data from post meta, and then term/term meta data for additional context can quickly multiply the number of cache requests to build a page. Instead, cache the generated HTML output to perform this process in one single cache request rather than dozens or hundreds.
  • Make use of helpers such as the Cache Nav Menus plugin, which provides a drop-in replacement to WordPress core’s wp_nav_menu() that adds additional caching to reduce database and Memcached usage when generating complicated navigation menus.

Do you have questions about any of these recommendations? Our Support Team would be happy to help.


30 days to better site performance

If you’re interested in taking site and application performance to the next level, we encourage you to explore our VIP Performance Service

Led by WordPress Core experts, our team can dive deep into your code, determine the root cause of performance issues, highlight opportunities to improve SEO performance, and deliver tailored recommendations that prepare your site for exponential growth. 

We then partner with your engineering team and/or agency to apply those recommendations, and perform a secondary review to assess the impact of that work. 

If you’d like to learn more, please reach out to your Relationship Manager or VIP Support.

PHP 8.0 available on WordPress VIP

Update: Please read this Lobby post, which contains an updated timeline & deadline for PHP 8.0 updates. PHP 8.1 is fully supported and available as part of our Software Management feature.

VIP is excited to announce that PHP 8.0 is now available on WordPress VIP environments.  PHP 8.0 is a major update that provides a plethora of new features and under-the-hood improvements. 

Some highlights of PHP 8.0 include named arguments, union types, attributes, constructor property promotion, match expression, nullsafe operator, JIT, improvements in the type system, error handling, and consistency.

Since this is a new major version, we strongly recommend that you test your application in a non-production environment running PHP 8.0. PHP 7.4 will be going end-of-life soon, so even if you are not taking advantage of these new features, now is the time to get a jump on the upgrade to PHP 8.0.

To test in a non-production environment, please open a ticket with our support team specifying which application(s) and the PHP version you’d like. In the near future, you’ll be able to control this via a setting in the VIP Dashboard and VIP-CLI.  We strongly encourage you to test in a non-production environment before updating in production.

To help your migration and testing of PHP 8.0, we recommend following these steps:

1. Update to our newly released version of VIP-CLI (2.9.5). You can pick the desired PHP version via `create` or `update` command to test new PHP versions locally.

Screenshot of VIP Local Development Environment Configuration Wizard showing PHP option

2. Update a non-production environment as soon as possible to get visibility into scan results and to ensure ongoing compatibility with PHP 8.0 via the VIP Code Analysis Bot. We have added multi-version linting to the bot, so if your repository powers several sites with different versions of PHP, the bot will check the Pull Request against each enabled version. See the screenshot below for an example:

VIP Code Analysis Bot flagging syntax errors for multiple versions

3. Review the WordPress VIP documentation “Prepare application code for a PHP version upgrade” guide and perform the recommended scan.

PHP 7.4 will stop receiving security updates on November 28, 2022. On this date, all sites on WordPress VIP still using PHP 7.4 will be upgraded to PHP 8.0 to minimize the risk from unpatched security vulnerabilities. 

PHP 8.1 is also available but WordPress core does not yet fully support PHP 8.1, so we strongly recommend against running it in production at this time. PHP 8.1 is fully supported and available as part of our Software Management feature. [Updated 24 August 2022]

Resolved: WordPress VIP Availability Issues

Update (20:24 UTC): On Thursday, June 3, beginning at 09:55 UTC  (5:55 a.m. ET), the VIP Platform experienced degraded performance that lasted approximately 100 minutes.

We take incidents like this very seriously and would like to outline what happened, as well as the steps we have taken to help prevent future occurrences.

What Happened

The issue affected response times in a single data center on the West Coast of the United States, which affected a subset of sites on the WordPress VIP platform.

This was caused by two memcached hosts having issues when network switches (hardware) they are connected to were upgraded and rebooted during regular maintenance operations.

Impact

Those sites in the West Coast datacenter experienced slower response times and, in some cases, 503 errors. The issue was apparent between 09:55 UTC and 11:35 UTC. Busier sites with more memcached traffic were more adversely impacted.

Not Impacted

Sites hosted in other data centers were not affected.

Timeline

  • At 10:20 UTC, we received the first reports of issues.
  • At 11:20 UTC, we identified that the issue was related to a specific datacenter.
  • At 11:35 UTC, performance was restored.

Future Prevention

We have already implemented, or are currently implementing, additional safeguards and process improvements designed to prevent similar issues from happening again. These include (but are not limited to):

  • Additional architecture changes to prevent memcache oversubscription. 
  • Internal monitoring to more quickly identify and resolve any issues.

Questions?

If you have any questions related to this incident, please open a support ticket and we will be happy to assist.


Update (12:22 UTC): The issue affecting response times in a single West Coast data center has been resolved. Only some VIP sites were affected. More detailed information will be posted here in the VIP Lobby, after the investigation has been completed.


Update (11:54 UTC): We’re currently seeing improvements and are continuing to monitor the situation.


We are currently troubleshooting issues with the WordPress VIP platform, affecting in a single data center on the West Coast of the United States. A limited number of sites may experience slow load times or errors in the meantime.

Sorry for the trouble! We are working on the issue, and will follow up with another alert once this is resolved

We will continue to update this post and tweet out status updates from @wpvipstatus until the issue is resolved

If you have any questions, please open a support ticket and we will be happy to assist.

Restoring Access to New Relic

This post is to provide an update about New Relic, and the steps we’ve taken to restore access.

A Brief History

In October 2020, New Relic was temporarily disabled on all sites across the VIP Go Platform due to performance issues. Our team, in coordination with engineers from New Relic, were able to narrow down the issues and as of November 13th, we had an update that brought the performance cost of enabling New Relic to within the expected range of 10–12% impact on response times, with the framework-specific options disabled.

Next Steps

Our teams have continued to work on improvements, and soon, we’ll have an update to the New Relic PHP agent that allows for most of the framework-specific functionality to be enabled without any additional performance impact. In the next iteration, the New Relic agent will ignore only the lib directory of our VIP Go MU Plugins, but the rest of the framework options, including the WordPress-specific functionality will be restored.

We expect to deploy the updated New Relic PHP agent in the last week of February. For anyone who has already requested to have New Relic re-enabled since November, your applications will automatically receive the updated New Relic agent as soon as it’s released.

Moving forward, New Relic will remain available for all VIP Go sites, but will be enabled upon request, and offered during the onboarding process for any new customers as part of a plan to meet specific monitoring objectives. We’ve updated our documentation to reflect the change.

Please open a support ticket if you’d like New Relic re-enabled on any of your environments or have any additional questions, and we’ll be happy to assist.

An Update on New Relic

On October 19, 2020, we made the difficult decision to disable New Relic across the VIP Platform. This post provides an update on where things are and our upcoming plans.

The change was an emergency response made after discovering that the New Relic integration was resulting in unreasonably high resource usage across the platform. While the impact varies per application, we noticed degraded application performance across the board, notably slower response times (over 150% slower in some cases), higher error rates (such as 5xx errors), and an increased risk of instability for sites.

Investigation

Our team is continuing to work with New Relic to investigate the root cause of the performance issues. We’ve spent extensive time testing different versions of the New Relic PHP agent against multiple versions of WordPress and VIP Go MU Plugins. We’ve also tested with the transaction tracing feature of New Relic disabled, and so far, the performance issues have persisted.

We’re now experimenting with various configuration options to narrow down which features are causing problems. Ultimately, we anticipate that this will lead to improvements in the PHP agent itself, and we are coordinating with New Relic to test experimental builds to address any issues that are found through this process.

Restoring Application Monitoring & Visibility

To help restore access to New Relic, we’ve made some changes to how the service is integrated with the VIP Platform. We can now re-enable the service in a limited manner for production sites. While we are confident that this will not impact site reliability, it may still impact performance for a subset of requests. Please open a support ticket if you’d like New Relic re-enabled with the limited allocation, or have any questions.

In the meantime, we’re also working on exposing some of the stats we use for internal monitoring via the VIP Dashboard. These are collected with no overhead to the application itself, and would provide similar application health data for VIP customers.

We hope to have this resolved soon, and we thank you for your patience. Please don’t hesitate to reach out if you have any further questions.

[Resolved] GitHub Performance Issues Affecting WordPress VIP

Resolved: GitHub is reporting resolution of performance issues, and all services operating normally. All VIP services should be operating as expected at this time. If you experience any further issues, please reach out to us directly at vip-support@wordpress.com.

Update: GitHub has deployed a fix and is monitoring recovery. We are continuing to monitor the situation. (17:32 UTC)

We are aware of ongoing performance issues on GitHub which are affecting some VIP sites. The issues may affect code deploys. We are monitoring the situation, and will follow up with another alert once this is resolved.

We will continue to update this post and tweet out status updates from @wpvipstatus until the issue is resolved. You can also subscribe for updates directly from GitHub regarding this incident here:
https://www.githubstatus.com/incidents/phnch1rww464

If you have any questions, or are experiencing any issues, please email vip-support@wordpress.com.

Notice: Change to ASYNC Login behaviour

This notice relates to the following platform: WordPress.com VIP

The default behaviour of the WordPress.com platform can cause a white screen whilst rendering the front-end of a site, as login credentials are verified. To speed this up, we added an asynchronous login feature which, if you wished, you could opt out from.

However, this has caused some technical issues for some clients and, as a result, we have made the decision to make this opt-in. With immediate effect, asynchronous login has therefore been switched off.

What do I need to do?

If you had previously opted-out to asynchronous login with the WPCOM_DISABLE_ASYNC_REMOTE_LOGIN constant, then please remove this as the functionality no longer exists. This is optional, but recommended.

If you experience any issues with render blocking and would like to re-enable asynchronous login then please add the following code to your site…

if ( ! defined( 'WPCOM_ENABLE_ASYNC_REMOTE_LOGIN' ) ) {
    define( 'WPCOM_ENABLE_ASYNC_REMOTE_LOGIN', true );
}

If you have any questions, please open a support ticket and we’ll be happy to assist.

Have questions?

If you have any questions related to this release, please open a support ticket with details and we will be happy to assist.

Co-Authors Plus Global Filter

This notice relates to the following platforms: WordPress.com VIP, VIP Go

This post is regarding the VIP plugin Co-Authors Plus (all versions).

Co-Authors Plus has an expensive join query on author pages, which performs a lookup using both the term and post_author columns.  Unfortunately, this can lead to slow or killed author queries on sites with large databases or high traffic.  To prevent this from occurring and to improve performance, we will be adding the coauthors_plus_should_query_post_author filter to all VIP sites using the Co-Authors-Plus plugin on WordPress.com via:

add_filter( 'coauthors_plus_should_query_post_author', '__return_false' );

This filter disables the compatibility mode and instead, uses a simpler, tax-only query and thus, results in a faster query because we are no longer performing a lookup on the  post_author column.  The compatibility mode is enabled by default in the plugin to ensure that any posts added prior to plugin activation will display properly, as they may not have the proper author terms associated.  However, as a precaution, one of the preparation steps that we have undertook involved running a script to back fill the associated author terms on all posts.

When will this be effective?

This filter will be added to the WordPress.com VIP platform on Tuesday, November 14 (subject to change if any critical issues are discovered prior).

We are planning on adding this to the VIP Go platform eventually and we will announce a date when that will occur.  If you don’t want to wait until the expected TBA date, feel free to add the filter to your VIP Go site.  Please request to have the following WP CLI command wp co-authors-plus create-terms-for-posts run prior to adding the filter to back fill author data on all posts.

What do I need to do?

Once the change has been deployed, please test your site (especially the author pages) to ensure that everything is working as expected.  We do not foresee any major issues that could arise, but if you wish to test the change prior to the scheduled date, feel free to add the filter to your theme.

If you have any concerns or suggestions, please get in touch by leaving a comment below!

PHP 7 on VIP Go

Next week on Thursday, December 8th, all VIP Go pre-production environments (not including production environments) will be switched over to PHP 7. After the switch, please thoroughly test all pre-production environments for any issues.

After upgrading pre-production environments, production environments will be switched to PHP 7 on Thursday, January 5th (4 weeks later).

PHP 7 brings important performance improvements and has been extensively tested with WordPress at scale on WordPress.com, including all WordPress.com hosted VIP sites.

Ahead of the upgrade, please ensure all local environments are running PHP 7 with WP_DEBUG enabled and pay close attention to any PHP warnings or other issues. For example, VVV has been using PHP 7 for several months, but if you have an out-of-date instance, simply pull the latest code from GitHub and run vagrant up --provision to be upgraded.

For more information on migrating from PHP 5.6 to PHP 7, please see the official Migration Guide.

If you’d like to switch over to PHP 7 ahead of this timeline, please let us know – we’re happy to upgrade you at a time of your choosing.

As always, if you have any questions, please let us know!

tl;dr – pre-production VIP Go sites will be upgraded to PHP 7 on December 8th, and production sites will be upgraded 4 weeks later on January 5th.