Restoring Access to New Relic

This post is to provide an update about New Relic, and the steps we’ve taken to restore access.

A Brief History

In October 2020, New Relic was temporarily disabled on all sites across the VIP Go Platform due to performance issues. Our team, in coordination with engineers from New Relic, were able to narrow down the issues and as of November 13th, we had an update that brought the performance cost of enabling New Relic to within the expected range of 10–12% impact on response times, with the framework-specific options disabled.

Next Steps

Our teams have continued to work on improvements, and soon, we’ll have an update to the New Relic PHP agent that allows for most of the framework-specific functionality to be enabled without any additional performance impact. In the next iteration, the New Relic agent will ignore only the lib directory of our VIP Go MU Plugins, but the rest of the framework options, including the WordPress-specific functionality will be restored.

We expect to deploy the updated New Relic PHP agent in the last week of February. For anyone who has already requested to have New Relic re-enabled since November, your applications will automatically receive the updated New Relic agent as soon as it’s released.

Moving forward, New Relic will remain available for all VIP Go sites, but will be enabled upon request, and offered during the onboarding process for any new customers as part of a plan to meet specific monitoring objectives. We’ve updated our documentation to reflect the change.

Please open a support ticket if you’d like New Relic re-enabled on any of your environments or have any additional questions, and we’ll be happy to assist.

An Update on New Relic

On October 19, 2020, we made the difficult decision to disable New Relic across the VIP Platform. This post provides an update on where things are and our upcoming plans.

The change was an emergency response made after discovering that the New Relic integration was resulting in unreasonably high resource usage across the platform. While the impact varies per application, we noticed degraded application performance across the board, notably slower response times (over 150% slower in some cases), higher error rates (such as 5xx errors), and an increased risk of instability for sites.

Investigation

Our team is continuing to work with New Relic to investigate the root cause of the performance issues. We’ve spent extensive time testing different versions of the New Relic PHP agent against multiple versions of WordPress and VIP Go MU Plugins. We’ve also tested with the transaction tracing feature of New Relic disabled, and so far, the performance issues have persisted.

We’re now experimenting with various configuration options to narrow down which features are causing problems. Ultimately, we anticipate that this will lead to improvements in the PHP agent itself, and we are coordinating with New Relic to test experimental builds to address any issues that are found through this process.

Restoring Application Monitoring & Visibility

To help restore access to New Relic, we’ve made some changes to how the service is integrated with the VIP Platform. We can now re-enable the service in a limited manner for production sites. While we are confident that this will not impact site reliability, it may still impact performance for a subset of requests. Please open a support ticket if you’d like New Relic re-enabled with the limited allocation, or have any questions.

In the meantime, we’re also working on exposing some of the stats we use for internal monitoring via the VIP Dashboard. These are collected with no overhead to the application itself, and would provide similar application health data for VIP customers.

We hope to have this resolved soon, and we thank you for your patience. Please don’t hesitate to reach out if you have any further questions.