On October 19, 2020, we made the difficult decision to disable New Relic across the VIP Platform. This post provides an update on where things are and our upcoming plans.
The change was an emergency response made after discovering that the New Relic integration was resulting in unreasonably high resource usage across the platform. While the impact varies per application, we noticed degraded application performance across the board, notably slower response times (over 150% slower in some cases), higher error rates (such as 5xx errors), and an increased risk of instability for sites.
Investigation
Our team is continuing to work with New Relic to investigate the root cause of the performance issues. We’ve spent extensive time testing different versions of the New Relic PHP agent against multiple versions of WordPress and VIP Go MU Plugins. We’ve also tested with the transaction tracing feature of New Relic disabled, and so far, the performance issues have persisted.
We’re now experimenting with various configuration options to narrow down which features are causing problems. Ultimately, we anticipate that this will lead to improvements in the PHP agent itself, and we are coordinating with New Relic to test experimental builds to address any issues that are found through this process.
Restoring Application Monitoring & Visibility
To help restore access to New Relic, we’ve made some changes to how the service is integrated with the VIP Platform. We can now re-enable the service in a limited manner for production sites. While we are confident that this will not impact site reliability, it may still impact performance for a subset of requests. Please open a support ticket if you’d like New Relic re-enabled with the limited allocation, or have any questions.
In the meantime, we’re also working on exposing some of the stats we use for internal monitoring via the VIP Dashboard. These are collected with no overhead to the application itself, and would provide similar application health data for VIP customers.
We hope to have this resolved soon, and we thank you for your patience. Please don’t hesitate to reach out if you have any further questions.
WordPress VIP Lobby 