Notice: New Relic Temporarily Disabled

This notice relates to the following platforms: VIP Go

UPDATE (Friday, November 6 @ 22:00 UTC):

New Relic continues to be disabled across the VIP Go Platform.

Our team is still working with New Relic engineers to resolve the performance issues, but the root cause has not yet been determined.

We’ll publish a new post on Monday with more details about about our investigation so far as well as options for customers that would like to restore their access in a limited manner.


UPDATE (Monday, November 2 @ 16:00 UTC):

New Relic is still disabled across the VIP Go platform and has been enabled on a select number of sites.

Working with the New Relic team, we have upgraded the PHP agent to a newer release. While there were some minor improvements noticed, the overall performance costs were still too high (compared to the expected baselines). Our investigation with their team continues.


UPDATE (Tuesday, October 27 @ 16:00 UTC):

New Relic continues to be disabled across the VIP Go Platform.

One clarification: our initial post mistakenly mentioned that this was a “maintenance” change. This change was an emergency response made after discovering that the New Relic integration was resulting in unreasonably high resource usage across the platform, resulting in degraded application performance (notably higher response times and increased error rates such as 5xx errors) and an increased risk of instability for sites. We know how important application monitoring is to our customers so this decision was not made lightly.

We’ve started initial testing to re-enable New Relic in a limited manner for a select number of sites until the root cause is identified and fixed (and proper New Relic coverage can be restored). Those investigations are ongoing with the New Relic team.


UPDATE (Friday, October 23 @ 22:00 UTC):

New Relic is still disabled and our investigation into the performance issues is ongoing. As mentioned yesterday, we have other streams of work in progress to explore alternate ways of surfacing performance and monitoring information (on a limited scale); we hope to have more information on this early next week. Note that we’re going to pause updates over the weekend but will have another update on Monday.


UPDATE (Thursday, October 22 @ 22:00 UTC):

New Relic continues to be disabled across the VIP Go Platform. Our investigation is ongoing and further changes have been made to narrow down the cause. We’re also continuing with work that will allow us to re-enable New Relic in a limited manner until the root cause as fixed. And finally, we’re looking at alternate ways to surface instrumentation, logging, and performance data that we know is critical to our customers.


UPDATE (Wednesday, October 21 @ 22:00 UTC):

New Relic is still disabled across the VIP Go Platform. We’re continuing our investigation alongside the New Relic team and have attempted some changes to rule out possible causes. We have a few other leads to follow and we’ll continue to update here as we learn more.


UPDATE (Tuesday, October 20 @ 22:00 UTC):

New Relic continues to be disabled across the VIP Go Platform. It was originally disabled because it was observed to have an adverse impact on site performance (much slower response times, higher error rate, etc). We are working with New Relic support to attempt to identify the root cause of and mitigate the performance impact. Our team is also working on a plan which would allow us to re-enable New Relic in a limited manner until the root cause is fixed.


Beginning at 17:00 UTC today, Monday, October 19, the New Relic application monitoring service will be disabled on VIP Go for some maintenance work. Disabling this service will not impact the performance of VIP Go applications.

For debugging slow queries in the interim, we recommend enabling Query Monitor.

We will update the VIP lobby when the New Relic service is re-enabled.

If you have questions about your application’s performance, or about New Relic on the VIP Go platform, please open a support ticket.

One thought on “Notice: New Relic Temporarily Disabled

Comments are closed.