Incident Report: October 21 Partial Service Interruption for WordPress.com VIP

This notice relates to the following platform: WordPress.com VIP

On October 21, some users experienced an interruption of service for sites running on WordPress.com VIP from 11:55 UTC to 12:21 UTC.

What happened

An overly aggressive filtering rule was added to WordPress.com to mitigate a denial of service attack.  This rule was designed to block only malicious traffic but inadvertently blocked some legitimate requests.

The incident was initially resolved by reverting the block, and we’ve since updated the rules to block the attack but allow all legitimate requests.

Impact

Some users were unable to access WordPress.com VIP sites between 11:55 UTC and 12:21 UTC on October 21, 2019. Requests from certain operating systems, like Windows, were impacted more than others.  Unfortunately, this also meant that our monitoring systems did not immediately detect the issue.  Our stats show that approximately 15% of all requests were potentially blocked or slowed down during the above time period.

Timeline

All times are in UTC.

• 11:55: Internal monitoring alerted us to a DDoS attack against WordPress.com. We deployed mitigations as a result.
• 12:17: Monitoring traffic levels, our team noticed that incoming traffic was lower than expected, potentially symptomatic that the DDoS mitigations were too restrictive.
• 12:18: We began receiving reports from clients that sites were unavailable.
• 12:21: Having already suspected an issue, we reverted the mitigations, resolving the issue.
• 12:26: We tweeted that the issue should be resolved.

If there are any questions or concerns related to this incident, please reach out to your VIP Relationship Manager or open a ticket via vip-support@wordpress.com.