Incident Report: Feb 13 Service Disruption

Overview

Between 11:33 and 12:35 UTC on 13 February 2022, WordPress VIP experienced a partial service disruption due to a Distributed Denial of Service (DDoS) attack. As a result, affected sites saw an intermittent increase in latency, timeouts, and 503 errors.

Chronology of Events

Date	UTC Time	Update
13 Feb. 2022	11:33	DDoS detected against the VIP Platform.
	11:36	VIP Edge Caches report being unable to reach Origin Data Centers.
	11:39	DDoS target identified.
	12:05	Targeted traffic-blocking rules implemented.
	12:35	Issue mitigated. Latency and error rates return to normal.
	12:36	VIP Lobby post updated.

What Happened

A Distributed Denial of Service (DDoS) attack caused congestion on a subset of VIP’s Globally Distributed Edge Cache resulting in intermittent latency, timeouts, and 503 errors. Targeted blocks were implemented which mitigated the attack and returned latency and error rates to normal.

Further Infrastructure details can be found at https://wpvip.com/infrastructure/

Future Prevention

VIP’s proactive monitoring and automated DDoS mitigation systems have been updated to more easily identify DDoS attacks of this nature. Additionally, the processes and tools used to identify and mitigate attacks are being reviewed to add additional protection and reduce the time between when an attack is detected and it is mitigated.