[CircleCI Security Alert] Warning: Phishing attempt for login credentials

WordPress VIP offers CircleCI as an optional service for customers. Please reach out to VIP if you think you may have accidentally clicked a link.

Reposting the CircleCI Security Alert from Thursday, September 15, 2022

Yesterday evening (Sept 15), we (CircleCI) became aware of a phishing attempt for customers’ CircleCI and GitHub credentials. We have no reason to believe your organization has been specifically targeted or that your account has been compromised, but want our customers to be aware that there is an ongoing phishing attempt and to exercise due caution.
This is an example of the email impersonating CircleCI in an attempt to gain access to your account:

CircleCI will not require users to login to review any updates to Our Terms of Service. Additionally, these phishing attempts include links that send users to circle-ci[.]com, which is not owned by CircleCI. Any emails from CircleCI should only include links to circleci.com or its sub-domains. If you believe you or someone on your team may have accidentally clicked a link in this email, please immediately rotate your credentials for both GitHub and CircleCI, and audit your systems for any unauthorized activity.

If you need help or have any questions, please do not hesitate to reach out to our team.
To better building,

The Team at CircleCI

Please reach out to VIP if you think you may have accidentally clicked a link.

WordPress VIP Status Page

From June 1, 2022, WordPress VIP service disruptions and incidents will no longer be announced in the VIP Lobby. Please check our WordPress VIP Status Page for any known issues before opening an urgent ticket.

For the latest updates, we recommend subscribing to the status page (email or RSS feed) using the subscribe button at the top of the page:

Why are we making this change?

A standalone status page provides automated monitoring of our key services as well as testing sites in each data center, giving customers a single location for service monitoring, site disruptions, and incident reports. This change also means faster, more timely updates regarding data center or platform-wide issues. Going forward, the VIP Lobby will focus on announcing product enhancements, releases, and other important platform messages.

Incident Report: Feb 13 Service Disruption

Overview

Between 11:33 and 12:35 UTC on 13 February 2022, WordPress VIP experienced a partial service disruption due to a Distributed Denial of Service (DDoS) attack. As a result, affected sites saw an intermittent increase in latency, timeouts, and 503 errors.

Chronology of Events

DateUTC TimeUpdate
13 Feb. 202211:33DDoS detected against the VIP Platform.
11:36VIP Edge Caches report being unable to reach Origin Data Centers.
11:39DDoS target identified. 
12:05Targeted traffic-blocking rules implemented.
12:35Issue mitigated. Latency and error rates return to normal.
12:36VIP Lobby post updated.


What Happened

A Distributed Denial of Service (DDoS) attack caused congestion on a subset of VIP’s Globally Distributed Edge Cache resulting in intermittent latency, timeouts, and 503 errors. Targeted blocks were implemented which mitigated the attack and returned latency and error rates to normal.


Further Infrastructure details can be found at https://wpvip.com/infrastructure/

Future Prevention

VIP’s proactive monitoring and automated DDoS mitigation systems have been updated to more easily identify DDoS attacks of this nature. Additionally, the processes and tools used to identify and mitigate attacks are being reviewed to add additional protection and reduce the time between when an attack is detected and it is mitigated. 

Incident Report: Jan 12 Service Disruption

Overview

Between 20:30 and 20:46 UTC on 12 January 2022, WordPress VIP experienced a partial service disruption due to a code change that impacted how HTTP requests are routed within the WordPress VIP infrastructure. As a result, the majority of uncached requests for affected sites were served 503 responses during this time.

Chronology of Events

DateUTC TimeUpdate
12 Jan. 202220:15Code change release causes an internal API to generate incorrect configuration data. 
12 Jan. 202220:30As part of normal operations, VIP routing configurations dynamically  update using data from an internal API. The data is incorrect  because of the previous update.
12 Jan. 202220:30:42 First failed request is recorded in the logs and internal alerts received. 
12 Jan. 202220:32VIP begins investigation.
12 Jan. 202220:40VIP identifies the problem.
12 Jan. 202220:43VIP reverts the offending code and reloads routing configurations.
12 Jan. 202220:46:08The last failed request resulting from this issue is recorded in our logs. Incident is resolved.
12 Jan. 202220:50VIP Lobby updated, post-outage process begins.

What Happened

A code release caused incorrect data to be materialized by an internal API.  Our systems use this data to determine how HTTP requests are routed within the WordPress VIP Infrastructure. With incorrect data, our systems were incapable of forwarding traffic to the correct destination, and returned errors to uncached requests on affected sites resulting in HTTP 503 errors.

Remediation

The issue was addressed by reverting the code change that led to incorrect routing configurations and deploying the correct configurations. 

Future Prevention

The process for code releases is being reviewed to add additional procedural safeguards. Automated checks are also being investigated to minimize the chance of a similar problem happening in the future.