Ben Metcalfe, Co-Founder of WP Engine writes…
Security is a core tenet of WP Engine’s platform but along with that comes transparency and accountability when issues arise.
A few days ago we identified the source of several malware attacks to be exploited SFTP passwords on a number of customer accounts – roughly 175 domains, or 0.005% of the ~35,000 domains WP Engine currently manages for clients.
While minor in terms of the overall number of accounts we manage, this is still a high-priority incident that we mobilized all resources within the company to resolve.
Here’s what we did:
- As a precaution we immediately cycled all SFTP passwords belonging to all users, including the 99.995% of clients that were not affected, in order to mitigate against any other compromised accounts and to preserve the integrity of the system going forward.
- We identified the source IP address of the attack, blocked that IP address and began monitoring for any similar behavior from other IP addresses.
- We removed any malware installed on any of the accounts.
- All accounts that had compromised SFTP passwords had their WordPress Core files reinstalled.
- We changed and improved some file permissions we identified during the incident.
- We implemented additional internal monitoring.
We will also be following up with the appropriate authorities and law enforcement channels in due course.
We are aware that a number of clients and other members of the WordPress community have been wondering what has been going on and why WP Engine has maintained radio silence for the last day or so.
I wanted to make sure that we had completely identified the entire scope of the vector, the accounts compromised, and most importantly the fixes put in place before we commented publicly. I believe this to be the right balance between the dual responsibilities of professional disclosure and protecting our network from any further attack. At WP Engine, transparency has always been a keystone of our culture and our ethics, and our clients’ security is our highest priority.
No one can ever guarantee with 100% confidence that security issues will never occur. This is especially true when one is dealing with the many variables associated with a project like WordPress – while WP Core is incredibly secure, well written and peer-reviewed, not every community-contributed plugin or theme may be written as securely as we might otherwise hope and aspire for them to be.
However I can assure you that you won’t find another company that is as diligent and proactive at protecting WordPress accounts than WP Engine, and who will appropriately disclose when things fall short. On the occasions when things do go wrong, we commit to pick up the pieces and put things right (or as we say “un-hack your site”).
Disappointingly, a small number of our competitors have already attempted to use this incident as an opportunity to sleight us. However I hope and trust that this post, and the procedures we implemented to resolve this incident, has restored and reaffirmed the confidence our customers have in us.
As always, we are here and available to listen and respond to any questions you may have. My personal email address is ben at wpengine dot com, and I am available for contact if you have questions or concerns related to this incident, or anything else for that matter.
Sincerely,
Ben Metcalfe
Co-Founder, WP Engine
Hi. Thanks for the quick response.
Did you notify the specific 0.005% of owners who did have malware installed?
A couple people mentioned they were getting malware warnings yesterday on my site, but I wasn’t able to trace it down. Wondering if you cleaned it up before I had a chance to review the files?
Hey Devin,
That’s a good question. Out of concern for your confidentiality, I won’t comment either way about one site in particular on a public forum, yours or anyone else’s. We can look in-depth at your site in Zendesk and answer any of your concerns there. But you’ve brought up a valid point that our automated security procedures do in-fact detect and clean up most malicious activity on our customer sites. We isolated and stopped the activity cold via a combination of our automated and redundant security protol and our top-notch sysadmin team.
If you have more questions, please let us know and we can answer them in a more secure channel.
-Austin
Same Q. as Devin – would you normally notify impacted site owners?
Juile,
Thanks for the question.
When our automated security measures detect and remove malware from a site, we always send an email to the customer to notify them of the activity and how we’ve kept their site clean.
-Austin
Thanks for transparency guys, I really like it when companies are honest with their customers. It’s to know that you are always looking into problems like this and we can trust you guys know what your doing to fix it.
Hey Paul,
Thanks for the kind words. We appreciate your confidence in WP Engine, and will continue to earn that day after day.
-Austin
I’m in the middle on this.
Is there anything you can consider if there is a similar breach in the future to not affect those of us who were doing their passwords correctly all along?
While I GREATLY appreciate the security of WP Engine and have moved over my most valued clients over to a business account with 19 domains thus far. I’m now faced with at least an hour of work resetting 38+ passwords and e-mailing each client with their new password – all the while having used http://www.strongpasswordgenerator.com‘s 15-character with symbols passwords for each and every password.
It’s a labor expense that I believed I was trading off on with this switch (backups and upgrades on my old host, now password resets here) that equals an entire month’s worth of hosting.
Part of the reason for requesting everyone to cycle their password was that (at the time) we couldn’t tell the extent of the attack and wanted to be sure there was no further potential for unauthorized access.
With password hashing we also can’t tell how ‘good’ or ‘bad’ your password is.
I understand your frustrations, and I can assure you we only forced a cycling of passwords as a last resort and to maintain best practice.
Hi, Ben… We had one client site compromised on Wednesday the 15th (not the issue you’re reporting on), which we detected and reported to WPE support who verified the issue and they (and we at the same time, gulp) fixed it within 30 minutes of our report. Altogether the javascript-based malware was in place around two hours as far as we can tell.
Your note and responses seem to say that you have automated scanning. Can you tell us the frequency of this scanning? I’m interested in understanding and improving the likely time from compromise to first action.
Conversations with WPE tech support alerted us to the mechanism by which compromises and fixes are logged by WPE on a per-site basis. We did not see the compromise or the fix logged in our case, so it looks like it was “manually” detected and “manually” fixed this time.
I’m aware of Sucuri and other paid services, and we use our own process composed of internal tripwires and external scans, which is how we can be on top of compromises so quickly ourselves (generally within 10 minutes). Just interested in the granularity of all of the involved processes.
One of our monitoring mechanisms is actually Sucuri – we contract them to provide their services upon all of WP Engine’s clients. That’s part of the service and is done transparently and at no extra charge.
We also have processes that monitor for file changes – there isn’t a specific time these occur however. There is also a balance to be struck as constantly comparing files for changes creates load upon the server which in turn reduces the available resources for serving your site at optimum performance levels.
What information was leaked to the attacker? Email/passwords? Or just password to sftp accounts?
Hi Peter – We’re posting all security updates here: https://wpengine.com/support/infosec/.