CUNY Academic Commons

Boone, André - any thoughts?

Hi all. The nature of this attack is, in summary, that a botnet using 90,000+ IP addresses is attempting brute-force logins on administrator-level WordPress accounts all over the web. From what I can gather, this works by using a whitelist of known WP usernames that are often mapped to admin accounts, such as 'admin' and 'root'. As such, the #1 thing that should be done immediately is that Matt (the 'admin' user) should triple-check that his password is unique and extremely secure. As long as this is the case, it's highly unlikely that this particular hack will result in a security vulnerability on the Commons.

Aside from the security issues, the real problem for a site like the Commons is that this sort of attack works like a DDoS, and threatens to tax our resources by causing WP to initialize over and over and over again to process the login attempts. The plugins listed above will not help with that, by virtue of the fact that they're WordPress plugins. Moreover, Limit Login Attempts works by blacklisting IPs, but this particular event spans across tens of thousands of IPs. So, while I'm not opposed to Limit Login Attempts in general, and I think we can consider it for the next release (though I want André to chime in first), I don't think it's going to help with this particular set of attacks.

I'm much more dubious about Better WP Security. Many of its features involve security-through-obscurity practices, and the particular ones they list seem to me to be pretty specious. (Script kiddies are not checking Generator tags, which can easily be spoofed.) The backup features seem fine for a lone blogger, but inappropriate for a site like ours where we have formal backup procedures. The monitoring stuff sounds interesting, but I would want to go over that code with a fine-toothed comb before installing it, and in any case it would protect against a different kind of attack than the one we're discussing here. So I'd like to table that particular plugin for the moment.

DoS style attacks really ought to be handled at the webserver level. André, I know we are running mod_security, but I don't know enough about how it works to know how it helps. Does it download known attack signatures from a centralized database? Is there any way you can check the logs to see if we're getting hit?

The best mode of protection against this particular kind of attack is to block login attempts that look like bots, which in the case of WordPress means blocking POST requests to wp-login.php that do not list one of our whitelisted domains as HTTP_REFERER. See eg http://halfelf.org/2013/wp-login-protection-htaccess/. But (a) I don't want to implement this (which might in some rare cases result in false positives) if mod_security already takes care of it, and (b) if we do need to write this kind of rule, I might need help, because we have a pretty complicated setup ((*.)?commons.gc.cuny.edu, plus all of our mapped domains).

André, any feedback you could provide on the above - especially any thoughts you have about how mod_sec protects (or could protect) against this kind of attack - would be most helpful. Thanks.

Thanks for this, Boone. Just updated my password using a strong password generator, so that is taken care of.

One feature of the second plugin that seemed worthwhile to me, potentially, is changing the admin account from "admin" to something else. Do you think that's worth doing or is it just security-through-obscurity and thus not worth the trouble.

Just updated my password using a strong password generator, so that is taken care of.

Awesome, thanks.

changing the admin account from "admin" to something else. Do you think that's worth doing

As long as you have a secure password, I'd say it's not worth the trouble. Changing your user_login means that all links to your stuff (like http://commons.gc.cuny.edu/admin) will break. We can fix this internally, but anyone who has ever created an external link will find that it no longer works.

Ok, so mod_security is signature based, but we don't subscribe to a signature update service because last time I checked it was quite expensive (~5,000/yr if I remember it correctly). Having said that new signatures are added every time a new version is uploaded to EPEL (Extra Packages for Enterprise Linux).

I don't know if mod_security is a good defense against DDoS attacks. While it does have rules that prevent simple DDoS attacks I don't know how sophisticated the level of protection is. I had heard about another apache module that is dedicated to this purpose and it's something we can look into but may be hard to manage (updates, etc).

Thanks for the feedback, André.

Here's what I've decided to do: https://github.com/castiron/cac/commit/deed9980df6cd49e84d3568d0a5717282fb3e5c5

Summary: Since the attack consists of bots that send direct POST requests to wp-login.php, and since the biggest threat from the attack is not necessarily a security violation but is instead a resource issue, I've decided to block POST requests to wp-login.php (and wp-comment-post.php, for good measure) that do not originate from one of our domains. That means that normal web browser logins - where you navigate to wp-login.php via GET, and from that page you POST to log in, will work normally, while attempts to send the login form directly to the server will be redirected to the new unauthorized.html.

There are two tricky aspects. First, we have a number of mapped domains on the Commons. Most of these domains are "secondary", in that they redirect to another domain - eg, stephenneale.com redirects to neale.commons.gc.cuny.edu. These secondary domains will never be the source of a legit POST request, since legit pages are never served from them. So the only domains it's necessary to whitelist are the "primary" ones, which includes *.commons.gc.cuny.edu and a handful of others (see the changeset for the list). The second trick is that I wanted to lift the POST restriction for non-production sites. Our staging server is cdev.gc.cuny.edu, my local is local.cicdev.com, etc. We want to be sure that logins are possible on these environments, even though they're not whitelisted (and since our .htaccess is version controlled and shared between installations). So I only maintain the POST restriction when the endpoint of the POST request falls within the whitelist.

It's important to note that future mapped domains must manually be added to these whitelists. Otherwise, users will not be able to comment on or log into those sites.

I can deploy this as an emergency release, but I would like to have another set of eyes on it first. André, would you mind having a quick look at the changeset linked above to see if the logic looks correct to you?

Hang on, I believe I can put similar logic in a easier to manage custom mod_security rule. Hold that thought...

Thanks, Boone. The perhaps in the interest of time I'll leave t at that for now and we can always revisit later if the need should arise.well sone and many thanks.