CUNY Academic Commons

I can see from running top that we're serving lots of pages, and mysqld is getting bogged down. It could be that we're experiencing some sort of DOS attack, or that some bug is ratcheting our traffic in some way. But I can't tell without accessing the logs in /var/log/httpd, and they're currently off-limits to the commons user.

Matt, you'll have to open a ticket with IT to see if they can shed some light on the situation. If they aren't interested in doing active debugging, they can send a copy of the access log, and I can start with that.

Matt has followed up with IT, and we'll continue to examine this off the thread. I'm going to close the ticket for now as it appears that there's no actual bug on the Commons, but that this was a network issue. I'll reopen with details when and if they emerge, and are Commons-specific.

I've had a chance to examine the logs.

From the looks of things, the Commons is pretty regularly attacked by bots. Most of them are brute-force attacks on wp-login.php. This sort of attack is blocked by some .htaccess rules I instituted some time ago, resulting in a 301 redirect to a static html page. It's highly unlikely that this sort of attack (which is ongoing) is related to the slowdown.

I do see a pretty major anomaly with registration. In the logs, I see a series of attempts that look like this:

142.91.111.107 - - [15/Sep/2013:05:17:31 -0400] "GET /register/ HTTP/1.0" 200 44092
142.91.111.107 - - [15/Sep/2013:05:17:32 -0400] "POST /register/ HTTP/1.0" 200 44649


An IP address GETs /register/, and then POSTs to it. I'm guessing this is a spambot, attempting to register for splogs. Apparently, these attempts are failing - probably because of our *.cuny.edu whitelist. However, on the Sunday in question, we were getting fielding these requests on the order of 20-40 times per minute. Each one of these attempts loads the entirety of WordPress - twice, actually - before finally disallowing the registration.

For wp-login.php, I'm using a trick that denies POST requests that don't originate from one of our domains - this catches most brute-force login attempts. The same technique won't work for the /register/attack, because they appear to be moving through the /register/ page. (At least, some of them do.) Stuff like CAPTCHAs, honeypots, etc also are unlikely to do much, because (a) we're not having a spam problem, and (b) these techniques require loading WP, which is the root of our problem.

One idea might be to move the /register/ page to a different URL, one that BP spammers are less likely to guess. This is a pretty thin veil, but it might help. Then again, it might cause other problems with incoming links if we were to change it.

Ray, Dom, I'm adding you as watchers in case you have any ideas about how this might be tackled at the Apache level.

The plugin looks pretty nifty, Ray. Thanks!

In https://github.com/castiron/cac/commit/cbec00c1642c4a4a8ba9ca8923685441e0ad17ee, I added cookies-for-comments along with Ray's companion plugin, and made the .htaccess changes in support of the cookie check. For the time being, I'm not making any more changes to the .htaccess file - some of them look like good protection against hacks, but that's not exactly what we're looking at in this ticket, and I only want to introduce one potentially breaking change at a time into the voodoo of our .htaccess.

Tentatively marking this as resolved.