CUNY Academic Commons

I've shut down the site in order to try to fix it. Please stand by.

The site is back up and should be responding normally.

Something caused the Memcached cache to be massively corrupted, such that WordPress was successfully fetching cached values, but all the values were ending up as empty strings. This includes options like 'stylesheet' and 'template', which dictate the active theme. Since an empty string is not a valid value for most of these settings, WordPress was falling back on default settings and otherwise trying not to fail catastrophically.

Once I identified the Memcached cache as the problem, the fix was fairly straightforward: a hard flush (wp_cache_flush()).

I don't have any good way to figure out how the cache got corrupted. Luke mentions that it happened when he clicked into the Customizer on the main site. Tomorrow, I will do some tests locally and in cdev to see if I can reproduce this. I'll also check with Lihua to see if there was a system-level Memcached issue. If this were the case, it'd generally mean that Memcached simply goes away, resulting in a site that is non-corrupt but loads extremely slowly. For this reason, I doubt it's a server-level issue, but it doesn't hurt to check.

I'll leave this ticket open while I'm investigating.

Thanks Boone!

Not sure if related but may be an additional data point: just before the clic above I had accepted a friend request, and the notification alert didn't clear when I did.

Many, many thanks for your immediate work on this in the evening hours, Boone. You've saved us once again.

Happy to confirm that I did, indeed, break the Commons!

DON'T CLICK

The link is there, but if you clicked it, you'd be led to an error page. The plugin didn't remove the link itself, and I didn't feel like doing the extra work to do it myself.

I figured, but wanted to confirm-- gratzi.