CUNY Academic Commons

A handful of times over the past month or so, the Memcached connection seems to have failed completely. This manifests first by extreme slowness in Commons load times. With Query Monitor enabled, I can then see that there are hundreds of database hits instead of the normal small handful. Occasionally, `wp cache flush` fixes it. But more often, I have to go in and switch from one to the other item in the $memcached_servers array. When I do this (and, importantly, run `wp cache flush` immediately afterward, to ensure that WP doesn't access stale cache in the other node) things come back to life immediately.

I don't really have the tools at my disposal to debug this fully. If the problem is that the cache is corrupted and thus inaccessible, I guess I would know that by looking at some sort of Memcached log https://serverfault.com/questions/422350/memcached-debuging-server-logs-monitor-the-memcached-servers and perhaps by examining the contents of the cache directly via Telnet. Or the problem could be a connection issue, though I'm unsure how to test that directly.

I don't know the proper next steps, but I'm starting this ticket so that the issue is on Ray's radar and so that we have a place to collect ideas and incident reports.