I emailed Boone about this, but I'm creating this ticket to document the outage
Updated by Raymond Hoh 11 months ago
FYI, I noticed the Commons maintenance page was displaying on production just now. At the same time, cdev was functional, but was referencing a stale object cache that I previously reported in #14908.
wp cache flush in WP-CLI and that appears to have fixed things.
Updated by Boone Gorges 11 months ago
I was able to determine that the Elementor update was the source of the problem. Here's the original update: https://github.com/cuny-academic-commons/cac/commit/a5015ca4e134a8310fdcc01ad64ad35688b22f66 and the revert https://github.com/cuny-academic-commons/cac/commit/abbf0061f153fa32d32954f30b138775714a84e4
I looked through the Elementor changeset and I don't understand what could be causing the problem. It's indicative of either (a) some sort of database upgrade/migration, or (b) a self-request loop. But I didn't see anything of that sort in the update.
I couldn't reproduce the problem locally, but I also don't have a full clone of the production site, and even if I did, it doesn't get the traffic of the production site. I'll take more time in the upcoming days to try to make an educated guess at what's going on.