CUNY Academic Commons

Site is back up.

I just noticed that it is back down.

I am seeing it is down now -- thursday morning

Seems to be back up now.

Just an update: I am still experiencing significant slowness, and have had a few users reach out to me to ask about stability.

Thanks, Laurie. I've seen this sort of behavior before too, and hypothesized that it had something to do with the TLS handshake. But in those instances, it ended up looking like the TLS stuff was just a symptom of the underlying problem, which is a problem with the database connection.

I am at a loss for how to proceed.

In our last maintenance update (v2.4.1) on June 25th, we updated the Editoria11y Accessibility Checker plugin to v1.0.17. This plugin update included a database upgrade routine that can degrade performance especially on a multisite install. See https://github.com/itmaybejj/editoria11y-wp/issues/32 for a detailed report.

The plugin author has updated the plugin to v1.0.18 to address this issue; I've committed it here -- https://github.com/cuny-academic-commons/cac/commit/bcc4595d1b0a37f4514e6c85ed715a37b40ee445 -- and pushed the update to production in hopes that the routine is fixed. Hopefully this helps with database responsiveness, but will keep checking the plugin update list to see what else could have caused the database slowness.

Hi Ray,

Thanks for this insight. I believe Editorially is currently installed on every site on the Commons at a network level (similar to Imsanity, etc). We had decided to add this plugin on all sites as an effort towards making sites more accessible and alerting users to accessibility tips. I have actually gotten feedback that this plugin confuses users because it adds many many flags to the front end that are only visible (I think) to site admins.

Would it be worth deactivating it on sites at network level (if possible?) to see if this helps with the current loading and database issues?

Thanks for your work on this, Ray. Coincidentally or not, I am seeing the "technical problems" downtime message on the Commons right now.

Coincidentally or not, I am seeing the "technical problems" downtime message on the Commons right now.

Yeah, the Commons is down again unfortunately.

Would it be worth deactivating it on sites at network level (if possible?) to see if this helps with the current loading and database issues?

I've written a filter to disable Editoria11y temporarily on production in /wp-content/mu-plugins/20515-disable-editoria11y.php. We'll see if that helps once the Commons is back up. I'm still going through the plugin and theme list to see if anything else jumps out.

The Commons is back online. We'll see if disabling the Editoria11y plugin will stabilize the database responsiveness issues. I can confirm in the slow query log Yiu Ming provided that Editoria11y did generate many slow queries, which could have caused a cascading delay with other database queries. Will check in a bit later to see how the Commons is doing.

Adding Scott and Marilyn as watchers.

I haven't received a JetPack alert since 6 pm last night, but I'd say that my site still seems extremely slow.

I can confirm it is running a lot faster now.

Seems to be working ok for me. I just posted an update to the team forum, which took forever a few days ago to register/send, and it was pretty instant.

I got an email notification that there was some downtime today between 4:35am ET and 6:11am ET.

I looked at the error log to see what could be a cause and saw these entries:

[Tue Jul 09 04:35:01 2024] [notice] [pid 108823] sapi_apache2.c(349): [client 116.202.254.214:54230] WordPress database error WSREP has not yet prepared node for application use for query SELECT * FROM `wp_cavalcade_jobs` WHERE site = 1120 AND hook = 'jetpack_v2_heartbeat' AND args = 'a:0:{}' AND status IN('waiting','running') ORDER BY nextrun ASC LIMIT 1 made by require('wp-blog-header.php'), require_once('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), do_action('plugins_loaded'), WP_Hook->do_action, WP_Hook->apply_filters, Jetpack->configure, Jetpack_Heartbeat::init, Automattic\\Jetpack\\Heartbeat::init, Automattic\\Jetpack\\Heartbeat->__construct, wp_next_scheduled, wp_get_scheduled_event, apply_filters('pre_get_scheduled_event'), WP_Hook->apply_filters, HM\\Cavalcade\\Plugin\\Connector\\pre_get_scheduled_event, HM\\Cavalcade\\Plugin\\Job::get_jobs_by_query
[Tue Jul 09 04:35:02 2024] [notice] [pid 109152] sapi_apache2.c(349): [client 84.51.29.236:4803] WordPress database error WSREP has not yet prepared node for application use for query SELECT display_meta, notifications FROM wp_1859_gf_form_meta WHERE form_id=1 made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/twentyfourteen/single.php'), get_sidebar, locate_template, load_template, require_once('/themes/twentyfourteen/sidebar.php'), dynamic_sidebar, WP_Widget->display_callback, GFWidget->widget, GFFormsModel::get_form_meta

I looked up this error -- https://galeracluster.com/library/documentation/crash-recovery.html -- and it looks like one of the database cluster nodes went down, which might have triggered a database restart from Lihua's scripts. I haven't contacted IT about this yet.

Sorry for the double-post for those that already received it. I previously posted this message in the wrong ticket (#18841).