CUNY Academic Commons

Looks like the problem came from one site: sthomas21003.commons.gc.cuny.edu.

For some reason, the template option for the site was set to twentytwelve, whereas the stylesheet for the site was set to academica.

This resulted in the following fatal error when viewing the site:

Fatal error: require(): Failed opening required '/PROD_WWW/html/commons/www/wp-content/themes/twentytwelve/inc/customizer/bootstrap.php' (include_path='.:/usr/share/p
ear:/usr/share/php') in /PROD_WWW/html/commons/www/wp-content/themes/academica/functions.php on line 10

I switched the template back to academica with the following WP-CLI line:

wp option update template academica --url=sthomas21003.commons.gc.cuny.edu --skip-themes=twentytwelve,academica

The --skip-themes parameter was needed otherwise using the wp option update command would also cause the fatal error.

The failed Cavalcade tasks are no longer growing.

The next step is figuring out what might have caused the template DB option to change to twentytwelve. In our wp-config.php file, I see that we have set the WP_DEFAULT_THEME constant to twentytwelve. In WordPress, this constant is used only in validate_current_theme(). This function is used in a couple of places, but most notably during the WP Customizer: https://github.com/cuny-academic-commons/cac/blob/491f7adadb505958a6e8c5c38e6a7d5b4ac119b3/wp-includes/class-wp-customize-manager.php#L654-L660. Still not sure exactly what caused the template DB option to change though. Any ideas, Boone or Jeremy?

Nice find, Ray! Good to know this is not a broader problem with Cavalcade.

Still not sure exactly what caused the template DB option to change though. Any ideas, Boone or Jeremy?

I don't have any clear thoughts. I wonder if perhaps there was a cache-related glitch, as we've sometimes experienced in the past. Like perhaps the 'template' option wasn't properly loaded from the 'alloptions' cache on some specific pageload, and WP fell back on the default, saving the default theme to the database. It does seem strange, though, because validate_current_theme() uses switch_theme(), which should change both the 'template' and 'stylesheet' options.

Given that we haven't heard of this problem before, I'd say that perhaps it's not worth sinking a lot of time into investigating.

Great work, Ray! Thank you!

Over the long weekend, we had around 1800 failed Cavalcade tasks from July 2nd until now due to a problem with Jetpack and invalid syntax type. Boone, you can run the following DB query to view the failed tasks:

select * from wp_cavalcade_jobs where nextrun >= '2023-07-02 23:34:22' and status = 'failed' and site = 23793 order by nextrun asc;

The issue is Jetpack was throwing the following fatal error during their scheduled tasks on one specific site -- apiza.commons.gc.cuny.edu:

Fatal error: Uncaught Error: [] operator not supported for strings in /var/www/html/commons/www/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-connection/src/class-urls.php:102

Stack trace:
#0 /var/www/html/commons/www/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-connection/src/class-urls.php(137): Automattic\Jetpack\Connection\Urls::get_protocol_normalized_url()
#1 /var/www/html/commons/www/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-connection/src/class-urls.php(146): Automattic\Jetpack\Connection\Urls::get_raw_or_filtered_url()
#2 /var/www/html/commons/www/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-identity-crisis/src/class-identity-crisis.php(201): Automattic\Jetpack\Connection\Urls::home_url()
#3 /var/www/html/commons/www/wp-includes/class-wp-hook.php(308): Automattic\Jetpack\Identity_Crisis->add_idc_query_args_to_url()
#4 /var/www/html/commons/www/wp-includes/plugin.php(205): WP_Hook->apply_filters()
#5 /var/www/html/commons/www/wp-content/plugins/jetp in /var/www/html/commons/www/wp-content/plugins/jetpack/jetpack_vendor/automattic/jetpack-connection/src/class-urls.php on line 102

The specific Jetpack call was looking for the "jetpack_sync_https_history_home_url" option and this option was returning the string "100" instead of an array, which caused the fatal error.

I fixed this by deleting the "jetpack_sync_https_history_home_url" option for the apiza.commons.gc.cuny.edu site. I'm guessing this might have occurred for the same reason as my original report, some cache-related glitch.

Thanks for catching this!