Broken Link Checker cron jobs running long
The broken-link-checker plugin runs a cron job on a regular basis that scans for links in post content, pings the links, and records those that 404 or time out. Unsurprisingly, this process can take a very long time. Recent debugging leads me to believe that it might be one of the culprits in our ongoing saga of open connections.
The cron job seems to be important to the way the plugin works (link rot happens over time) so I don't want to disable it completely. The plugin does have a few limiters built in for execution time and memory usage, but the limits are set too high for a site like the Commons. I'd suggest we filter them so that they can't be set over a certain amount. See https://boonesparty.commons.gc.cuny.edu/wp-admin/options-general.php?page=link-checker-settings etc.
#1 Updated by Boone Gorges 11 months ago
I've imposed some limits in https://github.com/cuny-academic-commons/cac/commit/ebabec4b679e0eb924af7b6bdc061bc357a63d6b.
Basically, the defaults of the plugin - max execution time, target load, etc - were clearly written for a situation where the plugin is running on a single WordPress site. The math doesn't scale at all with large installations, and the lock mechanisms aren't multisite-aware at all.
The adjustments I've made, when combined with each other, should reduce the likelihood of concurrent broken-link-checker cron jobs by about an order of magnitude. I'm going to put this into production right away, as we've had an uptick in performance issues over the last week. I'll continue to monitor.