Bug #9865
closed
Broken Link Checker cron jobs running long
Added by Boone Gorges almost 6 years ago.
Updated over 5 years ago.
Description
Previously: #6731
The broken-link-checker plugin runs a cron job on a regular basis that scans for links in post content, pings the links, and records those that 404 or time out. Unsurprisingly, this process can take a very long time. Recent debugging leads me to believe that it might be one of the culprits in our ongoing saga of open connections.
The cron job seems to be important to the way the plugin works (link rot happens over time) so I don't want to disable it completely. The plugin does have a few limiters built in for execution time and memory usage, but the limits are set too high for a site like the Commons. I'd suggest we filter them so that they can't be set over a certain amount. See https://boonesparty.commons.gc.cuny.edu/wp-admin/options-general.php?page=link-checker-settings etc.
I've imposed some limits in https://github.com/cuny-academic-commons/cac/commit/ebabec4b679e0eb924af7b6bdc061bc357a63d6b.
Basically, the defaults of the plugin - max execution time, target load, etc - were clearly written for a situation where the plugin is running on a single WordPress site. The math doesn't scale at all with large installations, and the lock mechanisms aren't multisite-aware at all.
The adjustments I've made, when combined with each other, should reduce the likelihood of concurrent broken-link-checker cron jobs by about an order of magnitude. I'm going to put this into production right away, as we've had an uptick in performance issues over the last week. I'll continue to monitor.
nice work, Boone -- thank you
- Target version changed from 1.13.3 to 1.13.4
- Target version changed from 1.13.4 to 1.13.5
- Target version changed from 1.13.5 to 1.13.6
- Target version changed from 1.13.6 to 1.13.7
- Target version changed from 1.13.7 to 1.13.8
- Target version changed from 1.13.8 to 1.13.9
- Status changed from New to Resolved
I've reviewed a few sites where BLC is running, and it appears that the cron jobs are running (and completing) as expected. I think we can close.
Also available in: Atom
PDF