Project

General

Profile

Bug #9865

Broken Link Checker cron jobs running long

Added by Boone Gorges about 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority name:
Normal
Assignee:
Category name:
-
Target version:
Start date:
2018-05-31
Due date:
% Done:

0%

Estimated time:

Description

Previously: #6731

The broken-link-checker plugin runs a cron job on a regular basis that scans for links in post content, pings the links, and records those that 404 or time out. Unsurprisingly, this process can take a very long time. Recent debugging leads me to believe that it might be one of the culprits in our ongoing saga of open connections.

The cron job seems to be important to the way the plugin works (link rot happens over time) so I don't want to disable it completely. The plugin does have a few limiters built in for execution time and memory usage, but the limits are set too high for a site like the Commons. I'd suggest we filter them so that they can't be set over a certain amount. See https://boonesparty.commons.gc.cuny.edu/wp-admin/options-general.php?page=link-checker-settings etc.


Related issues

Related to CUNY Academic Commons - Feature #8987: Migrate away from wp-cronResolved2017-12-07

History

#1 Updated by Boone Gorges about 1 year ago

I've imposed some limits in https://github.com/cuny-academic-commons/cac/commit/ebabec4b679e0eb924af7b6bdc061bc357a63d6b.

Basically, the defaults of the plugin - max execution time, target load, etc - were clearly written for a situation where the plugin is running on a single WordPress site. The math doesn't scale at all with large installations, and the lock mechanisms aren't multisite-aware at all.

The adjustments I've made, when combined with each other, should reduce the likelihood of concurrent broken-link-checker cron jobs by about an order of magnitude. I'm going to put this into production right away, as we've had an uptick in performance issues over the last week. I'll continue to monitor.

#2 Updated by Matt Gold about 1 year ago

nice work, Boone -- thank you

#3 Updated by Boone Gorges about 1 year ago

#4 Updated by Boone Gorges about 1 year ago

  • Target version changed from 1.13.3 to 1.13.4

#5 Updated by Boone Gorges 12 months ago

  • Target version changed from 1.13.4 to 1.13.5

#6 Updated by Boone Gorges 11 months ago

  • Target version changed from 1.13.5 to 1.13.6

#7 Updated by Boone Gorges 11 months ago

  • Target version changed from 1.13.6 to 1.13.7

#8 Updated by Boone Gorges 10 months ago

  • Target version changed from 1.13.7 to 1.13.8

#9 Updated by Boone Gorges 10 months ago

  • Target version changed from 1.13.8 to 1.13.9

#10 Updated by Boone Gorges 9 months ago

  • Status changed from New to Resolved

I've reviewed a few sites where BLC is running, and it appears that the cron jobs are running (and completing) as expected. I think we can close.

Also available in: Atom PDF