Project

General

Profile

Actions

Bug #9865

closed

Broken Link Checker cron jobs running long

Added by Boone Gorges over 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority name:
Normal
Assignee:
Category name:
-
Target version:
Start date:
2018-05-31
Due date:
% Done:

0%

Estimated time:
Deployment actions:

Description

Previously: #6731

The broken-link-checker plugin runs a cron job on a regular basis that scans for links in post content, pings the links, and records those that 404 or time out. Unsurprisingly, this process can take a very long time. Recent debugging leads me to believe that it might be one of the culprits in our ongoing saga of open connections.

The cron job seems to be important to the way the plugin works (link rot happens over time) so I don't want to disable it completely. The plugin does have a few limiters built in for execution time and memory usage, but the limits are set too high for a site like the Commons. I'd suggest we filter them so that they can't be set over a certain amount. See https://boonesparty.commons.gc.cuny.edu/wp-admin/options-general.php?page=link-checker-settings etc.


Related issues

Related to CUNY Academic Commons - Feature #8987: Migrate away from wp-cronResolvedBoone Gorges2017-12-07

Actions
Actions #1

Updated by Boone Gorges over 6 years ago

I've imposed some limits in https://github.com/cuny-academic-commons/cac/commit/ebabec4b679e0eb924af7b6bdc061bc357a63d6b.

Basically, the defaults of the plugin - max execution time, target load, etc - were clearly written for a situation where the plugin is running on a single WordPress site. The math doesn't scale at all with large installations, and the lock mechanisms aren't multisite-aware at all.

The adjustments I've made, when combined with each other, should reduce the likelihood of concurrent broken-link-checker cron jobs by about an order of magnitude. I'm going to put this into production right away, as we've had an uptick in performance issues over the last week. I'll continue to monitor.

Actions #2

Updated by Matt Gold over 6 years ago

nice work, Boone -- thank you

Actions #3

Updated by Boone Gorges over 6 years ago

Actions #4

Updated by Boone Gorges over 6 years ago

  • Target version changed from 1.13.3 to 1.13.4
Actions #5

Updated by Boone Gorges over 6 years ago

  • Target version changed from 1.13.4 to 1.13.5
Actions #6

Updated by Boone Gorges over 6 years ago

  • Target version changed from 1.13.5 to 1.13.6
Actions #7

Updated by Boone Gorges over 6 years ago

  • Target version changed from 1.13.6 to 1.13.7
Actions #8

Updated by Boone Gorges over 6 years ago

  • Target version changed from 1.13.7 to 1.13.8
Actions #9

Updated by Boone Gorges about 6 years ago

  • Target version changed from 1.13.8 to 1.13.9
Actions #10

Updated by Boone Gorges about 6 years ago

  • Status changed from New to Resolved

I've reviewed a few sites where BLC is running, and it appears that the cron jobs are running (and completing) as expected. I think we can close.

Actions

Also available in: Atom PDF