CUNY Academic Commons

I spent some time analyzing this. I don't see any out-and-out bugs. Most of the queries are a consequence of wp_delete_post(), which is pretty inefficient. It would be possible to reduce the number of required queries by introducing a tool that deletes items in bulk, but this would be a pretty large amount of work. Another mitigation would be to force the plugin to run its routine more frequently, so that it was processing fewer posts each time. Again, this would take a pretty large amount of work.

It looks like there might be some cases where the wp-rss-multi-importer deletion routine times out, because gcenglish has a fair number of items that ought to have been deleted by now, but have not been. I just ran a custom script to clean them out. This may help with memory overhead in the future.

All of the feeds on gcenglish were added months (or years) ago, so I'm fairly sure this is not the source of the problem described in #6731. So I think I've done all I can reasonably do here.

During recent debugging, it became pretty clear that this plugin is a culprit in some recent resource issues.

I've looked at a couple other RSS aggregator plugins, and it's possible that we could migrate. Specifically, https://wordpress.org/plugins/wp-rss-aggregator/ is quite a bit smarter than wp-rss-multi-importer in terms of the way it processes imports. Namely, during each import run, it schedules separate cron jobs for each subscribed feed, which greatly decreases the likelihood of timeouts and other resource issues.

The problem with migrating is that the data has to be moved over, and the front-end functionality has to be duplicated somehow. It appears that the only site actively using the plugin is gcenglish, and they use the front-end widget, but wp-rss-aggregator either doesn't have a widget or only has it as a paid add-on, and even then it would likely work quite differently. The amount of effort required to migrate gcenglish to the new plugin might therefore be fairly large. Another option, especially since wp-rss-multi-import has been removed from the wordpress.org repo and thus is no longer receiving updates, is to fork the plugin and fix it so that it does one-cron-job-per-feed. I have a feeling that this will be easier in the short term. We can then hide wp-rss-multi-importer for new installations, and perhaps install wp-rss-aggregator if anyone requests the functionality in the future.

I'm going to reopen this ticket and move it into a milestone so I can take care of some of these tasks.

In https://github.com/cuny-academic-commons/cac/commit/b8db29c94e306a79da7b6a339988d8e637730877 I disabled the plugin for future activation.

In https://github.com/cuny-academic-commons/cac/commit/9a1b83b9a2469ceeaff937b3d5517528396e99b6 I split the cron job into per-feed events, which should lighten the load.