CUNY Academic Commons

Yes, most likely this is due to the new bbPress hardening changes.

The crosspost feature works by setting up a cronjob to post the topics later. It's most likely that since the cronjob does not use the authenticated user, that the topics are failing to post due to the new hardening changes. Will see how we can work around this.

I tested locally and the multiple forum post functionality is working, so I checked production and the cronjob to duplicate the forum post hasn't run yet because there is a cronjob lag on production of about six days.

I'm working on clearing the log.

Boone, can you tell if Cavalcade is running on production? Because it doesn't look like the queue is going down at all.

According to the Cavalcade database, the next job to run is 2026-09-22 21:47:09, which is six days ago.

Hm. It appears to be running. Here's a couple checks, about 5 seconds between:

litespeed@node15377-cunyacprod-1 ~ $ ps aux | grep caval
litespe+ 1033877  0.4  0.0 261000 39828 ?        SNs  21:08   0:00 php /usr/local/bin/cavalcade
litespe+ 1048520 81.5  0.1 287812 128816 ?       RN   21:10   0:01 php /usr/local/bin/wp cavalcade run 6758431 --url=abifacultyseminarseriessummer23.commons.gc.cuny.edu/
litespe+ 1048707 53.0  0.0 274204 64376 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 6740803 --url=senses.commons.gc.cuny.edu/
litespe+ 1048708 42.0  0.0 269948 52372 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 6094975 --url=zainabkhan73.commons.gc.cuny.edu/
litespe+ 1048709 31.0  0.0 269948 52216 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 6075797 --url=ktg2004.commons.gc.cuny.edu/
litespe+ 1048720  0.0  0.0   7068  2040 pts/2    S+   21:10   0:00 grep --color=auto caval
litespeed@node15377-cunyacprod-1 ~ $ ps aux | grep caval
litespe+ 1033877  0.4  0.0 261000 39828 ?        SNs  21:08   0:00 php /usr/local/bin/cavalcade
litespe+ 1049272  177  0.1 287808 118356 ?       RN   21:10   0:01 php /usr/local/bin/wp cavalcade run 6748173 --url=bmccmusic.commons.gc.cuny.edu/
litespe+ 1049532  0.0  0.1 276636 80056 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 6744037 --url=transformation.commons.gc.cuny.edu/
litespe+ 1049534  0.0  0.1 274588 72264 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 6750919 --url=help.commons.gc.cuny.edu/
litespe+ 1049535  0.0  0.0 274204 66028 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 6641181 --url=brownf26110.commons.gc.cuny.edu/
litespe+ 1049648  0.0  0.0   7068  2052 pts/2    S+   21:10   0:00 grep --color=auto caval
litespeed@node15377-cunyacprod-1 ~ $ ps aux | grep caval
litespe+ 1033877  0.4  0.0 261000 39828 ?        SNs  21:08   0:00 php /usr/local/bin/cavalcade
litespe+ 1050245  105  0.2 380932 196172 ?       SN   21:10   0:03 php /usr/local/bin/wp cavalcade run 6742525 --url=newtestsite.commons.gc.cuny.edu/
litespe+ 1050497  108  0.2 298220 159656 ?       RN   21:10   0:02 php /usr/local/bin/wp cavalcade run 6739689 --url=help.commons.gc.cuny.edu/
litespe+ 1050507 96.0  0.2 368804 152712 ?       RN   21:10   0:01 php /usr/local/bin/wp cavalcade run 6742435 --url=acertdev2.commons.gc.cuny.edu/
litespe+ 1050538 94.0  0.2 373588 148344 ?       RN   21:10   0:01 php /usr/local/bin/wp cavalcade run 6739827 --url=tokyo2021.commons.gc.cuny.edu/
litespe+ 1050686 88.0  0.1 278684 85172 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 6742821 --url=ccnypsaproject.commons.gc.cuny.edu/
litespe+ 1050721  0.0  0.1 276636 76700 ?        RN   21:10   0:00 php /usr/local/bin/wp cavalcade run 4304253 --url=ausher.commons.gc.cuny.edu/
litespe+ 1050869  0.0  0.0   7068  2052 pts/2    S+   21:10   0:00 grep --color=auto caval

So you can see that the daemon is indeed working through the queue, though I can't say from this alone whether it's enough to catch up.

Thanks for confirming, Laurie. Can you also check the site cloning issue to see if that is resolved as well?

The cronjob queue is now clear.

Boone, I made a change to Editoria11y on production to change their cron job's batch size from 1000 to 100: https://github.com/cuny-academic-commons/cac/blob/f892c9950d621354f4c940357a4e4488b070233c/wp-content/plugins/editoria11y-accessibility-checker/src/Installer.php#L75. I also made a change so their cronjob only runs every 30 mins instead of every 5 mins. Both of these changes haven't been committed yet.

I've also posted an issue on Editoria11y's Github repo about their cronjob, aided by an AI audit: https://github.com/itmaybejj/editoria11y-wp/issues/61.

Let's keep an eye on Cavalcade and see if there any more lockups over the next day or so.

Update: Dropping the batch size didn't help as I noticed a 2-hr hanging task a couple of hours after the queue was cleared. I've since killed that task.

Thanks to you both for looking into this.

I think we already have done this but Editorially could be removed as one of the plugin that comes pre-installed on every site during site creation (since it sounds like it may be part of the issue)

I've added the most recent update for `editoria11y` (v3.0.5) to production for testing and there are still some problems during editoria11y's cronjob routine. I've left some comments for the developer here: https://github.com/itmaybejj/editoria11y-wp/issues/61#issuecomment-5908103075. Boone, when you have a moment, would be great to get another set of eyes on this.

Overnight, there were three hanging jobs.

The editoria11y developer wrote back and asked us to try the latest update (v3.0.6), so I'm about to add the new changes to production for testing.

We had a user report that "scheduled posts are not going live as scheduled" could the cron jobs also be causing this? And if so, this problem with scheduled posts would also now be resolved, correct?

Hi Laurie,

There is currently no cronjob lag so the scheduled post should have been sent out by now. Can you provide any other details about the scheduled posts?

Boone, I'm going to quote your comment from https://redmine.gc.cuny.edu/issues/25528#note-7 as that was posted in the wrong ticket:

Thanks, Ray. I've just been reading through the Editoria11y thread. I have to say it's pretty dense and I'm not sure I have a clear grasp on it. I don't fully understand https://github.com/itmaybejj/editoria11y-wp/commit/768e439f54fc7b0ebac2a1d36419ecfd035458a4. I guess the idea is that on a setup like ours (where Cavalcade is a separate appliance) there might be some concurrency problems or race conditions where running wp_unschedule_event() wouldn't immediately cause wp_next_scheduled() to return false? This seems like a bug in Cavalcade as much as in Editoria11y. But yeah, given that we haven't seen this particular type of problem before, I think it's cool to just go with the fix and see what happens.

The latest 3.0.6 change has addressed the cronjob hang issue. I'll let the cronjobs run overnight to be sure, but I think we're good.

The developer also mentioned giving us a free license for a year: https://github.com/itmaybejj/editoria11y-wp/issues/61#issuecomment-5910128681. I said I'll talk things over with the team.

As for the bug itself, there is definitely an issue in Cavalcade. wp_unschedule_event() returns true in Cavalcade: https://github.com/humanmade/Cavalcade/blob/b3158545f6068e21dbb873202759d8a0340347cb/inc/connector/namespace.php#L271-L274

However, editoria11y uses wp_unschedule_event() during the actual cronjob, and Cavalcade doesn't allow a current cronjob to delete itself: https://github.com/humanmade/Cavalcade/blob/b3158545f6068e21dbb873202759d8a0340347cb/inc/class-job.php#L85-L87, so Cavalcade should really be returning the result from $job->delete() instead of true . Since the previous version of editoria11y was using wp_unschedule_event() in a while() loop, the loop never ended since the event was never actually unscheduled in Cavalcade. I've opened a PR at the Cavalcade repo here: https://github.com/humanmade/Cavalcade/pull/131.

editoria11y worked around this bug by no longer using the while() loop and using wp_unschedule_hook() instead of wp_unschedule_event().

Ray, thanks a million for following up on this (and sorry for posting in the wrong ticket, d'oh). Your patch for Cavalcade looks correct to me.

I guess we could take a look at the "CSA" license. I'm a bit wary of becoming reliant on third-party tools, community-provided tools, particularly if we are going to lose access a year from now without purchasing a renewing license. That's not to say I don't want to support the Editoria11y project! Just that I don't want to get locked in. That said, I guess it would be fine to have a look at what's available once we have the license. We'd really need to do this with help from Laurie and other members of the team who have a clearer grasp of how editoria11y is currently used, and how it could potentially be used further, by real Commons members.

I've committed the editoria11y change to 2.7.x branch in https://github.com/cuny-academic-commons/cac/commit/e45fc6a133e6ac928da93a06c30b36abb6bc07e1, but haven't cherry-picked it to production yet.

About the CSA license, I agree that we shouldn't be too reliant on third-party tools, which is why I mentioned I would discuss it with the team. Here's what is mentioned on the editoria11y website about the CSA license: https://editoria11y.com/features/#csa. It doesn't look like there are any additional user-facing benefits at this time, but it looks like there might be in the project roadmap.

Just want to clarify (and sorry, I may be glazed over from ticket catchup) but is there a direct way that buying a CSA license would help us with cronjob issue? Or is it more a political or access cost, in that being a license holder might allow us more access to the developer for future fixes or even feature input in the future? No "user-facing benefit" as Ray said gives me pause.

I agree it would be great to look further at the usage and importance of editorially these days in our teaching projects in particular before proceeding. I know it's given us headaches before, but also that it gets use and is important for accessibility compliance.

Thanks, Ray - I think this confirms that we don't really need to spend time looking into this CSA membership business for the time being.

Just want to clarify (and sorry, I may be glazed over from ticket catchup) but is there a direct way that buying a CSA license would help us with cronjob issue?

No. The cron issue was a bug in the plugin, and it's been fixed. The developer is really nice about dealing with our bug reports, regardless of whether we have this license.

Ok Boone, thanks. I will put this on our agenda for an upcoming meeting just so we're all on the same page.

We discussed in the meeting today removing Editorially from our teaching and digital plugin packages, not so much due to the issues above (and perhaps this should be another ticket, but keeping this continuous for now) but rather because the plugin overlays too much on sites and is difficult to understand for many users.

Chris mentioned that the plugin's author is working with a colleague at SUNY Oneonta on some improvements, so perhaps we revisit this once we see what those are.

In the meantime, I started #25901 for our offshoot discussion about better handling attempts to upload non-OCR PDFs.

Yes, removing from the plugin packages sounds good.

So the default for editorially will be available for activation, but not auto-activated on any sites.