CUNY Academic Commons

this works fine for a group.

Sorry to be pedantic, but by "this" I assume you mean "creating a new group with the Teaching purpose results in the group appearing on the Courses tab"

Can you please confirm the corresponding behavior on the GC production site before we decide whether this is a new issue?

I can confirm Colin's report. Tried creating a site with Teaching set as Primary Purpose on the Reclaim testing server. Site doesn't show up in Course Directory. When testing locally, the teaching site is listed in the Course Directory.

Just a guess, but it's possible that the scheduled task to create the Course after the teaching site is created is not being run on the Reclaim server.

That's ok, and yes, that's what I meant. The issue with sites (new and existing) not being added to Courses when Teaching primary purpose etc. is selected seems to be happening on GC production too. I am doing a lot of toggling around and testing though, so Ray or someone if you're able to help me confirm I'd appreciate it.

Thanks so much, Ray. The cron job appears to be the culprit. When I forced the pending jobs to complete, I saw a course "Ray is Teaching!" appear at http://commons.gc.cuny.edu/courses/, which I didn't see before the cron jobs ran. Colin, you didn't provide any details on what names etc I should be looking for (when reporting an issue, please provide these details or I can't debug properly) but I'm guessing your courses will appear there now too.

Ray, the Cavalcade queue on Reclaim appears to be way behind. I know we've faced this in the past. It looks like this stupid plugin is gumming up the works:

> select j.hook, l.* from wp_cavalcade_logs l join wp_cavalcade_jobs j on j.id = l.job
order by l.id desc limit 100;
+----------------------------+-----------+---------+-----------+---------------------+---------+
| hook                       | id        | job     | status    | timestamp           | content |
+----------------------------+-----------+---------+-----------+---------------------+---------+
| action_scheduler_run_queue | 151243795 | 3801309 | completed | 2025-01-13 20:02:43 |         |
| action_scheduler_run_queue | 151243793 | 3790801 | completed | 2025-01-13 20:02:42 |         |
| action_scheduler_run_queue | 151243791 | 3751995 | completed | 2025-01-13 20:02:42 |         |
| action_scheduler_run_queue | 151243789 | 3741355 | completed | 2025-01-13 20:02:41 |         |
| action_scheduler_run_queue | 151243787 | 3783377 | completed | 2025-01-13 20:02:40 |         |
| action_scheduler_run_queue | 151243785 | 3746357 | completed | 2025-01-13 20:02:40 |         |
| action_scheduler_run_queue | 151243783 | 3715827 | completed | 2025-01-13 20:02:39 |         |
| action_scheduler_run_queue | 151243781 | 3801307 | completed | 2025-01-13 20:02:38 |         |
| action_scheduler_run_queue | 151243779 | 3799849 | completed | 2025-01-13 20:02:38 |         |
| action_scheduler_run_queue | 151243777 | 3751151 | completed | 2025-01-13 20:02:37 |         |
| action_scheduler_run_queue | 151243775 | 3799847 | completed | 2025-01-13 20:02:36 |         |
| action_scheduler_run_queue | 151243773 | 3749649 | completed | 2025-01-13 20:02:35 |         |
| action_scheduler_run_queue | 151243771 | 3786213 | completed | 2025-01-13 20:02:35 |         |
| action_scheduler_run_queue | 151243769 | 3783375 | completed | 2025-01-13 20:02:34 |         |
| action_scheduler_run_queue | 151243767 | 3764895 | completed | 2025-01-13 20:02:33 |         |
| action_scheduler_run_queue | 151243765 | 3690469 | completed | 2025-01-13 20:02:33 |         |
| action_scheduler_run_queue | 151243763 | 3778963 | completed | 2025-01-13 20:02:32 |         |
| action_scheduler_run_queue | 151243761 | 3785227 | completed | 2025-01-13 20:02:32 |         |
| action_scheduler_run_queue | 151243759 | 3802621 | completed | 2025-01-13 20:02:31 |         |
| action_scheduler_run_queue | 151243757 | 3767947 | completed | 2025-01-13 20:02:30 |         |
| action_scheduler_run_queue | 151243755 | 3770467 | completed | 2025-01-13 20:02:30 |         |
| action_scheduler_run_queue | 151243753 | 3751649 | completed | 2025-01-13 20:02:29 |         |
| action_scheduler_run_queue | 151243751 | 3797163 | completed | 2025-01-13 20:02:28 |         |
| action_scheduler_run_queue | 151243749 | 3775305 | completed | 2025-01-13 20:02:28 |         |
| action_scheduler_run_queue | 151243747 | 3742193 | completed | 2025-01-13 20:02:27 |         |
| action_scheduler_run_queue | 151243745 | 3717513 | completed | 2025-01-13 20:02:26 |         |
| action_scheduler_run_queue | 151243743 | 3722509 | completed | 2025-01-13 20:02:26 |         |
| action_scheduler_run_queue | 151243741 | 3801305 | completed | 2025-01-13 20:02:26 |         |
| action_scheduler_run_queue | 151243739 | 3794391 | completed | 2025-01-13 20:02:24 |         |
| action_scheduler_run_queue | 151243737 | 3776645 | completed | 2025-01-13 20:02:24 |         |
| action_scheduler_run_queue | 151243735 | 3728507 | completed | 2025-01-13 20:02:23 |         |
| action_scheduler_run_queue | 151243733 | 3773547 | completed | 2025-01-13 20:02:22 |         |
| action_scheduler_run_queue | 151243731 | 3753265 | completed | 2025-01-13 20:02:22 |         |
| action_scheduler_run_queue | 151243729 | 3797157 | completed | 2025-01-13 20:02:21 |         |
| action_scheduler_run_queue | 151243727 | 3747881 | completed | 2025-01-13 20:02:21 |         |
| action_scheduler_run_queue | 151243725 | 3803873 | completed | 2025-01-13 20:02:20 |         |
| action_scheduler_run_queue | 151243723 | 3716439 | completed | 2025-01-13 20:02:20 |         |
| action_scheduler_run_queue | 151243721 | 3792021 | completed | 2025-01-13 20:02:19 |         |
| action_scheduler_run_queue | 151243719 | 3767943 | completed | 2025-01-13 20:02:18 |         |
| action_scheduler_run_queue | 151243717 | 3785221 | completed | 2025-01-13 20:02:17 |         |
| action_scheduler_run_queue | 151243715 | 3798349 | completed | 2025-01-13 20:02:17 |         |
| jetpack_sync_cron          | 151243713 | 3789897 | completed | 2025-01-13 20:02:17 |         |
| action_scheduler_run_queue | 151243711 | 3783371 | completed | 2025-01-13 20:02:16 |         |
| action_scheduler_run_queue | 151243709 | 3766623 | completed | 2025-01-13 20:02:15 |         |
....

I see in https://redmine.gc.cuny.edu/issues/20088#note-3 that you've previously just cleared out these jobs. I've just done the same (delete from wp_cavalcade_jobs where hook = 'action_scheduler_run_queue'). The plugin continues to create new instances of its hook (~2 minutes later):

> select count(*) from wp_cavalcade_jobs where hook = 'action_scheduler_run_queue';
+----------+
| count(*) |
+----------+
|       49 |
+----------+
1 row in set (0.000 sec)

We previously throttled it in https://redmine.gc.cuny.edu/issues/17672 though we might have to modify this as well. The events get scheduled now, which means they jump in the line ahead of other pending items. Ray, do you see any harm if I increase this stupid plugin's cron job interval and also newly scheduled tasks to take place a few minutes in the future?

For now, Cavalcade is working its way through the queue. Ray, what's a good way to see how it's making progress? When I look at the next pending tasks:

> select * from wp_cavalcade_jobs where  status = 'waiting' order by nextrun ASC limit 10;

they're constantly changing, which means the daemon is hard at work. But what should I be watching to see if the queue is decreasing? select count(*) from wp_cavalcade_jobs where status = 'waiting'; actually seems to be going up over time. Maybe what I want is:

> select min(nextrun) from wp_cavalcade_jobs where status = 'waiting';
+---------------------+
| min(nextrun)        |
+---------------------+
| 2025-01-13 20:07:21 |
+---------------------+
1 row in set (0.062 sec)

which tells me that the oldest pending task is a couple minutes old UTC?

I see in https://redmine.gc.cuny.edu/issues/20088#note-3 that you've previously just cleared out these jobs. I've just done the same (delete from wp_cavalcade_jobs where hook = 'action_scheduler_run_queue'). The plugin continues to create new instances of its hook (~2 minutes later)

It looks like sites running The Events Calendar should be the main culprit behind the Action Scheduler tasks. I audited the code for TEC awhile back and it looks like they only use it for migration routines. The ideal scenario would be for TEC to only use Action Scheduler if a migration is necessary, not on every page load: https://github.com/cuny-academic-commons/cac/blob/da3ad43a95f0a6b6ed38c3f9f9cf21685227e262/wp-content/plugins/the-events-calendar/src/Events/Custom_Tables/V1/Migration/Provider.php#L318

Ray, do you see any harm if I increase this stupid plugin's cron job interval and also newly scheduled tasks to take place a few minutes in the future?

Feel free to adjust as necessary.

Ray, what's a good way to see how it's making progress? When I look at the next pending tasks:

See https://redmine.gc.cuny.edu/issues/14276#note-12 where I've outlined what I've done in the past.

Your min() query is basically the same as my select * from wp_cavalcade_jobs where status = "waiting" and nextrun < utc_timestamp() order by nextrun asc limit 10;

I am now seeing in the Courses tab the sites I was unable to get to appear there earlier. I also received a bunch of the "teaching intro" emails that we send to the admin of new teaching-purpose sites. Sorry for not including those details in my initial report.

I am doing a lot of toggling around and testing though, so Ray or someone if you're able to help me confirm I'd appreciate it.

Our scheduled task runner on GC production can sometimes run a bit behind. I see that there is a site that Colin set up -- "gobble teaching site" -- that is set to be a course on GC production:

mysql> select * from wp_cavalcade_jobs where hook = 'cac_trigger_site_course_sync' and id = 3461745;
+---------+------+------------------------------+------------------------------+---------------------+---------------------+----------+----------+---------+
| id      | site | hook                         | args                         | start               | nextrun             | interval | schedule | status  |
+---------+------+------------------------------+------------------------------+---------------------+---------------------+----------+----------+---------+
| 3461745 |    1 | cac_trigger_site_course_sync | a:1:{s:7:"site_id";i:36734;} | 2025-01-13 19:56:18 | 2025-01-13 19:56:18 |     NULL | NULL     | waiting |
+---------+------+------------------------------+------------------------------+---------------------+---------------------+----------+----------+---------+
1 row in set (0.01 sec)

According to the scheduled task logs, the scheduled task completed for that one:

mysql> select * from wp_cavalcade_logs where job = 3461745;
+-----------+---------+-----------+---------------------+---------+
| id        | job     | status    | timestamp           | content |
+-----------+---------+-----------+---------------------+---------+
| 147727925 | 3461745 | completed | 2025-01-13 20:07:57 |         |
+-----------+---------+-----------+---------------------+---------+
1 row in set (0.01 sec)

But the site does not show up in the Course Directory, however the site was saved as a course because if you do a search for it, it shows up in the directory: https://commons.gc.cuny.edu/courses/?filters%5Bkeyword%5D=gobble&is-order-by-refresh=&order-by=semester

I'm tempted to flush the object cache, but that might degrade server performance.

But the site does not show up in the Course Directory, however the site was saved as a course because if you do a search for it, it shows up in the directory: https://commons.gc.cuny.edu/courses/?filters%5Bkeyword%5D=gobble&is-order-by-refresh=&order-by=semester

False alarm! The Gobble site shows up on page 3 of the Course Directory :) There appears to be a sorting bug. Fall courses currently show up before Spring courses; Fall courses should follow Spring and Summer ones.

Thanks, Ray! And hope you enjoyed my tendency to put an odd word in the title of my test sites so they're easier to search for...