CUNY Academic Commons

When a group crosspost is created, we add a scheduled task to create the new forum topics to the other groups. During this scheduled task, we create another scheduled task to post the activity items into the group feed. This change was made in #17241 to address issues with crossposted topics not being sent out via email.

So if there is a lag in the scheduled tasks system, there is a lag when group emails are scheduled to go out. (Also see #14276 about the scheduled tasks lag.) Right now, there is a lag of about 35 mins in the scheduled tasks queue, so that would explain the roughly 1 hour delay between the initial post in the forum and the group email sendout.

I'll look to clear out the scheduled tasks queue so the delay will be less noticeable.

Ray, thanks for looking into the issue of delays and for outlining how the task-scheduling works.

2. Why wasn't the email I received from the CUNY Manifold Group, where the forum post was originated?

Laurie, it looks like you're subscribed to the Manifold group with level 'All Email'. Is that correct? Can you confirm that you receive other email notifications from this group?

The tool we built to allow crossposting, bp-multiple-forum-post, has a mechanism to prevent users from receiving multiple emails about the same item when it's crossposted to multiple groups. The logic looks like this.
1. A new activity item is posted to group A. At this time, the item should be immediately added to the queue of users who are subscribed in group A. The emails should go out in the next few seconds, via the functionality in BPGES (our email notification plugin) that sends email asynchronously.
2. Just after the activity item is posted in A, bp-multiple-forum-post schedules a task for the delayed creation of duplicate forum topics in groups B, C, etc. As we've seen, these topics may not be created for a few minutes. When they are created, BPGES should trigger the creation of emails for those items as well.
3. Whenever BPGES attempts to send a forum topic notification, bp-multiple-forum-post intercedes. It says: if the user has already received an email for this topic in the current batch of emails, skip it. This means that the user should only receive a single notification for a given topic in a batch.

If you're a member of groups A, B, and C, my reading of the code is that you ought to receive a notification from A, and that in some cases (depending on whether it's run as part of the same BPGES batch) you would also receive a notification from B, but not C. Laurie, it sounds to me like you never received the one from A. Is that correct?

I will note that this whole duplicate-prevention mechanism seems pretty iffy. It was built before BPGES moved to an asynchronous system for the actual sending of emails, and so it doesn't keep persistent information about which users have received which emails. It should probably be reworked so that BPGES doesn't insert the item into its internal queue if it detects that a user already has a queued notification for that activity item (it already enforces uniqueness per activity_id+group_id https://github.com/boonebgorges/buddypress-group-email-subscription/blob/1971a19d9efb4097d5ace1952c630f38abc58c7d/admin.php#L647). Because of the fact that the notification from A is queued during the same pageload as the original forum post, it should always take precedence over crossposted groups. I'll look into this for a future release.

In the meantime, I'm not really sure how all of this explains what Laurie is seeing. If you are a member of A and B, you should always receive a notification from A, regardless of what happens in B.

Created #19980 to track the improvements to bp-multiple-forum-post.

Laurie, it looks like you're subscribed to the Manifold group with level 'All Email'. Is that correct? Can you confirm that you receive other email notifications from this group?

1. A new activity item is posted to group A. At this time, the item should be immediately added to the queue of users who are subscribed in group A. The emails should go out in the next few seconds, via the functionality in BPGES (our email notification plugin) that sends email asynchronously.

Based on when it was posted in Manifold group and when i received, there is lag here, and maybe that explains some of the issue?

2. Just after the activity item is posted in A, bp-multiple-forum-post schedules a task for the delayed creation of duplicate forum topics in groups B, C, etc. As we've seen, these topics may not be created for a few minutes. When they are created, BPGES should trigger the creation of emails for those items as well.

3. Whenever BPGES attempts to send a forum topic notification, bp-multiple-forum-post intercedes. It says: if the user has already received an email for this topic in the current batch of emails, skip it. This means that the user should only receive a single notification for a given topic in a batch.

If you're a member of groups A, B, and C, my reading of the code is that you ought to receive a notification from A, and that in some cases (depending on whether it's run as part of the same BPGES batch) you would also receive a notification from B, but not C. Laurie, it sounds to me like you never received the one from A. Is that correct?

That is correct.

In the meantime, I'm not really sure how all of this explains what Laurie is seeing. If you are a member of A and B, you should always receive a notification from A, regardless of what happens in B.

Yes, that was my sense, and why I was confused and wanted to raise here. I think the user should get the email from the group where the post originated, even if they receive it multiple times after that because of cross posting (though that is not ideal, and the current set up appears to try to tamp down on that)

Based on when it was posted in Manifold group and when i received, there is lag here, and maybe that explains some of the issue?

Yes, it could be, particularly if the BPGES asynch send routine kicked in after the bp-multiple-forum-post routine.

Yes, it could be, particularly if the BPGES asynch send routine kicked in after the bp-multiple-forum-post routine.

This is indeed the case. If a crossposted forum topic is made, then the BPGES async send routine is kicked off after the second scheduled task when the activity item is created. See https://redmine.gc.cuny.edu/issues/17241#note-6 . In an ideal world with no scheduled task lag, the email should be sent out two minutes after the crossposted forum topic.

Hi Laurie,

Thanks for the update. The email delay is caused by a backlog in our scheduled tasks system. Some of this can be attributed to a plugin update we made to the PDF Embedder plugin last week (see #20088).