Project

General

Profile

Actions

Bug #6746

closed

Bug #6731: Out-of-control queries on production server

bp-multiple-forum-post activity "backfilling" is sometimes causing infinite query loop

Added by Boone Gorges over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority name:
High
Assignee:
Daniel Jones
Category name:
Group Forums
Target version:
Start date:
2016-11-15
Due date:
% Done:

0%

Estimated time:
Deployment actions:

Description

See attached log.

I have a gut feeling that this may be at the root of #6731, because (a) significant changes to bp-multiple-forum-post were included in the 1.10.1 release, and (b) the pattern of use of this plugin's features is such that I can easily imagine a few people cross-posting at the same time, and bringing things to a screeching halt.

Dan, I've had a brief look at the logic here, and nothing jumped out at me as obviously wrong. I also tried reproducing the issue in both a local environment and on cdev, but I couldn't force the backfiller logic to break. That said, I remember that when I wrote the original version of this logic for bp-event-organiser, I tore my hair out about recursion, and I remember that you tore some of your own out as well during the process of porting it over :)

For the time being, I have commented out the block that's responsible for the backfill on the production site (the block that begins "if ($removed)"...). If my gut feeling is correct, this will fix the problem. We should know more after another day or two.

Could you please have a look? In particular, look over the following changesets, and see whether the logic (or load order?) inadvertently changed:
- https://github.com/cuny-academic-commons/cac/commit/57c83f7807bda6b374fb0397657419fc29f79b07
- https://github.com/cuny-academic-commons/cac/commit/162852233193a276771b28ccdc117f1c163b7a08
- https://github.com/cuny-academic-commons/cac/commit/ca162ca086db0477297be04d72bff74f91693094

Thank you!


Files

bp-multiple-forum-post.txt (3.04 MB) bp-multiple-forum-post.txt Boone Gorges, 2016-11-15 09:57 PM

Related issues

Related to CUNY Academic Commons - Bug #10706: bp-multiple-forum-post triggers query loop on AJAX heartbeatResolvedBoone Gorges2018-11-15

Actions
Actions #1

Updated by Boone Gorges over 7 years ago

  • Parent task set to #6731
Actions #2

Updated by Boone Gorges over 7 years ago

  • Target version changed from 1.10.2 to 1.10.3
Actions #3

Updated by Daniel Jones over 7 years ago

I've spent some time with this, and I can't seem to figure out the issue either. Did the issue stop happening now that you've commented out that code block? I can sometimes get a timeout issue when I'm on the My Commons page and I refresh it with cross-posted activities present, but it isn't every time and I can't figure out what exact sequence causes it, besides waiting for a while before reloading. Once it starts, the issues remains until I restart the server.

My best guess, given that the problem only comes up when I wait before reloading the page, is that the problem is somehow triggered when My Commons makes its AJAX call to see if there's any new activity items to display. Could you point me in the direction of that logic so I can test that theory out? Since we're messing with the $activity global it might be related.

Actions #4

Updated by Boone Gorges over 7 years ago

Thanks for investigating, Dan.

Did the issue stop happening now that you've commented out that code block?

I haven't seen it recur, but I also disabled the detailed logging that I'd had turned on at the time.

I can sometimes get a timeout issue when I'm on the My Commons page and I refresh it with cross-posted activities present, but it isn't every time and I can't figure out what exact sequence causes it, besides waiting for a while before reloading. Once it starts, the issues remains until I restart the server.

Yeah, this definitely sounds related - the fact that you have to restart the server suggests that there's a stuck httpd process.

My best guess, given that the problem only comes up when I wait before reloading the page, is that the problem is somehow triggered when My Commons makes its AJAX call to see if there's any new activity items to display. Could you point me in the direction of that logic so I can test that theory out? Since we're messing with the $activity global it might be related.

Sure thing. The main logic for my-commons is in wp-content/plugins/cac-bp-custom-includes/my-commons.php. The activity filtering functions here are the first places I'd start. I'm CCing Ray here, in case you need his feedback (he wrote My Commons).

Actions #5

Updated by Boone Gorges over 7 years ago

  • Target version changed from 1.10.3 to 1.10.4
Actions #6

Updated by Daniel Jones over 7 years ago

I think I've fixed this here, and that it was actually 2 separate but related issues with my design: https://github.com/cuny-academic-commons/cac/commit/a7393c51b281b998aea1fd861368b7d80af928e0

1. First, I think in an earlier version you had but in a failsafe depth-tested just to make sure the recursion (which I changed into a while loop) didn't go on forever. I'm not sure why I took that out, but I've added it back in. I've set the maximum number of loops to 3, which should be enough for most cases.
2. Second, there wasn't any support for AJAX queries. I've added in some things that respond specifically to AJAX use-cases: First, I don't bother with backfilling at all in AJAX queries, because it's hard to know whether or not we're trying to get to a specific number, or (as in the My Commons case) just showing some other posts. Second: I set it to automatically hide ANY duplicates queried using AJAX, whether or not the original activity item has also been queried. This has the drawback of maybe hiding posts from people who are in a group where something was cross-posted, but not in the group where the original was posted. Still, I think it's better than the alternative of people seeing a bunch of duplicates of posts because we can't tell from the AJAX query which activity items are already being displayed. This way of handling things might cause some issues with templates, etc., that use ONLY AJAX to display activity items, without first doing a regular query for them, but maybe that's the trade-off.

Let me know what you think of all this!

Actions #7

Updated by Boone Gorges over 7 years ago

  • Status changed from New to Resolved

Thanks, Dan!

. First, I think in an earlier version you had but in a failsafe depth-tested just to make sure the recursion (which I changed into a while loop) didn't go on forever. I'm not sure why I took that out, but I've added it back in. I've set the maximum number of loops to 3, which should be enough for most cases.

This looks good to me.

I don't bother with backfilling at all in AJAX queries

Yeah, this seems sensible, I guess.

I set it to automatically hide ANY duplicates queried using AJAX, whether or not the original activity item has also been queried.

This is a bit offputting, but I doubt anyone will notice unless they're looking. Let's ship it and see how it works.

Actions #8

Updated by Boone Gorges over 5 years ago

  • Related to Bug #10706: bp-multiple-forum-post triggers query loop on AJAX heartbeat added
Actions

Also available in: Atom PDF