Bug #3164
closedDuplicate RBE messages
0%
Description
Per this ticket -- http://redmine.gc.cuny.edu/issues/3163 -- I received two email notifications for a message I posted to the forums. So I deactivated/reactivated RBE, which in the past has solved this issue.
But I just posted a reply to another message. My message posted, but then I received a second message saying the following:
Subject: Your Reply By Email message could not be posted [CUNY Academic Commons]
Hi there,
Your forum reply:
"Sure, that would be fine. Might be best for you to contact Jason Lau in IT
directly (cc me, please). Don't think it's the cloud; don't know what
version it is. Jason's email is jlau@gc.cuny.edu"Could not be posted because you have already posted the same message in the forum topic you were attempting to reply to.
We apologize for any inconvenience this may have caused.
So, it seems like there is still an issue with double posting.
Perhaps related: our sys admin may have restarted the serve to install a patch this morning.
Updated by Boone Gorges over 10 years ago
- Priority name changed from Normal to High
- Target version set to 1.6.2
Ray, can I ask you to take a look at this in the near future? At least have a look in the logs to see if there's anything obvious that jumps out. Double content posting is a fairly serious UX problem.
Updated by Raymond Hoh over 10 years ago
The problem looks like it occurs due to overlapping IMAP inbox connections.
I've been working on a new version of RBE for awhile now.
This new version includes a new operating mode apart from the current IMAP implementation. It uses inbound POST requests (via an external provider like Mandrill). (View this ticket for some background info.) Using the new inbound email mode will get rid of these IMAP issues, however this new mode requires at least an account on Mandrill and a custom subdomain.
What I can do is push what I have to a new branch on CAC for testing on cdev. I'll also create a temporary account on Mandrill and use a dummy subdomain for the inbound email requests for now.
Give me a few days to review the code and I'll try and make the necessary changes for testing on cdev.
Updated by Boone Gorges over 10 years ago
- Target version changed from 1.6.2 to 1.6.3
Updated by Raymond Hoh over 10 years ago
Just to update, I've added a new branch called "rbe-2014" on Github and I've switched cdev to run this new branch.
This is a major update. A lot of code has been rewritten. It supports the new inbound email mode as outlined here. IMAP mode has also been overhauled to use transients, which should result in less duplicate message occurrences.
Some administrative stuff:
- I don't have access on cdev to administer plugins. Can someone deactivate RBE on cdev and reactivate it or provide me with access to administrate plugins on cdev?
- We can't test inbound email mode on cdev because it is password-protected and Mandrill requires sites to be publicly-accessible. Not a big deal, we can test the improvements to IMAP mode and see if it works.
Updated by Boone Gorges over 10 years ago
Awesome, thanks, Ray!
I've just made you a super admin on cdev, so test away.
Re Mandrill vs IMAP: Let's stick with IMAP for now since it's a known quantity, and since it's already configured. Maybe Mandrill is something we can toy with for a future release (at which point we can open up an endpoint on cdev or something like that, for testing).
Can you be sure to document any manual changes that need to be made at deployment on https://github.com/castiron/cac/wiki/Release-ACTION_REQUIRED-list? Even something like toggling the plugin. And let me know what/if you need me to review something.
Updated by Boone Gorges over 10 years ago
- Target version changed from 1.6.3 to 1.6.4
Updated by Boone Gorges over 10 years ago
- Target version changed from 1.6.4 to 1.6.5
Updated by Boone Gorges over 10 years ago
- Target version changed from 1.6.5 to 1.6.6
Updated by Boone Gorges over 10 years ago
- Target version changed from 1.6.6 to 1.6.7
Updated by Boone Gorges over 10 years ago
- Status changed from Assigned to Resolved
The updates to RBE in https://github.com/castiron/cac/commit/76735ae215c5a5f0aaf75436fb7b19a0d5b6786a should address this problem. I'm closing this ticket, to clear the milestone. If more problems arise, feel free to reopen with details.
Updated by Raymond Hoh over 10 years ago
- Status changed from Resolved to Assigned
- Target version changed from 1.6.7 to 1.6.8
Reopening.
It looks like the duplicate connection issue is still happening (and actually is a little worse during my previous refactor).
To summarize the problem, if a site has a ton of concurrent users per second during the time when a new IMAP inbox connection is needed, duplicate IMAP connections will occur, which could cause duplicate posts.
The problems didn't arise during my testing on NYCDH because the web traffic on NYCDH is lower compared to CAC. Now, that the latest RBE code is running on CAC, the duplicate connection issue is presenting itself.
My fault for not doing better stress testing locally. I spent some time late last night stress testing RBE using OpenWebLoad to simulate concurrent users and have coded a workaround. The results are better now, but not 100% foolproof.
See commit 7c1f72cc02.
The IMAP locking system now uses the filesystem instead of the database. In my testing, RBE can now handle up to 12 concurrent users on the same second without launching a duplicate connection. By that I mean, the site has to have more than 12 users (or page loads) on the same second when RBE needs to launch a new connection to the inbox. These two conditions should happen rarely.
WP-cron also runs into these race conditions and is not perfect, but I've tried to mitigate this as best as I can.
Boone: What do you think about a 1.6.7.1 release?
Updated by Boone Gorges over 10 years ago
- Status changed from Assigned to Resolved
- Target version changed from 1.6.8 to 1.6.7.1
Thanks for the diligence, Ray. I'm going to do a 1.6.7.1 release right now. Let's reopen this ticket as necessary.
Updated by Raymond Hoh over 10 years ago
Thanks for doing a 1.6.7.1 release, Boone!