Bug #3164: Duplicate RBE messages - CUNY Academic Commons - CUNY Graduate Center - Project Tracking System

Custom queries

. 2.4.11 open
2.5.0 open
2.5.0 open (by category)
2.5.0 open (by tracker)
High Priority Issues by User
Inbox (no milestone)
My Assigned Issues (by milestone)
My Watched Issues (by milestone)
Not Tracked - grouped by assignee
Not tracked (milestone-independent)

Actions

Copy link

Bug #3164

closed

Duplicate RBE messages

Added by Matt Gold over 10 years ago. Updated over 10 years ago.

Status:

Resolved

Priority name:

High

Assignee:

Raymond Hoh

Category name:

Reply By Email

Target version:

1.6.7.1

Start date:

2014-04-10

Due date:

% Done:

Estimated time:

Deployment actions:

Description

Per this ticket -- http://redmine.gc.cuny.edu/issues/3163 -- I received two email notifications for a message I posted to the forums. So I deactivated/reactivated RBE, which in the past has solved this issue.

But I just posted a reply to another message. My message posted, but then I received a second message saying the following:

Subject: Your Reply By Email message could not be posted [CUNY Academic Commons]

Hi there,

Your forum reply:

"Sure, that would be fine. Might be best for you to contact Jason Lau in IT
directly (cc me, please). Don't think it's the cloud; don't know what
version it is. Jason's email is jlau@gc.cuny.edu"

Could not be posted because you have already posted the same message in the forum topic you were attempting to reply to.

We apologize for any inconvenience this may have caused.

So, it seems like there is still an issue with double posting.

Perhaps related: our sys admin may have restarted the serve to install a patch this morning.

History
Notes
Property changes

Actions

Copy link

Updated by Boone Gorges over 10 years ago

Priority name changed from Normal to High
Target version set to 1.6.2

Ray, can I ask you to take a look at this in the near future? At least have a look in the logs to see if there's anything obvious that jumps out. Double content posting is a fairly serious UX problem.

Actions

Copy link

Updated by Raymond Hoh over 10 years ago

The problem looks like it occurs due to overlapping IMAP inbox connections.

I've been working on a new version of RBE for awhile now.

This new version includes a new operating mode apart from the current IMAP implementation. It uses inbound POST requests (via an external provider like Mandrill). (View this ticket for some background info.) Using the new inbound email mode will get rid of these IMAP issues, however this new mode requires at least an account on Mandrill and a custom subdomain.

What I can do is push what I have to a new branch on CAC for testing on cdev. I'll also create a temporary account on Mandrill and use a dummy subdomain for the inbound email requests for now.

Give me a few days to review the code and I'll try and make the necessary changes for testing on cdev.

Actions

Copy link

Updated by Boone Gorges over 10 years ago

Target version changed from 1.6.2 to 1.6.3

Actions

Copy link

Updated by Raymond Hoh over 10 years ago

Just to update, I've added a new branch called "rbe-2014" on Github and I've switched cdev to run this new branch.

This is a major update. A lot of code has been rewritten. It supports the new inbound email mode as outlined here. IMAP mode has also been overhauled to use transients, which should result in less duplicate message occurrences.

Some administrative stuff:
- I don't have access on cdev to administer plugins. Can someone deactivate RBE on cdev and reactivate it or provide me with access to administrate plugins on cdev?
- We can't test inbound email mode on cdev because it is password-protected and Mandrill requires sites to be publicly-accessible. Not a big deal, we can test the improvements to IMAP mode and see if it works.

Actions

Copy link

Updated by Boone Gorges over 10 years ago

Awesome, thanks, Ray!

I've just made you a super admin on cdev, so test away.

Re Mandrill vs IMAP: Let's stick with IMAP for now since it's a known quantity, and since it's already configured. Maybe Mandrill is something we can toy with for a future release (at which point we can open up an endpoint on cdev or something like that, for testing).

Can you be sure to document any manual changes that need to be made at deployment on https://github.com/castiron/cac/wiki/Release-ACTION_REQUIRED-list? Even something like toggling the plugin. And let me know what/if you need me to review something.

Actions

Copy link

Updated by Boone Gorges over 10 years ago

Target version changed from 1.6.3 to 1.6.4

Actions

Copy link

Updated by Boone Gorges over 10 years ago

Target version changed from 1.6.4 to 1.6.5

Actions

Copy link

Updated by Boone Gorges over 10 years ago

Target version changed from 1.6.5 to 1.6.6

Actions

Copy link

Updated by Boone Gorges over 10 years ago

Target version changed from 1.6.6 to 1.6.7

Actions

Copy link

#10

Updated by Boone Gorges over 10 years ago

Status changed from Assigned to Resolved

The updates to RBE in https://github.com/castiron/cac/commit/76735ae215c5a5f0aaf75436fb7b19a0d5b6786a should address this problem. I'm closing this ticket, to clear the milestone. If more problems arise, feel free to reopen with details.

Actions

Copy link

#11

Updated by Raymond Hoh over 10 years ago

Status changed from Resolved to Assigned
Target version changed from 1.6.7 to 1.6.8

Reopening.

It looks like the duplicate connection issue is still happening (and actually is a little worse during my previous refactor).

To summarize the problem, if a site has a ton of concurrent users per second during the time when a new IMAP inbox connection is needed, duplicate IMAP connections will occur, which could cause duplicate posts.

The problems didn't arise during my testing on NYCDH because the web traffic on NYCDH is lower compared to CAC. Now, that the latest RBE code is running on CAC, the duplicate connection issue is presenting itself.

My fault for not doing better stress testing locally. I spent some time late last night stress testing RBE using OpenWebLoad to simulate concurrent users and have coded a workaround. The results are better now, but not 100% foolproof.

See commit 7c1f72cc02.

The IMAP locking system now uses the filesystem instead of the database. In my testing, RBE can now handle up to 12 concurrent users on the same second without launching a duplicate connection. By that I mean, the site has to have more than 12 users (or page loads) on the same second when RBE needs to launch a new connection to the inbox. These two conditions should happen rarely.

WP-cron also runs into these race conditions and is not perfect, but I've tried to mitigate this as best as I can.

Boone: What do you think about a 1.6.7.1 release?

Actions

Copy link

#12

Updated by Boone Gorges over 10 years ago

Status changed from Assigned to Resolved
Target version changed from 1.6.8 to 1.6.7.1

Thanks for the diligence, Ray. I'm going to do a 1.6.7.1 release right now. Let's reopen this ticket as necessary.

Actions

Copy link

#13

Updated by Raymond Hoh over 10 years ago

Thanks for doing a 1.6.7.1 release, Boone!

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

CUNY Academic Commons

Custom queries

Bug #3164

Duplicate RBE messages

Updated by Boone Gorges over 10 years ago

Updated by Raymond Hoh over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Raymond Hoh over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Raymond Hoh over 10 years ago

Updated by Boone Gorges over 10 years ago

Updated by Raymond Hoh over 10 years ago