Feature #21456
openSwitching to SMTP for outgoing email
Added by Boone Gorges 2 months ago. Updated 2 days ago.
0%
Description
Following up on our conversation last week, we need to make a decision about how outgoing email will be sent in the Reclaim environment. I see three options:
1. Continue to send email using PHP and the system mail program
2. Use a transactional provider
3. Use SMTP
Items 1 and 2 both require MX DNS changes. Option 1 will require special vigilance, since Reclaim IP addresses will be sending cuny.edu email (presumably transactional providers are more used to setting this sort of thing up). Option 2 also costs money.
So I think that option 3 is probably the best. Emails will continue to be sent through the Graduate Center, so there should be no DNS changes, and there should be no additional spam problems. The one hitch is that we'll need an email account that can be used for this purpose. We already have commons@gc.cuny.edu, but it's already in use for other purposes. We also have a gmail account we use for inbound RBE, but this is not a gc.cuny.edu account, so it will face spam issues. So I think we'll need to request a new account. Then we need to decide which WP plugin to use - there's a number of them that would probably be fine.
Reclaim's team has also recommended SMTP, for what it's worth.
Ray, can you think about the reasoning above and let me know whether it seems right? If so, what are our next steps?
Related issues
Updated by Raymond Hoh 2 months ago
Option 3, SMTP, seems to be the path of least resistance. I guess the next step is to see whether we can get another @gc.cuny.edu email address for use with Reclaim. Are @gc.cuny.edu email addresses provisioned the same way as CUNYFirst? If so, we might run into problems with having to change the password every 180 days according to the following document: https://www.gc.cuny.edu/sites/default/files/2022-02/Microsoft-Modern-Authentication_1.pdf (see point 8).
Updated by Boone Gorges 2 months ago
Colin and Matt, a reminder that we should probably decide in advance what specific inbox we'll be asking for. It could be something quasi-technical like commons-sender@gc.cuny.edu. Or something more detailed than the existing commons, like cunyacademiccommons@gc.cuny.edu.
A side note that I just had to have a conversation with the IT department at another organization about this very issue. They use Office 365 for email, and they use account delegation to grant access to organizational email addresses. As such, they don't hand out SMTP credentials for those accounts. I don't know if this is a technical issue or a policy one, but it's possible we'll get a similar answer from the Graduate Center.
Updated by Matt Gold 2 months ago
Hi Boone -- yes, you're right. As far as I know, GC IT discontinued SMTP access for email when it switched to Office365. I'm happy to ask to confirm, though, and I don't think it would be a big deal politically to ask the question. Let me know whether you want me to ask.
Updated by Boone Gorges 2 months ago
Hi Boone -- yes, you're right. As far as I know, GC IT discontinued SMTP access for email when it switched to Office365.
They discontinued SMTP access for delegated accounts, not for all accounts, right? The latter would be a much harder problem. In any case, yes, please go ahead and put in the request.
Updated by Boone Gorges about 2 months ago
IT has set up our SMTP account and shared the credentials with me. The account is cuny-academic-commons@gc.cuny.edu.
In https://github.com/cuny-academic-commons/cac/commit/a8024fa9f93eeff976e0ac45a29115dc8c78d7a7 I added the wp-mail-smtp plugin. To configure it, the following goes in cac-env-config.php:
define( 'WPMS_ON', true ); define( 'WPMS_MAIL_FROM', 'cuny-academic-commons@gc.cuny.edu' ); define( 'WPMS_SMTP_HOST', 'relayauth.gc.cuny.edu' ); define( 'WPMS_SMTP_PORT', 587 ); define( 'WPMS_SSL', '' ); define( 'WPMS_SMTP_AUTOTLS', true ); define( 'WPMS_SMTP_AUTH', true ); define( 'WPMS_SMTP_USER', 'commonsmail.svc' ); define( 'WPMS_SMTP_PASS', ENTER PASSWORD HERE ); define( 'WPMS_MAILER', 'smtp' ); define( 'WPMS_MAIL_FROM_FORCE', true ); // GC SMTP will refuse to send emails from other addresses
I've got the password privately saved and I'll set it up when I'm ready to deploy to the new production site.
With this change, our old commons-not-reply Sender address will go away. This is necessary, because as a security precaution, the SMTP account is set up to send emails only with the Sender address of cuny-academic-commons@gc.cuny.edu. I've removed this in https://github.com/cuny-academic-commons/cac/commit/93504fee415b26ee1c0166f8983937402422f175. I don't believe that these changes have any effect on Reply-To addresses, but this will need testing, especially in the context of RBE.
Ray, can you think about how RBE ties into this? I know you mentioned that now would be a good time to switch for incoming-mode. Perhaps you could start a new ticket, referencing this one so that we can coordinate, where you lay out the steps that need to be taken?
Updated by Raymond Hoh about 2 months ago
- Related to Bug #16177: Switch to Inbound mode for RBE added
Updated by Raymond Hoh about 2 months ago
Ray, can you think about how RBE ties into this?
I don't anticipate issues with WP Email SMTP. I've used RBE with basic PHPMailer SMTP settings and it worked, but I'll do some local testing with WP Email SMTP to be sure.
I know you mentioned that now would be a good time to switch for incoming-mode. Perhaps you could start a new ticket, referencing this one so that we can coordinate, where you lay out the steps that need to be taken?
I've already set up a ticket for this -- #16177. Basically, what needs to be done next is to set up an auto-replier in our existing GMail IMAP account and then in the RBE admin settings page in WordPress, switch from IMAP mode to Inbound mode. I'll ping both you and Colin to proofread the copy for the auto-replier I have so far and then we can move forward.
Updated by Boone Gorges 6 days ago
- Related to Bug #21754: Reclaim email notifications added
Updated by Boone Gorges 6 days ago
It was reported in #21754 that email notifications of member-to-member private messages were not being sent. This was because bp-reply-by-email was forcing the From header to be the email address of the sender. This won't work on the new setup - it must be the canonical Commons SMTP email address. In https://github.com/cuny-academic-commons/cac/commit/2ac91bba88487875865e92dfd2f6a4a3bbe82a8d I added a filter callback that prevents RBE from making this change. I also removed a bit of customization, added in #6677, that caused the From address on BPGES email notifications to match the activity-creating user's email address in some cases. This won't work, for related reasons, so I lumped the changes together.
Ray, could you review these to make sure the logic makes sense? I feel like we may need a further very-late filter to enforce the sender email, in case there are other plugins doing odd things, but this seems like a good start in any case.
Updated by Raymond Hoh 6 days ago
Ray, could you review these to make sure the logic makes sense? I feel like we may need a further very-late filter to enforce the sender email, in case there are other plugins doing odd things, but this seems like a good start in any case.
Does the GC SMTP 'From' header allow for the Name to change, but keeping the email address to the new cuny-academic-commons@gc.cuny.edu email address? For example, instead of CUNY Academic Commons <cuny-academic-commons@gc.cuny.edu>, can we change the "CUNY Academic Commons" portion to anything we want?
RBE did also change the email address from cuny-academic-commons@gc.cuny.edu to noreply@gc.cuny.edu: https://github.com/cuny-academic-commons/cac/blob/master/wp-content/plugins/bp-reply-by-email/bp-rbe-core.php#L299-L311 . We can fix that by adding another filter in RBE to just change the name, but without making changes to the email address. Are you able to test my theory to see if the email is able to be sent with only the name change? I should have asked for this sooner, but after the migration is over, can we set myself and Jeremy up with access to the Reclaim server? There is an older email that has some information about this, but I can't seem to find it.
Updated by Boone Gorges 6 days ago
Thanks so much, Ray. Yes, it appears that we can override the name. So:
$email->set_from( "noreply@{$host}", bp_core_get_user_displayname( $this->listener->user_id ) );
does not work, but
$email->set_from( "cuny-academic-commons@gc.cuny.edu", bp_core_get_user_displayname( $this->listener->user_id ) );
does work
Can I ask you to set this up in the way that makes the most sense? I suppose we may want to restore the GES callback I removed in https://github.com/cuny-academic-commons/cac/commit/2ac91bba88487875865e92dfd2f6a4a3bbe82a8d, but only change the FromName? We could probably do something similar with a late callback on RBE emails, but it may be cleaner, as you note, to have a filter for this purpose right in bp-reply-by-email.
Updated by Raymond Hoh 6 days ago
I've restored the GES email changes and have added a new commit for RBE, which removes the previous, custom 'From' email address: https://github.com/cuny-academic-commons/cac/commit/8eecb23dff245b4a066ae8dadfb8e86299f156b7. Boone, can you pull the recent commits and see if this will allow email to work on the Reclaim testing server?
Updated by Boone Gorges 6 days ago
Thanks, Ray. This looks like an improvement for RBE. Regarding the GES changes, I think this line will need to be changed, right? https://github.com/cuny-academic-commons/cac/commit/eb6e354a91570cc0d6e3cace37ecead045aec045#diff-ebb3ca9d8538bee7d186fd882e3e80311f551f06a91bf517a134bb14909c1714R218
Updated by Raymond Hoh 6 days ago
Regarding the GES changes, I think this line will need to be changed, right?
I think that should be okay; it's just re-using whatever the 'From' email address is, which hopefully isn't manipulated at that juncture.
Updated by Boone Gorges 6 days ago
Sounds good, Ray. I have pulled your changes to the reclaim server. If there are issues with GES sending, we'll figure that out after the migration + DNS switchover.
Updated by Boone Gorges 3 days ago
- Status changed from New to Resolved
SMTP seems to be working without a problem. There may still be mail issues trickling in, but they're more likely to be due to specifics of BPGES offloading, or scheduled tasks as those used by cac-onboarding.
Updated by Colin McDonald 3 days ago
The only email I haven't been able to generate in testing today is a group membership invite. I think I've gotten all others from my past lists to come through. I do get invites to the Group for Group admins when I create a new group, but I think that's because it's a separate trigger.
Updated by Colin McDonald 3 days ago
One more email I haven't been able to get is the authentication codes it says will be delivered when switching to Email 2FA in settings.
Updated by Boone Gorges 2 days ago
- Status changed from Resolved to Assigned
Thanks, Colin. These missing emails don't have anything to do with SMTP. If you're still experiencing them, please post to the more general email thread at https://redmine.gc.cuny.edu/issues/21754
Reclaim has told me that the site was going down in part because of problematic queries in wp-mail-smtp. I believe this - I saw some bad queries myself in my testing (SHOW TABLES) - but Reclaim didn't provide enough information to do a full debug at this time. What they did provide was a barebones mu-plugin that forces email through SMTP:
<?php
add_action( 'phpmailer_init', 'send_smtp_email' );
function send_smtp_email( $phpmailer ) {
$phpmailer->isSMTP();
$phpmailer->Host = SMTP_HOST;
$phpmailer->SMTPAuth = SMTP_AUTH;
$phpmailer->Port = SMTP_PORT;
$phpmailer->Username = SMTP_USER;
$phpmailer->Password = SMTP_PASS;
$phpmailer->SMTPSecure = SMTP_SECURE;
$phpmailer->From = SMTP_FROM;
$phpmailer->FromName = SMTP_NAME;
}
I've modified this for use on the Commons. It's sitting in production at wp-content/mu-plugins/smtp.php. It seems to be working for now. I think it needs more thought and research to see whether there are specific sending use cases that might not be covered by this technique, to see if we need to do any research on other potential plugins. For the time being, I'm going to reopen this ticket so that I can look more into it in a few days.