CUNY Academic Commons

Boone, I believe this is due to the SSL cert for mapped domains not working properly on certain URLs.

For example, if you try to go to https://acert.hunter.cuny.edu/wp-admin or even https://acert.hunter.cuny.edu/wp-login.php, you get the following error: ERR_SSL_PROTOCOL_ERROR. This used to work on the GC servers, which ran on Apache. Perhaps some other LiteSpeed configuration needs to be done to ensure these URLs can resolve properly, and thus, the SSO should work once we solve this.

Laurie, I believe your issue is somewhat related, but not the same. Since the Reclaim team installed the LiteSpeed Cache plugin with page caching turned on, it interferes with our Cookies for Comments (and Login) integration (see https://redmine.gc.cuny.edu/issues/2796#note-11). As a workaround, can you go to https://commons.gc.cuny.edu/wp-admin/ and attempt to login and see if you are able to authenticate?

Boone, do we want to disable our Cookies for Comments integration for now? Since we're now using Cloudflare on Reclaim, we can probably configure the Cloudflare firewall to limit wp-login bot attempts to a configurable value. See https://www.homannsoftware.com/2024/03/11/wordpress-cloudflare-configuration-guide/.

Boone, I believe this is due to the SSL cert for mapped domains not working properly on certain URLs.

Wrong theory! The issue is caused by the Lightspeed Cache plugin. It is caching pages for logged-in users by default. Mercator injects some inline JS depending if the user is logged-in or not. Since the Lightspeed Cache plugin was using the cached logged-out homepage even when logged-in, the redirect loop occurs because the Mercator JS is still using the logged-out JS. Once I disabled caching for logged-in users on the ACERT site, the redirect loop stopped.

The default settings for the Lightspeed Cache plugin need to be changed. By default, each site has different Lightspeed settings. I have just set the Lightspeed Cache network settings so each site's Lightspeed settings will use the same settings as the primary site's. I have also configured Lightspeed to turn off caching for logged-in users. This might affect performance on the Reclaim server. If so, toggle the "Cache Logged-in Users" option back on this page: https://commons.gc.cuny.edu/wp-admin/admin.php?page=litespeed-cache

Ray, thanks a million for debugging this. Caching for logged-in users was one of the things I worried about when Reclaim installed litespeed-cache, and I'm unsurprised to hear that it caused problems. The server seems to be holding up okay for now, so I think it's good to leave that setting as-is.

Boone, do we want to disable our Cookies for Comments integration for now? Since we're now using Cloudflare on Reclaim, we can probably configure the Cloudflare firewall to limit wp-login bot attempts to a configurable value. See https://www.homannsoftware.com/2024/03/11/wordpress-cloudflare-configuration-guide/.

Yeah, let's turn that off for now. Setting cookies is going to mess with any kind of static page caching (another problem with just switching on CDN integration). Cloudflare can worry about blocking brute-force attempts.

For example, if you try to go to https://acert.hunter.cuny.edu/wp-admin or even https://acert.hunter.cuny.edu/wp-login.php, you get the following error: ERR_SSL_PROTOCOL_ERROR. This used to work on the GC servers, which ran on Apache. Perhaps some other LiteSpeed configuration needs to be done to ensure these URLs can resolve properly, and thus, the SSO should work once we solve this.

I don't have the bandwidth to report this to Reclaim right now. If you would like to respond to one of our ongoing threads with them, that'd be OK, otherwise it's probably OK to wait on this one until Monday.

For example, if you try to go to https://acert.hunter.cuny.edu/wp-admin or even https://acert.hunter.cuny.edu/wp-login.php, you get the following error: ERR_SSL_PROTOCOL_ERROR. This used to work on the GC servers, which ran on Apache. Perhaps some other LiteSpeed configuration needs to be done to ensure these URLs can resolve properly, and thus, the SSO should work once we solve this.

On second thought, I believe the problem is due to the acert.hunter.cuny.edu site using both CNAME and A records for their DNS. In particular, they have 2 A records: https://www.whatsmydns.net/#A/acert.hunter.cuny.edu . Is it correct that both A records can be removed now that the CNAME record has been changed to mapped-domains.commons.gc.cuny.edu?

Yeah, let's turn that off for now. Setting cookies is going to mess with any kind of static page caching (another problem with just switching on CDN integration). Cloudflare can worry about blocking brute-force attempts.

I've network-deactivated both Cookies for Comments and Cookies for Logins and also commented out the related .htaccess rules to block access to the login and register pages when the CFC cookie does not exist; logins should now be working again.

Since these plugins are now deactivated, we should look at configuring the Cloudflare firewall to limit access to https://commons.gc.cuny.edu/register/ and wp-comments-post.php. We can use the guide I linked to above to rate limit bots or we can look to issue a browser challenge: https://jackwhitworth.com/blog/block-wordpress-comment-spam-with-cloudflare/. This might add a slight delay when doing the form submission though.

I don't think that A vs CNAME could make a difference. The redirect clearly happened after the page started to render. If it were a DNS problem, the request would never reach the server.

After your recent config changes, I'm not seeing the redirect loop on acert.hunter.cuny.edu. Are you?

After your recent config changes, I'm not seeing the redirect loop on acert.hunter.cuny.edu. Are you?

Sorry for not being clear. I was referring to the SSL errors I was seeing on acert.hunter.cuny.edu. I just checked and I am not getting the ERR_SSL_PROTOCOL_ERROR error anymore. It looks like the second A record on acert.hunter.cuny.edu has been removed as well.

Thanks for the details, Colin. I don't believe that any of the notices are related to the redirect behavior (they are separate problems that could be addressed, I suppose), but it's helpful to see them.

From your video it appears that you were redirected back to the page just once, and it happened even though you were not logged in. Is this correct?

I did some more research and I might have been on the wrong track with the cookie stuff. The cookie configuration was the same on the GC site.

I think the root problem here is the same as #21801, namely that users are served cached pages immediately after login. In the case of mapped domains, this means that the cached page includes the `head_js()` JavaScript at https://github.com/cuny-academic-commons/cac/blob/d39ef90c1f2ff4bddd73f911f50dd4b350556971/wp-content/mu-plugins/mercator/sso.php#L345. The intended flow for Mercator is too complicated to lay out here, but it is supposed to end by returning the user to the originally requested URL with the auth cookies in place. That part is working, but the document body still has that JS because it's being served from the cache. So the redirect loop goes on and on.

I was able to put a temporary fix in place by adding a unique cache-busting param to the 'back' URL in the authenticating AJAX request: https://github.com/cuny-academic-commons/cac/blob/d39ef90c1f2ff4bddd73f911f50dd4b350556971/wp-content/mu-plugins/mercator/sso.php#L305 (it's not committed, this is just the line that's affected). This has the effect of sending the user back to a URL that is not cached by Litespeed, which therefore doesn't have the head_js script in it. The redirect loop is no longer happening for me.

I think this problem will go away, and we can roll back my workaround, once #21801 is resolved.

I think we probably need to look carefully at the way that the Litespeed-cached page contains the head_js. I think it means that anytime a non-logged-in user is served a cached version of a page on a mapped domain, an async Mercator login request is sent to the (non-cached) production site. This pretty much cancels out the benefit of caching these mapped domains. Ray, maybe you have some smart idea about how we can prevent this from happening. Here's an idea (not sure if it's smart, it's been a long day): Litespeed puts a tag like this in their cached documents:

<!-- Page cached by LiteSpeed Cache 6.5.4 on 2025-01-20 13:01:14 -->

We could do some client-side magic that causes the async request only to take place if a tag like this is not found. Of course, now that i think of it, this would probably break the current frictionless SSO for legitimate users.

Or maybe we leave this optimization for another day.

I think we probably need to look carefully at the way that the Litespeed-cached page contains the head_js.

Boone, before we even get to the 'back' mod, I found another problem with the head_js; Mercator is using a nonce.

For authenticated Commons users attempting to go to a mapped domain for SSO, in the <head>, this is outputted:

<script async src="https://commons.gc.cuny.edu/wp-admin/admin-ajax.php?action=mercator-sso-js&amp;host=newsliteracymatters.com&amp;back=https%3A%2F%2Fnewsliteracymatters.com%2F&amp;site=8164&amp;nonce=THIS_NONCE"></script>

There is a nonce and the nonce only validates if it is less than 24 hours: https://github.com/cuny-academic-commons/cac/blob/9efc198d925da00cab3fd39efbd0206ac7a71dff/wp-content/mu-plugins/mercator/sso.php#L308-L312

If the nonce is older (meaning if the cached page is older than 24 hours), Mercator's JS will exit and not attempt the SSO. This means the redirect loop will not even take place because SSO hasn't completed and the user isn't logged in to the mapped domain.

So there are two problems with using LSCache and SSO on mapped domains:
1. If the cached page is older than 24 hours, SSO will not complete. There are a few approaches I'm thinking about here:
a) We generate a fresh nonce for Mercator with AJAX if the cached page is older than 24 hours (we would check the <!-- Page cached by LiteSpeed Cache 6.5.4 on TIMESTAMP --> comment for the timestamp) and then use the nonce to load Mercator's async JS script again. This will take longer for SSO to complete because of the round-trip query for another nonce.
b) We bypass Mercator's nonce checking. Would require a core hack to Mercator. We've already made adjustments to Mercator so another hack here wouldn't be out of the question.
c) We tell LSCache not to cache mapped domain pages, or:
d) We tell mapped domain users to just navigate to the mapped domain's dashboard (/wp-admin) as that should trigger the SSO login process.

I don't like all of these approaches, but c) would be similar to when we were hosted on the GC servers and d) wouldn't require us to do anything other than address the next point.

2. If SSO works, it is possible for LSCache to serve a cached page from the SSO AJAX redirect. A variation of Boone's 'back' URL workaround will work for this. (This wouldn't be necessary if we decide to go with 1)c.

Thanks for thinking about this, Ray.

In testing this morning, I ran into the issue you noted about expired nonces. I was testing SSO login with a domain I didn't test yesterday. I was getting a cached version of the page, which was triggering the AJAX SSO request. That request was failing because of an expired nonce. When I commented out the nonce check in the Mercator SSO callback, the redirect worked and I was logged in.

I don't fully understand the security implications of the nonce check here. The nonce is not user-specific, so it's not intended to verify specific user intent in the same way as typical CSRF protection. Nonce verification happens at two points:
a. When the async request is sent from the mapped domain to the main domain. output_javascript_priv() verifies this nonce.
b. On the mapped domain, the login request from the main domain is handled. See handle_login_request().

The nonce we are concerned with, the one that can be cached and is causing us problems here, is (a). But this is the one that I don't really understand. That initial request doesn't initiate any changes ("database-level changes" is the threshold I often use when thinking about when to introduced CSRF tokens). Its only purpose is to verify authentication cookies on the main domain. But this happens with is_user_logged_in() and I don't understand why it's time-sensitive. But I would be glad for someone else to explain where I'm wrong here :-D

Regarding the more general issue, here are some tickets from Mercator's GitHub tracker that are adjacent to our issue:
https://github.com/humanmade/Mercator/issues/41 <-- this seems to circle around a similar problem to ours, though I don't fully grasp it
https://github.com/humanmade/Mercator/issues/110
https://github.com/humanmade/Mercator/issues/67
https://github.com/humanmade/Mercator/issues/63

I kinda like the idea of simply disabling this "magic" SSO for mapped domains, and requiring users to manually visit wp-admin. It solves a couple problems: the weird async; the cached nonces; the traffic generated by the AJAX requests on every front-end pageload. Let's talk this through on today's call.

I think we turn off Mercator's SSO module as you've mentioned, but I'd like to try an alternative SSO option: https://wordpress.org/plugins/multisite-multidomain-single-sign-on/. This one works with the user navigating to their WP admin bar's "My Sites" menu to initiate the SSO, which I think is a better approach than Mercator's "magic" version and solves all the edge cases that have occurred since we've enabled LSCache's page caching. We'll need to do some testing to see if this plugin works as advertised and how it interacts with LSCache.

Definitely worth a try, though it only improves the SSO user experience in one direction (you're looking at the main Commons site and see the My Sites dropdown, and are navigating to the mapped domain). Users looking at the mapped domain will have to do something manual to log in (such as visiting the main site and going through the My Sites menu).

Boone, to disable Mercator's SSO, add the following to wp-content/sunrise.php before Mercator is required (before line 5):

// Disable Mercator's SSO module.
add_filter( 'mercator.sso.enabled', '__return_false' );

Feel free to commit when ready.

Committed in https://github.com/cuny-academic-commons/cac/commit/82a5b7691eb2bd9799f03eba3d6575d9d4a9ee93 and deployed.

Colin, this may be related to caching, but I'm not sure. I've opened a separate ticket for it #21823 because, in any case, I don't think it's causing the current behavior.

I meant to post this here before but posted it to #21822, where it's now deleted. I thought it might have something to do with the discussion/work going on regarding the header and strange things appearing there:

Maybe this means nothing, but was anyone else aware that we have a couple of 11-year-old tickets here dealing with "chineseindian(s).org" issues? #3108 and #3055. I googled the "chineseindian.org" and it's all that came up.

Boone, I had to make a few other changes to fix some issues after disabling Mercator's SSO. See https://github.com/cuny-academic-commons/cac/commit/55c0b762e032da1f38fb35f50487709dd754eb97.

- Attempting to navigate to a mapped domain's dashboard (such as acert.hunter.cuny.edu/wp-admin), would redirect back to commons.gc.cuny.edu/wp-login.php?redirect_to=XXX. This redirection is now disabled because logging in from the main site will no longer authenticate the user on the mapped domain.
- After this change, navigating to a mapped domain's dashboard and attempting to login would fail due to invalid cookies. Since we disabled Mercator's SSO module, that module was responsible for setting the appropriate cookie domain when logging in from a mapped domain. I've added this back in https://github.com/cuny-academic-commons/cac/commit/55c0b762e032da1f38fb35f50487709dd754eb97#diff-164851a45c87512f7bff8d957892d922791f46c116f3a9182dffc50fdf38aa80R10

One more thing to note is I had to add the cac-release-manager email address on the Reclaim server's git config in order for me to push the change to production. Hope that's okay!

Thanks, Ray, this seems good to me. Are you going to move forward with https://wordpress.org/plugins/multisite-multidomain-single-sign-on/? Seems like it might ease the login process for users of mapped domains.

Are you going to move forward with https://wordpress.org/plugins/multisite-multidomain-single-sign-on/?

I've made some headway. I'm using a fork of the plugin I mentioned: https://github.com/mikelittle/multisite-multidomain-single-sign-on/. And I forked that fork to add support for Mercator's aliases: https://github.com/mikelittle/multisite-multidomain-single-sign-on/compare/master...r-a-y:multisite-multidomain-single-sign-on:mercator

I'm using a fork because it has more recent updates and uses WordPress's code standards. The problem with the original and the fork is it doesn't support Mercator's aliases in the WP Admin Bar's "My Sites" menu. My changes check to see if the site has any Mercator aliases. If it does, we use the first mapped alias and rewrite the site URL in the "My Sites" menu to use the mapped domain. I've tested this locally on my non-HTTPS development box and it works. The next step is to put this up on production and see if it works on the Reclaim servers. I'll do this at a time when the site is less busy.

Approach seems good. Thanks, Ray!

I've added the multisite SSO plugin to /wp-content/multisite-multidomain-single-sign-on.php and everything appears to be working.

For those of you who are super admins, to test:
1. Add your Commons account to a mapped domain site. (I tested by adding myself to literarybronx.com). To add yourself, go to /wp-admin/users.php and add your user account to the site.
2. Then, open a new, private browser window and login to the Commons as usual.
3. Next, navigate to the top-right hand corner, hover over your profile photo and hover over the "My Sites" menu. Find your mapped domain in the list. If you hover over that URL, in the browser's status bar, you should see the mapped domain URL followed by some URL parameters, which helps to log you in. Click on the link.
5. You should now be logged in to the mapped domain.

The process also works in the other direction:
1. Close your private browser window and open a new private browser window.
2. Try logging into a mapped domain. (If you are viewing an older cached page, you may need to go directly to either /wp-admin/ or /wp-login.php due to the admin bar's login form not using the mapped domain's URL. This is fixed in https://github.com/cuny-academic-commons/cac/commit/e971a2a1b30dd5e4b813de52ae72a88bdf40651a and pushed to production.)
3. Go back to the top-right hand corner, hover over your profile photo and hover over the "My Sites" menu. Now select another site under the "My Sites" menu such as the main Commons site.
4. You should now be logged in there as well.

Boone, I know we're disabled page caching for URL querystrings in the .htaccess file, but just to be safe, I've also added the multisite SSO's URL parameters to the excludes list in the LSCache admin area here: https://commons.gc.cuny.edu/wp-admin/admin.php?page=litespeed-cache#excludes. I haven't committed the plugin yet so do give it a test and let me know if things are working for you.

Adding Marilyn as a watcher here, and maybe she can test the above when she gets a second. I am not a super admin and am actually not sure who else is. Probably not worth it to make me one just for this, though I'm open to it.

Colin, I've just added your Commons account to literarybronx.com so you can test the multisite SSO login. If anyone else wants to test the multisite SSO, let me know.

I've run some tests and this is working great for me. Thanks, Ray!

Thanks for the vid, Colin.

The step where I wasn't very clear is after you logged in to your mapped domain, you clicked on the "My Sites" menu item. You need to select one of your sites under the "My Sites" menu for the SSO to work. Can you try again? But this time, click on a site under the "My Sites" menu after logging into your mapped domain to see if the SSO is working?

No problem, and yes that sequence worked just fine. Thanks!

I've committed the multisite-multidomain-single-sign-on plugin as a mu-plugin in https://github.com/cuny-academic-commons/cac/commit/e80a28c460ec34428ed2d111fb7695247ab88901 and is pushed to production.

I'm going to leave this ticket open for the documentation team so they can notify mapped domain users about the SSO change. However, feel free to close.