CUNY Academic Commons

I just published an "upcoming downtime" post on the status blog:

https://commonsstatus.wordpress.com/

I slightly modified the language I just drafted in the forum for wider communication about the migration. I'll edit it as needed, but it seemed good to get something up for now.

I'm inclined to just keep linking to the status blog homepage, as opposed to a specific post, as we redesign this maintenance splash page. That way everyone will hopefully see the latest updates pretty easily.

For the splash page text itself, is it doable enough to change the text to something migration-specific for Jan 15, since the downtime is so notable? Then we could get the key info right on that page, something like:

"The CUNY Academic Commons is offline while we migrate to a new hosting provider. The process began at 10:30am Eastern on Wednesday, January 15th and will take 48-72 hours to complete because of the size and complexity of the data involved.

Please check our status blog (link:https://commonsstatus.wordpress.com) for the latest updates. Thank you for your patience as we take this important step for the stability and improvement of the Commons."

Let me know if you think we should just go with an updated but generic offline message instead. Either way, we could plan to use something like this post-migration:

"The CUNY Academic Commons is offline for maintenance. Our team is working to restore the site as soon as possible.

Please check our status blog (link:https://commonsstatus.wordpress.com) for the latest updates, and thank you for your patience."

Yes, we can have something specific to the migration on the maintenance page, and IMHO it's a good idea to do so.

Thanks so much, Colin. The post looks great. I think we should maybe add the help address so that people can email us if they have concerns/questions.

Perhaps one other thing we might want to mention is all content added to the Commons ahead of the downtime will be migrated to the new host. People may have questions/concerns about that, and/or they might not know/understand what a hosting migration means

Thanks, Matt. I'm going to start finalizing and posting outreach, also as per my forum post here:

https://commons.gc.cuny.edu/groups/cac-community-team-project-planning/forum/topic/host-migration-communications/

I will add the note about preserving and migrating all content. Part of me didn't want to even bring that up, at the risk of worrying users who hadn't considered it, but on second thought it seems better to say too much on that front and cut any concern off at the pass.

Thank you, Ray and Sara! Looks great!! So much better. One small note -- can you please edit

we take this important step for the stability and improvement of the Commons.

to

we take this important step to improve the stability of the Commons.

Looks great to me, too! And thanks for the copy edit, Matt.

Boone, for the reclaim-migration branch, did you want me to overwrite the existing maintenance.html file or did you want to use a different filename?

Let's use the same filename - there's no reason to keep the old one around. After the migration, we can change the language in that file to be more general.

It looks like it might be possible to customize the Cloudflare error page: https://developers.cloudflare.com/support/more-dashboard-apps/cloudflare-custom-pages/configuring-custom-pages-error-and-challenge/, but this feature is only available to Cloudflare Pro, Business, or Enterprise plans. We'd have to reach out to Reclaim to see if this is possible.

Thanks, Ray. As Colin says, this is not urgent, but we probably should reach out to Reclaim about it when things calm down a bit

As mentioned on the dev call yesterday, I'll copy over the latest from the support thread we have going with Reclaim about this soon. I think we might be getting closer to a resolution with them there, though, so I might wait on that a little longer.

In the meantime, I saw that one challenge is finding a way to test whether a new maintenance page is working as it should, since it's hard to replicate site downtime. However, as part of #22053 I just tried twice to clone the Help & Support site, and each time I ended up at the default Cloudflare timeout screen after a couple of minutes of the site presumably trying and failing to clone. Thought that might be helpful.

Latest from Reclaim on enabling the custom maintenance/downtime page:

The feature we have enabled on your prod is "Enable Origin Error pages" in Cloudflare. This simply allows the error pages that are served by the web server to be "passed through" and work as if they weren't served by Cloudflare. This is a feature exclusive to Cloudflare's Enterprise tier, which is what you all have on commons.gc.cuny.edu. Unfortunately, we do not have that plan or this feature available on reclaimhosting.dev, so we can't enable this feature on your dev site.

https://support.reclaimhosting.com/hc/en-us/articles/1500011279141-Pointing-localhost-File-to-Preview-Site

As a workaround, we could have you all point your hosts file for the dev site, effectively bypassing cloudflare, for the purpose of testing your error pages instead. If that works for you all we'll just need to know your IP address, so we can whitelist it in the firewall for direct web traffic to the dev server. Let us know if you want to pursue this option.

It seems like the "enable origin pages" feature Reclaim has enabled on Cloudflare would work for 5xx errors, but I wouldn't expect it to work for timeouts.

It also seems like a Cloudflare worker powered page would work for this. https://developers.cloudflare.com/rules/snippets/examples/maintenance/

I've done a very small amount with Cloudflare workers to route certain requests to a different server. They're kind of annoying to get going, especially if you don't stop to understand what you're doing. But, once it's up and running it doesn't really need any maintenance. I think this fits in the category of: "if we controlled Cloudflare, we could get this done fairly quickly", but I'm not sure we'd really want to take that step. :)

The feature we have enabled on your prod is "Enable Origin Error pages" in Cloudflare.

I think that since Reclaim are using Cloudflare's Enable Origin Error pages, we should try to utilize this feature first. I've implemented some custom Origin Error pages based off of the ones GitLab uses with Cloudflare.

• 5xx errors - https://commons.gc.cuny.edu/cf-500.html
• 1xxx errors - https://commons.gc.cuny.edu/cf-1000.html
• JS challenge - https://commons.gc.cuny.edu/cf-attack.html
• Captcha - https://commons.gc.cuny.edu/cf-captcha.html
• IP/Country Block - https://commons.gc.cuny.edu/cf-block.html

For the 5xx and 1xxx error pages, I've made a first draft at using more, friendlier copy, but Colin and others, you might want to rewrite this.

Once we're ready to, we can ask Reclaim to use these error pages as explained in step 4 of this page only on production, not cdev.

It seems like the "enable origin pages" feature Reclaim has enabled on Cloudflare would work for 5xx errors, but I wouldn't expect it to work for timeouts.

According to Cloudflare, 500, 501, 503, 505 and 520 to 527 errors are not covered by Origin Error Pages, but 502 (Bad Gateway) and 504 (Gateway Timeout) should be covered. In cases where we've experienced recent downtime, Cloudflare-specific errors (520 to 527) would not be covered. For an example, see #22540.

As Boone mentioned on a recent dev call, it might not be worth devoting much more time into this. Let's try for the custom Origin Error Pages and go from there.

Hi Ray, thanks for putting all of this together. The draft pages all read and look good to me. I have sent them along to Reclaim and asked them to move ahead with the next steps. Hopefully that all goes well, then I agree that we can put this to bed for now and see if other pressing needs arise on this front.