CUNY Academic Commons

It's now resolved... was happening for about 5-6 minutes.

Thanks, Luke. Scott is also reporting slowness in #24002 -- I will loop in Reclaim.

Thanks Colin. I am currently afk but I will be back in a few hours to chime in if needed.

I'm still experiencing issues logging in - first got a server error. then worked but was very slow

Adding Scott as a watcher here so we can use this as the main ticket for updates on this round of slowness/issues. Chris at Reclaim wrote me back about an hour ago with this:

"We're seeing the same on our end and are investigating. I've restarted a few services since then, but it may be related to the previous Cloudflare issues; since they've made some changes we've seen some increased Bot traffic."

Update earlier today from Chris at Reclaim: "We're still monitoring, but it looks like Cloudflare's made their bot/crawler rules a bit more lax as we're seeing a particular user-agent that's given us problems elsewhere show up in the logs: python-httpx. We've gone ahead and blocked this in Cloudflare and are looking at any other option to reduce illegitimate traffic."

Anyone still seeing performance issues?

Recapping a couple of recent exchanges with Chris at Reclaim here. Quoting at length so I don't butcher anything trying to summarize. We can touch on this in the dev call if there's anything else I should ask them about, or what we think about getting their performance notifications.

- After the Friday community call when me, Scott, and Matt were all experiencing site lag, I asked: "Can you give us an update on what you're seeing and doing to mitigate this? Also hoping you could recap for us what you have in place to monitor for slowness from your end and respond to it quickly. By the time our team is experiencing issues and coming to you, it's almost certain that a large number of our users have been negatively impacted for some time."

Reply: "We have a monitoring system in place that checks dynamic pages. It's only GET requests, so it seems to only ping when slowness becomes very bad. We're still looking into this and making tweaks on Cloudflare's end, and I'll speak with Taylor about increasing the sensitivity of our monitoring system."

- Then I asked: "How do you get notified when the monitoring system flags something? Is there any way to automate a message to a few emails on our side as well when that happens, so we know something is up (and so we already know you are getting the same notifications)? That might help keep us on the same page without more back and forth."

Reply: "I've replicated some ModSecurity rules we use across our other servers in Cloudflare's WAF which seems to have helped somewhat. We are looking at some cache optimization but given that may be more impactful for users we're holding off on that for a bit. Likewise I'm currently testing some other speed optimization settings Cloudflare has on DEV, and assuming there are no issues will apply them to PROD.

We currently have the monitors set up at the following URLs but are in the process to migrate them elsewhere. These send Slack notification to us, but you can subscribe and get more updates. I'll also check to see if Cloudflare can also send you all origin notifications if you can let me know what email you'd like them to go to."

https://status.reclaimhosting.com/798372919
https://status.reclaimhosting.com/798372917