CUNY Academic Commons

It's now resolved... was happening for about 5-6 minutes.

Thanks, Luke. Scott is also reporting slowness in #24002 -- I will loop in Reclaim.

Thanks Colin. I am currently afk but I will be back in a few hours to chime in if needed.

I'm still experiencing issues logging in - first got a server error. then worked but was very slow

Adding Scott as a watcher here so we can use this as the main ticket for updates on this round of slowness/issues. Chris at Reclaim wrote me back about an hour ago with this:

"We're seeing the same on our end and are investigating. I've restarted a few services since then, but it may be related to the previous Cloudflare issues; since they've made some changes we've seen some increased Bot traffic."

Update earlier today from Chris at Reclaim: "We're still monitoring, but it looks like Cloudflare's made their bot/crawler rules a bit more lax as we're seeing a particular user-agent that's given us problems elsewhere show up in the logs: python-httpx. We've gone ahead and blocked this in Cloudflare and are looking at any other option to reduce illegitimate traffic."

Anyone still seeing performance issues?

Recapping a couple of recent exchanges with Chris at Reclaim here. Quoting at length so I don't butcher anything trying to summarize. We can touch on this in the dev call if there's anything else I should ask them about, or what we think about getting their performance notifications.

- After the Friday community call when me, Scott, and Matt were all experiencing site lag, I asked: "Can you give us an update on what you're seeing and doing to mitigate this? Also hoping you could recap for us what you have in place to monitor for slowness from your end and respond to it quickly. By the time our team is experiencing issues and coming to you, it's almost certain that a large number of our users have been negatively impacted for some time."

Reply: "We have a monitoring system in place that checks dynamic pages. It's only GET requests, so it seems to only ping when slowness becomes very bad. We're still looking into this and making tweaks on Cloudflare's end, and I'll speak with Taylor about increasing the sensitivity of our monitoring system."

- Then I asked: "How do you get notified when the monitoring system flags something? Is there any way to automate a message to a few emails on our side as well when that happens, so we know something is up (and so we already know you are getting the same notifications)? That might help keep us on the same page without more back and forth."

Reply: "I've replicated some ModSecurity rules we use across our other servers in Cloudflare's WAF which seems to have helped somewhat. We are looking at some cache optimization but given that may be more impactful for users we're holding off on that for a bit. Likewise I'm currently testing some other speed optimization settings Cloudflare has on DEV, and assuming there are no issues will apply them to PROD.

We currently have the monitors set up at the following URLs but are in the process to migrate them elsewhere. These send Slack notification to us, but you can subscribe and get more updates. I'll also check to see if Cloudflare can also send you all origin notifications if you can let me know what email you'd like them to go to."

https://status.reclaimhosting.com/798372919
https://status.reclaimhosting.com/798372917

As I mentioned on the dev call today, Reclaim has set up on the dev site for testing a challenge system (not outright blocking) for China, Russia, Iran, and Vietnam due to significant bad traffic we see from these places.

The challenge set up is a managed challenge which may be interactive or silent. An interactive challenge will often only show up if the traffic looks to be a bot or is from a source Cloudflare knows is malicious, but if it's a known good VPN it may be silent. Reclaim has temporarily stipulated ONLY interactive challenges so we can see how it'd look to bad traffic if we use a VPN to spoof visiting from one of the above countries.

I haven't been able to replicate the challenge yet, but that may be user error or an issue with my VPN program. Ray, thanks for offering to give it a try. I think if we can confirm that this works/looks ok for any possible visitors from these places that are legitimate (even though many may only get a silent challenge), we can roll this out on production.

FYI, I am also getting Cloudflare email alerts now for downtime (and when uptime resumes). So far it seems to be pretty routine, and maybe not that useful, but I'll keep monitoring and we can see if anyone else should receive these.

Thanks Ray, I think this looks good, thank you for testing. I have told Reclaim to roll the managed challenge system for the countries in question out on production.

Closing this ticket, as Reclaim has rolled this out on production. Just an FYI from them also: "I will mention that we've seen an increase in bad traffic from Indonesia across our infrastructure, so I've added that to the challenge list as well; let me know if this causes any problems."

That makes the full country list for this China, Russia, Iran, Vietnam, and Indonesia.