CUNY Academic Commons

Hi Ray, thanks for noticing and posting. Do you think we should just wait an hour and see if this is resolved on its own, rather than contacting CUNY IT right now?

The database server usually takes between 1.5 to 2hrs to reboot judging from some of the recent downtimes from July: https://hetrixtools.com/report/uptime/2024-07/65b1dc51c2df4bf4b5cf78ba6e7bb4b6/#downtime_when. Will check in in about half an hour to see if the Commons is back up.

Boone, I've added a few robots.txt changes to wp-content/mu-plugins/cac-functions.php on production but haven't committed them yet:

User-agent: AhrefsBot
User-agent: ClaudeBot
User-agent: DataForSeoBot
User-agent: DotBot
User-agent: GPTBot
User-agent: SeekportBot
User-agent: SemrushBot
User-agent: MJ12bot
Crawl-delay: 10

This restricts these bots from crawling a domain to every 10 seconds per page. If we want to be more drastic, we can block them entirely.

The list is based on looking at some of the recent access logs with 'Bot' in the user agent.

Thanks, Ray. I think it's OK to make these robots.txt changes permanent. I would be more wary of total blocks on any user agent, unless we had a pretty clear indication that those specific agents were causing specific performance problems.

I've committed the robots change in https://github.com/cuny-academic-commons/cac/commit/acdb50bc1c2087f57ef5d0b81d4ec5b764429e38.

I would be more wary of total blocks on any user agent, unless we had a pretty clear indication that those specific agents were causing specific performance problems.

That's sensible, though I'd personally like to block all SEO and marketing-related bots :)

Unreal. I've got to run to a meeting, but can you please short-circuit this in production while we figure out what to do longer term?

I've limited Editoria11y's uninstall routine to just the current site in https://github.com/cuny-academic-commons/cac/commit/fc928dcff406cfb135cf882d5f070527ab2b2552. I've also pushed the fix to production.

FYI, the Commons status blog says that site is back up, but I still see an under maintenance message

Still down, alas.

When I updated the status blog, the site was back up online, but the site went down again shortly afterwards and is back up again. So it appears that something else is the cause of the downtime.

I'm currently away from my PC and won't be able to look at this for at least another hour or so. If Boone is around, hopefully he can jump in.

I've opened a PR for the Editoria11y repo: https://github.com/itmaybejj/editoria11y-wp/pull/35

Thanks for opening the PR, Boone!

The maintainer accepted my PR: https://github.com/itmaybejj/editoria11y-wp/pull/35#issuecomment-2374014411