CUNY Academic Commons

Thanks, Boone, I let him know.

And he replies "Thanks for all your work on this, Boone. Btw, long time and I hope that you are doing well.

Thanks for adding the domains to the exclusion list of Raffi's site. Yes, dpi.org was a typo and no we have not heard from the DOI Foudation only ACM. I love the idea of a tool to enforce the restriction across the entire Commons, so let's see if this test works. I will report back if the publisher notifies us, which would mean a failed test."

Another reply- "So far, so good, so maybe the solution you put into place worked.
I just want to point out that the Computing Reviews URL is computingreviews.com, not computerreviews.org; I shared the incorrect URL."

I told him I'm closing the ticket but to be in touch if needed.

Stephen also asks "
Since your 'Exclusion list' approach appears to have resolved the issue of arbitrary rate limits of this publisher, I am wondering if you still intend to put a tool in place that will enforce restrictions across the entire Commons?"

Can we add Stephen to Redmine?

Here's his latest:
"I’ve got bad news:

Our e-resources librarian has reported that we received another "excessive session" notification from ACM on 6/21.

Best,
Stephen"

I've reactivated Stephen's dormant Redmine account and added him as a watcher.

Did ACM provide any more log files? Before taking next steps, I need to ascertain whether this is the very same set of URLs (which would indicate that this is a bug in Broken Link Checker's "exclusion_list" tool) or whether it's coming from somewhere else on the Commons.

A quick follow-up, having added Matt and Colin as watchers:

As obnoxious as it may be, we should consider the possibility that we'll simply need to disable Broken Link Checker across the entire network. It's not acceptable that this nice-to-have but not critical functionality on the Commons would cause the library to be banned from important services.

FWIW, I think the blacklist functionality is working. Last Friday, I asked the link checker to recheck individual ACM links and it seemed to refuse to do so.

I wonder if they are somehow counting redirects. I've appended "doi.org" to the blacklist.

Thanks, Raffi.

Stephen, thanks for passing along the log. You can see by the timestamps that the plugin made 35 requests in the course of 4 minutes. As I noted in my initial analysis, this level of resource usage is extremely minor, and underscores the unreasonably high sensitivity of their filters.

Raffi, you're almost certainly correct that ACM is counting redirects from doi.org. Adding doi.org to the block list is probably going to work, for that reason. However, blocking doi.org is like blocking t.co or Bitly or other URL shorteners, or indeed like removing the plugin altogether. In all these cases, we're blocking access to a tool whose intended use is quite broad, because a very small subset of affected sites are unhappy with the way the tool works.

The CAC dev team will talk about this issue during today's standing call to see if we can come up with a path forward.

Boone Gorges wrote in #note-15:

...

Raffi, you're almost certainly correct that ACM is counting redirects from doi.org. Adding doi.org to the block list is probably going to work, for that reason. However, blocking doi.org is like blocking t.co or Bitly or other URL shorteners, or indeed like removing the plugin altogether. In all these cases, we're blocking access to a tool whose intended use is quite broad, because a very small subset of affected sites are unhappy with the way the tool works.

I'm OK with blacklisting "doi.org." I'm pretty confident that DOI links will persist.

In the old days, I was only received notifications for the tickets that I created and still want that setting, because I just received over 20 notifications. Is there a way to remove me from all notifications other than the ones I care about?

Thanks, Boone.

Stephen, that's a setting at https://redmine.gc.cuny.edu/my/account. I've made the change for you.

Boone Gorges wrote in #note-17:

Thanks, Raffi. I don't think that doi.org redirects will break, but the destination URL might, which is something that broken-link-checker would presumably catch.

Sorry, Boone. I meant to say that I'm also pretty confident that the destination URLs coming from doi.org will be persistent.

Stephen Klein wrote in #note-12:

The activity on Fri Jun 21 09:02:48 PDT 2024 is coming from:
146.96.128.221

Attaching the detailed log.

This isn't going to add much to the discussion, but I just took a look at this log and can confirm that Broken Link Checker is the cause of the external pings.

The user agent in the log is from an older version of Chrome from close to two years ago. Since this Chrome version is quite old, I checked this user agent against our codebase and I can confirm that this user agent is used by Broken Link Checker -- https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/modules/checkers/http.php#L179 .

I think adding the content publisher's domains to BLC's exclusion list should resolve this issue.

Thank you, Raymond. Appreciate your work on this.

Let's roll with Raffi's blocking of doi.org for a week or two. Stephen, please let us know if you get more hate mail from the publisher. If not, the Commons team agrees that adding doi.org to the global exclusion list for broken-link-checker is the logical next step.

BTW, I don't think that ACM is tracking the redirects on second thought. What I feel is happening is that the plug-in is resolving the redirect and then checking the resolved destination URL.

If that's true, perhaps the bug in the plug-in is that it doesn't consult the blacklist when resolving redirects.

According to the BLC codebase, if a link is excluded, then it is omitted from link checks entirely: https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/core/core.php#L3086

• https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/modules/checkers/http.php#L258
• https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/modules/checkers/http.php#L337

This lines up with Stephen's provided audit_log(1).txt as each link is hit twice. It looks like we can omit this second check by using the following snippet -- add_filter( 'blc_use_get_checker', '__return_true' );

Boone Gorges wrote in #note-24:

Let's roll with Raffi's blocking of doi.org for a week or two. Stephen, please let us know if you get more hate mail from the publisher. If not, the Commons team agrees that adding doi.org to the global exclusion list for broken-link-checker is the logical next step.

Thanks, Boone. I will.

Raymond Hoh wrote in #note-26:

According to the BLC codebase, if a link is excluded, then it is omitted from link checks entirely: https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/core/core.php#L3086

If a link isn't excluded, then it looks like the BLC codebase can do up to two checks for a link:

• https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/modules/checkers/http.php#L258
• https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/modules/checkers/http.php#L337

Raymond, I believe you have confirmed my suspicion! The core module checks if it is "allowed" to check the link. For example, before I added "doi.org" to the blacklist today, the core module would then proceed to check a doi.org link. Now, the doi.org link resolves to an ACM link because it is a redirecting service. What actually happens is that, despite ACM being on the blacklist, the http module never checks whether it is allowed to check the link or not. Thus, we have a situation where a link is checked despite being on the blacklist, which is what I think was happening last Friday.

This is all happening because of this line in the http module: https://plugins.trac.wordpress.org/browser/broken-link-checker/tags/2.2.4/legacy/modules/checkers/http.php#L192:

curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );

Here, curl is set follow redirects (see https://curl.se/libcurl/c/CURLOPT_FOLLOWLOCATION.html). What it should do instead is not follow redirects and instead:

1. Resolve the redirect.
2. Check if the destination URL is on the blacklist before following it.

In other words, following redirects should be done "manually" since there is a blacklist to be consulted. The http module doesn't have the blacklist.

In fairness to the plugin authors, I don't consider this a "bug". The purpose of the plugin is to identify broken links. A link that contains one or more 30x redirects is "broken" if one or more of the links in the redirect chain is broken. So it's perfectly natural for the plugin to follow redirects.

It would be a nifty feature if the plugin instead did not follow redirects blindly, instead going to link 0 in the chain, then 1, then 2, and doing allowed/disallowed checks on each one of these. But building this is non-trivial, if for no other reason than that it'll increase execution time considerably, which could cause resource utilization issues. So other mitigations would be necessary.

More generally, doing this kind of improvement to a third-party plugin that is not central to our work on the Commons is not viable. We try to be good stewards of the tools we use by volunteering fixes and improvements, but we also have to be good stewards of our limited development resources. So I think that we'll probably have to be satisfied with other solutions, like the proposed block on doi.org URLs (ie, any redirect chain that has doi.org as item 0)