CUNY Academic Commons

I've been doing some thinking and research about this and I have some initial thoughts.

Pre-upload checks using pdf.js¶

First, I think we can do fairly lightweight pre-upload checks using the pdf.js library. This can be done totally in-client, which means it's very fast, and we can make it blocking, in the sense that we can prevent upload if we detect that certain pre-upload checks have failed. See eg https://www.nutrient.io/blog/pdfjs-accessibility-structtree-printing/ Specifically, pdf.js can check the following:

- getMarkTree() tries to get the tag tree for a PDF. If it fails, it means the PDF has no tags. If a PDF has no tags, I think it's an automatic a11y fail. See https://www.section508.gov/create/pdfs/common-tags-and-usage/
- getTextContent() lets you check whether a given page has any extractable text content. Scans that have not had any sort of OCR will fail this check. PDFs that have pages that include only photos, maps, etc will also fail this check, so it's not watertight, but it's a helpful heuristic.
- getMetadata() checks for things like document title and language, so it can be a reasonable proxy for other accessibility features.

It's possible that a document could fail these checks and still be OK (ie an image-only PDF with alt text for the images) so I think we need some sort of bypass. I worked with Claude to generate a couple "pre-upload" mockups that show you how we might integrate these sorts of pdf.js-based checks into the existing media-upload flows - ways to block upload but provide a 'bypass' button, and ways to give context and links to external documentation.

Server-side checks using veraPDF¶

veraPDF is a robust solution that does two things. (1) It does a more in-depth analysis of accessibility on PDFs. See https://docs.verapdf.org/validation/ for details, but essentially it does a point-by-point validation on all the ISO standads for PDF accessibility. This means that any feedback from the tool will be far more in-depth and detailed, and much less subject to false positives/negatives, than the heuristics we would leverage in pdf.js. (2) It has tools for remediating PDFs to meet these accessibility standards (something that the proposed pdf.js pipeline doesn't include at all).

Setting up veraPDF would be challenging. It would require separate a separate server environment, with whatever additional costs and developer time this entails. Then, we have to have networking setups to allow the Commons to speak to veraPDF in a secure and private way. The fact that it's a remote and non-instantaneous process means that we can't really build a "blocking" mechanism - we would have to run post-upload processing on the files, and then have a way of reporting the results back to the user. (see the "Media grid" and "Editor" mockups for what this might look like.) There are also some technical challenges related to this type of post-upload processing combined with the way WP works; WP inserts uploaded file paths directly into the post content, but if you fix an already-embedded PDF in a third-party tool and then re-upload it to your site, it'll have a new URL. So you would need some sort of "replace this file" functionality (or some sort of search-and-replace), which doesn't currently exist.

As Laurie notes, the CUNY AI tool appears to use veraPDF in at least part of its pipeline. If we could partner with them, we might be able to offload at least some of the running costs that I discussed above. (The WP-specific ones would still have to be solved.)

I feel like a decent strategy is to begin with the easy thing, which is the pdf.js, client-side checks. It's error prone (both false positives and false negatives are possible), but it's fairly straightforward to imagine how we'd integrate it into the existing interfaces on the Commons. And it would be a meaningful step toward strongly discouraging or altogether preventing the upload of non-conforming PDFs. The veraPDF integration is far more complicated, both in terms of design and in terms of technical lift.

There's the further question of how and whether we would ever run a tool to bulk-remediate existing items on the Commons. It's possible to imagine a veraPDF (or CUNY AI Lab) based workflow here, but to my mind it's really a separate issue from that of preventing or discouraging the upload of non-compliant PDFs.

Sorry about the delay here (and thanks to Laurie for the Brightspace guidance). I've attached a screencast capture of the Brightspace accessibility situation when one uploads a non OCR PDF. The platform lets the upload proceed just fine, but then it shows you a red icon indicating a low accessibility score and a couple of paths to converting the file, along with educational prompts about the associated issues and a request to replace the file. Let me know if I can help with anything else.

HI all

sorry for the long video but wanted to share this in addition to Colin's video. You can use the ally tool to remediate PDFs in a variety of ways in Bright space. It actually worked okay this time around, but I have seen it do really wonky things to PDFs too. And though it worked, the user (me) did not really have full control to remediate the pdfs, for example I could not change the Tags that were assigned to a PDF (from a paragraph to title or header tag) I could only approve the tags in order to up my score.

FWIW - I have heard of some departments around CUNY asking professors to make sure their accessiblity score is at least a 90 or 98, depending in department standards. This is a really random metric since different courses will have differnet kinds and amount of content, and since you can "game" your access score by say, uploading very little (and so it remains high) or by using this tool to "remediate" content which increases the score even if its not correct.

Circling back on this after Tuesday's dev call discussion. The videos above are very helpful, and they demonstrate that Ally is trying to do something far more complex than what we can possibly do on the Commons. At least at first, we should focus on informing users about accessibility issues, and then pointing them toward external documentation and tools that allow users to remediate themselves. This is not a cure-all, and it still allows people to upload non-compliant PDFs. But, based on the videos I've seen, so does Ally - so at least on that front, we'd be adopting a similar policy. Laurie and others seemed to agree that the inform-only approach still adds real value, by acting as an educational tool and a nudge toward improved accessibility for newly uploaded items.

Several times during our discussions on this topic, the issue of existing PDFs has come up. I think we can and should discuss what it would look like to remediate these. But, to my mind, it's really a separate discussion, for several reasons:
- The technology must be server-side rather than client-side. In this ticket I'm proposing we start with a client-side-only check, for ease of implementation. So remediation of existing items would require a totally different technical approach anyway.
- Changing user content without the consent or involvement of the user is very thorny. It forces questions about content ownership. It makes the cost of errors - false positives, false negatives, mangled content - much higher, since content owners won't be involved to catch these errors.
- The design decisions involved in point-of-upload, single-PDF uploads are totally different from what would be required for (largely invisible) bulk remediation.

All this being said, I think we're coming to general agreement that the point-of-upload informational scan is worth doing. So let's slot this for work during the academic year.

Sara, I've added you as a watcher here. Perhaps you could start by looking at the AI-generated mockups I posted at https://redmine.gc.cuny.edu/issues/25901#note-3 to get a sense of the kinds of UX interventions we're thinking about. We'd also want to think about possibilities for informing users about a11y problems in PDFs after the initial upload point. So, for example, I upload a PDF; I see a notice that it has a11y problems but decide to go forward anyway; I embed or link the PDF in a blog post; should I see some sort of indication in the block editor that there's a problem with the PDF when editing the post, so that I can take action later on?

Aside from the primary point-of-upload UX and technical work, we'll also need to ensure that it's possible to replace media items (so that users can go to an external service, return to the Commons, and replace the original PDF with the remediated one, without breaking links and embeds). https://wordpress.org/plugins/enable-media-replace/ is a popular WP plugin that may be able to give us a technical foundation, or at least some design inspiration.