CUNY Academic Commons

I've also confirmed the behavior.

It's specific to X-Sendfile. When disabling Sendfile https://github.com/cuny-academic-commons/cac/blob/2.2.x/wp-content/cac-files.php#L155 the image loads as expected.

This may be a similar issue to what's happening in #19055.

Apache is serving 500 errors for all user-uploaded images on https://lmistest.commons.gc.cuny.edu/. Unfortunately, there's no corresponding notices in our httpd error logs, which makes it hard to do further debugging.

A couple lines of thought:
1. There could be file-permission or ACL issues that are preventing Apache or sendfile from serving the file. But I tested with the very same image uploaded to a different site, which had the same blog_public setting, and I was not able to reproduce the problem. https://boonesparty.commons.gc.cuny.edu/test-image/?bbgd=1 getfacl and ls -la are reporting the same schema for upload directories on both the working and non-working sites. So I don't think that file permissions are at the root of it.
2. It could be that we'd get better information from a higher Apache LogLevel. This will require coordination with IT, and it may affect sites other than the Commons. It might also turn out that IT doesn't feel comfortable making this change. Moreover, a higher log level is going to result in much higher log utilization, which has potential performance implications. For this reason it would be helpful if we could work closely with IT to pick a specific time for testing, so that LogLevel could be toggled up then toggled back down right away. But this kind of coordination is going to be difficult.
3. Because X-Sendfile works by passing the request along to the kernel's sendfile system utility, it could be that the problem is at the OS level. This means that relevant errors, if there were any, would be reported in system logs like journalctl. Here too I'd need support from IT to get any information.

Ray or Jeremy, do you have ideas about next steps? If you had ideas about how we could do more debugging on our end, it'd be especially helpful - I only want to go to IT if we feel like we have concrete things to ask, and if we've exhausted all reasonable possibilities that are under our team's control. If not, I'm thinking that a reasonable next step might be to send a message to Ming in IT that summarizes the issue, asks for them to check journalctl (something that should be quick), and asks if they have other ideas for debugging.

Laurie, As a temporary workaround, I've added an "exclude from sendfile" list to cac-files.php on the production server. As a result, images are now loading as expected from the site in question. If Ray or Jeremy want to experiment with anything, feel free to have a look or modify this:

149 // For debugging: Domains to exclude from Sendfile support.
150 $exclude_from_sendfile = [
151     28514, // lmistest.commons.gc.cuny.edu
152 ];
153 
154 // Optional support for X-Sendfile and X-Accel-Redirect
155 if ( WPMU_ACCEL_REDIRECT ) {
156     header( 'X-Accel-Redirect: ' . str_replace( WP_CONTENT_DIR, '', $file ) );
157     exit;
158 } elseif ( WPMU_SENDFILE && ! in_array( get_current_blog_id(), $exclude_from_sen    dfile, true ) ) {                                                               
159     $wpdb->close();
160     header( 'X-Sendfile: ' . $file );
161     exit;
162 }

WPMU_SENDFILE is false by default: https://github.com/WordPress/WordPress/blob/a2bcc36278ea932406a3e0044a6d9f96c8493c10/wp-includes/ms-default-constants.php#L108 . However, it looks like we set this to true on production's cac-env-config.php. Should we comment that line out?

I'm concerned with the performance implications if we disable Sendfile across the board. That change would mean that anything that's routed through /files/ would be served via PHP's readfile(). If you're downloading a many-MB image or PDF, this could result in PHP threads being held open for a very long time. I suppose we could give it a try and monitor performance, but here too we'd probably need help from IT to get an accurate sense of whether the performance degradation is severe.

I disabled the Jetpack plugin on lmistest.commons.gc.cuny.edu and that appears to have fixed things. I haven't looked into what Jetpack is doing that might be causing the 500 errors though.