Project

General

Profile

Actions

Bug #19400

open

Images not loading

Added by Laurie Hurson 3 months ago. Updated 3 months ago.

Status:
New
Priority name:
Normal
Assignee:
-
Category name:
-
Target version:
-
Start date:
2023-12-11
Due date:
% Done:

0%

Estimated time:
Deployment actions:

Description

Hi All,

This is the LMIS test site I was refering to on the team call last friday.

The web desginer got in touch with me to report that he is having trouble with the images loading slowly or not at all and I am able to re-create this issue in firefox and chrome.

https://lmistest.commons.gc.cuny.edu/

Are you seeing this issue? Can you look into why these images are not loading?

I have asked for more info about any plugin installs recently, but I met with him last Wednesday and the site was working fine. I will send any more info he can provide soon.

Thanks,
Laurie


Related issues

Related to CUNY Academic Commons - Bug #19055: Images not appearing on siteResolved2023-10-19

Actions
Actions #1

Updated by Boone Gorges 3 months ago

  • Related to Bug #19055: Images not appearing on site added
Actions #2

Updated by Boone Gorges 3 months ago

I've also confirmed the behavior.

It's specific to X-Sendfile. When disabling Sendfile https://github.com/cuny-academic-commons/cac/blob/2.2.x/wp-content/cac-files.php#L155 the image loads as expected.

This may be a similar issue to what's happening in #19055.

Apache is serving 500 errors for all user-uploaded images on https://lmistest.commons.gc.cuny.edu/. Unfortunately, there's no corresponding notices in our httpd error logs, which makes it hard to do further debugging.

A couple lines of thought:
1. There could be file-permission or ACL issues that are preventing Apache or sendfile from serving the file. But I tested with the very same image uploaded to a different site, which had the same blog_public setting, and I was not able to reproduce the problem. https://boonesparty.commons.gc.cuny.edu/test-image/?bbgd=1 getfacl and ls -la are reporting the same schema for upload directories on both the working and non-working sites. So I don't think that file permissions are at the root of it.
2. It could be that we'd get better information from a higher Apache LogLevel. This will require coordination with IT, and it may affect sites other than the Commons. It might also turn out that IT doesn't feel comfortable making this change. Moreover, a higher log level is going to result in much higher log utilization, which has potential performance implications. For this reason it would be helpful if we could work closely with IT to pick a specific time for testing, so that LogLevel could be toggled up then toggled back down right away. But this kind of coordination is going to be difficult.
3. Because X-Sendfile works by passing the request along to the kernel's sendfile system utility, it could be that the problem is at the OS level. This means that relevant errors, if there were any, would be reported in system logs like journalctl. Here too I'd need support from IT to get any information.

Ray or Jeremy, do you have ideas about next steps? If you had ideas about how we could do more debugging on our end, it'd be especially helpful - I only want to go to IT if we feel like we have concrete things to ask, and if we've exhausted all reasonable possibilities that are under our team's control. If not, I'm thinking that a reasonable next step might be to send a message to Ming in IT that summarizes the issue, asks for them to check journalctl (something that should be quick), and asks if they have other ideas for debugging.

Laurie, As a temporary workaround, I've added an "exclude from sendfile" list to cac-files.php on the production server. As a result, images are now loading as expected from the site in question. If Ray or Jeremy want to experiment with anything, feel free to have a look or modify this:

149 // For debugging: Domains to exclude from Sendfile support.
150 $exclude_from_sendfile = [
151     28514, // lmistest.commons.gc.cuny.edu
152 ];
153 
154 // Optional support for X-Sendfile and X-Accel-Redirect
155 if ( WPMU_ACCEL_REDIRECT ) {
156     header( 'X-Accel-Redirect: ' . str_replace( WP_CONTENT_DIR, '', $file ) );
157     exit;
158 } elseif ( WPMU_SENDFILE && ! in_array( get_current_blog_id(), $exclude_from_sen    dfile, true ) ) {                                                               
159     $wpdb->close();
160     header( 'X-Sendfile: ' . $file );
161     exit;
162 }
Actions #3

Updated by Raymond Hoh 3 months ago

WPMU_SENDFILE is false by default: https://github.com/WordPress/WordPress/blob/a2bcc36278ea932406a3e0044a6d9f96c8493c10/wp-includes/ms-default-constants.php#L108 . However, it looks like we set this to true on production's cac-env-config.php. Should we comment that line out?

Actions #4

Updated by Boone Gorges 3 months ago

I'm concerned with the performance implications if we disable Sendfile across the board. That change would mean that anything that's routed through /files/ would be served via PHP's readfile(). If you're downloading a many-MB image or PDF, this could result in PHP threads being held open for a very long time. I suppose we could give it a try and monitor performance, but here too we'd probably need help from IT to get an accurate sense of whether the performance degradation is severe.

Actions #5

Updated by Raymond Hoh 3 months ago

I disabled the Jetpack plugin on lmistest.commons.gc.cuny.edu and that appears to have fixed things. I haven't looked into what Jetpack is doing that might be causing the 500 errors though.

Actions

Also available in: Atom PDF