Antonia Levy reports that her team has been experiencing an issue with the following site for the last few weeks:
Several images have disappeared from some of the pages. First, we thought it was our mistake but it keeps happening without any of us touching the pages.
We discovered another instance today, and I collected the URLs of the missing media files (before we edited the page as our faculty are using them as reference):
The URLs seem wrong to me, but I am not sure. I tried to upload an image and its URL came back with the file system url. Shouldn't it be the relative address (i.e. without the "wp-content/blogs.dir/1145"?
Anyway, they have a lot of media with URLs of this nature - I am not sure if this is causing the issue she reports, or if it is something else.
#1 Updated by Boone Gorges over 2 years ago
- Status changed from New to Reporter Feedback
Thanks for the report, Scott.
The URL is not the issue. URLs like spsfaculty.commons.gc.cuny.edu/files/2017... are rewritten internally to ...edu/wp-content/blogs.dir/1145/files.... The latter are the "true" URLs, and are used by default on the Commons in many cases.
I can confirm that the files here do not exist on the server. Based on the folder last-touched date, the files in question were deleted on January 25, 2019.
These fact that these files are in the /2017/11 folder means they were updated via the Dashboard > Add Media (or another corresponding WP media tool). This means there'd be an 'attachment' object in the posts table corresponding to the object. I've looked, and I don't see these objects. Posts have numeric IDs that should be sequential, and I see several gaps in the sequence for items created in November 2017. This suggests that the attachments were deleted through WordPress - it's not a case where files have gone missing from the server, but instead a case where WordPress was responsible for deleting the attachments. This could happen through the interface (there's a Delete Permanently link in the Media UI) or through some script (plugin, etc) that deletes content.
I suspect that this may possibly be linked to #10821. Can Antonia provide information about what user originally uploaded these images? Even a few possible names would help - if I find that all the potential user accounts still exist on the Commons, then we can rule out this possibility.
Unfortunately, the deletion took place long enough ago that we no longer have easily restorable backups. MySQL binary logs would probably provide clues (see eg #10997) but these are no longer available for the time in question.
If Antonia or another admin is able to provide more details, either about the circumstances of this specific instance (user who uploaded, date it went missing, etc), or about other instances, it would help me to narrow down the possibilities.
#2 Updated by scott voth over 2 years ago
Here is response from Antonia:
The users who’d have uploaded these images would be either
Also, here is another page where the images are broken (this one was created a while ago so these aren’t recent uploads):
#3 Updated by Boone Gorges over 2 years ago
Thanks for these details.
The three user accounts in question still exist, so my theory about deleted users is probably not correct.
The missing file at https://spsfaculty.commons.gc.cuny.edu/wp-content/blogs.dir/1145/files/2018/04/2018-04-25_1319.png also appears (based on filesystem timestamps) to have been deleted on January 25. Looking back through all directories related to this site, the following folders were touched at the same time (blogs.dir/1145/files)
2017/11 2018/03 2018/04 2018/09
The specificity of this - only a handful of items appear to have been deleted - causes me to suspect that there was some user action at the time in question (10:08am on Jan 25 2019) that triggered these deletions.
I'm going to send a note to Lihua in IT to see if we can pull any backups or old logs that might help us either to restore the lost items, or to diagnose the incident.
#4 Updated by Boone Gorges over 2 years ago
- Target version set to Not tracked
I heard back from Lihua, and we do not have backups going back this far. So it's not possible either to diagnose this far into the past, or to restore the data.
As noted above, my guess is that this was a one-time issue, caused by some user action. I'm unsure whether it was "intentional" action (someone deleting when they shouldn't have been) or a bug (content being deleted as a side-effect to some other action), but the evidence suggests that it's not a chronic problem that happens of its own accord.
I'm afraid the only way I can do more debugging is if the problem recurs, and the team lets me know about it while we're still in the backup window (~30 days).