CUNY Academic Commons

Ray and Jeremy, I've never run a large site with S3-offloaded content, and I've definitely never run a migration of an existing site. Have you?

I've worked on a multisite site that uses S3 Uploads. We maintain a fork that addresses a few issues. See
https://github.com/humanmade/S3-Uploads/compare/master...hwdsb:S3-Uploads:hwdsb-mods.

For the Commons, this would namely include the following:
- Fixes an issue with some older multisite URLs that use /blogs.dir/ in their uploads directory: https://github.com/WordPress/WordPress/blob/master/wp-includes/ms-default-constants.php#L31. The /blogs.dir/ uploads directory would apply for sites that existed before WordPress MU was merged into WordPress Core. S3 Uploads does not take this into account. We probably have older sites that use /blogs.dir/ for their uploads directory as well so this issue would apply to us as well. I forget the particulars of this issue, but I note this just so we are aware.
- Incompatibility with Gravity Forms. As you mentioned in point 2, there is an issue here with some non-standard plugins. I didn't look too far into the actual issue with Gravity Forms. I'm doing a dirty bail fix in the s3-uploads fork.
- Image URL rewriting via filters needed to be done as well for themes using a custom header image and background image since these URLs are written into the DB as theme mods and these URLs were referencing the local URLs instead of the S3 URLs.
- Also for this site, BuddyPress avatar uploads remain being served locally.

Reclaim is serving as our vendor for AWS (ie we're paying Reclaim, and they're paying AWS), so we would need Reclaim to help us configure our bucket(s) in order for us to move forward with this.

Do we have an estimate on the potential cost of using AWS? This could be in the thousands of dollars per year.

Ray, thanks so much for this!

- Fixes an issue with some older multisite URLs that use /blogs.dir/ in their uploads directory

Do you think Human Made would accept a PR for this? Seems like a general problem.

Do we have an estimate on the potential cost of using AWS? This could be in the thousands of dollars per year.

It's rolled into the top-line number that Reclaim is charging us. This is by design: I didn't want our team to be responsible for covering these variable costs, not to mention the overhead associated with configuring, maintaining, troubleshooting, itc.

Reclaim has asked what we'd like to use as the rewrite domain for S3-stored uploads. By default, S3 URLs are long and unwieldy, but we can rewrite them as something like files.commons.gc.cuny.edu/sites/1234/2024/01/foo.jpg. Are we OK with using files.commons.gc.cuny.edu for this purpose, or is there a better idea floating out there? We should decide this soon, because it'll require a DNS change at the Graduate Center, and I would like to be able to include this ask in our initial round of communication with GC IT.

S3-Uploads is now running in the Reclaim dev environment.

At first, I tried running Ray's fork. But this caused a couple problems. First, it used an old copy of the AWS SDK, which made it difficult to debug. Second, the blogs.dir fixes didn't work right; from my reading, they assume that the webserver kept the upload files in blogs.dir, but that on S3 they'd be in the /uploads/sites/ bucket. I don't think it's necessary to do this: we can simply upload the blogs.dir and the uploads buckets separately, and everything appears to work correctly with the latest S3-Uploads, out of the box.

Basic media uploads, as well as BP avatars, appear to be working properly. I'll be assembling a list in the upcoming days of all upload types I can think of, and then I'll run some preliminary tests on each. Based on those tests, we can decide whether it's necessary and/or desirable to filter non-standard upload types to S3.

As a reminder, in today's dev call we discussed point 3 above and tentatively decided to do the following:
a. Sometime in the upcoming weeks, get the production S3 bucket set up
b. Modify the S3-Uploads plugin so that it can upload existing content to the bucket, and also sync new uploads. Activate that modified plugin on the legacy production site.
c. Run the upload-directory tool in the s3-uploads plugin. Start with small amounts of content to ensure it doesn't crash the production site.
d. When the Reclaim production environment is created and we're near switchover time, hook it up to the pre-filled S3 bucket.

I'll begin working on the necessary mods to the S3-Uploads plugin, and I'll share them here for review before we deploy them to the legacy site.

At first, I tried running Ray's fork. But this caused a couple problems. First, it used an old copy of the AWS SDK, which made it difficult to debug. Second, the blogs.dir fixes didn't work right; from my reading, they assume that the webserver kept the upload files in blogs.dir, but that on S3 they'd be in the /uploads/sites/ bucket. I don't think it's necessary to do this: we can simply upload the blogs.dir and the uploads buckets separately, and everything appears to work correctly with the latest S3-Uploads, out of the box.

I think the other issue with the site I was working on is they were migrating from another S3 storage plugin, WP Offload Media, over to Human Made's S3 Uploads and they were using an existing S3 bucket rather than starting from scratch. I'm glad that this isn't a problem with our install!

Basic media uploads, as well as BP avatars, appear to be working properly.

Glad that this is working as well!