CUNY Academic Commons

If it's OK with Matt and Micki, I'll do a rollback of the plugin, and an emergency release within the next day or two. Kimon, can you give me a short list of things to check pre- and post-update in my staging environments, so that I can verify that the rollback works as expected before releasing?

Yes, of course. Thanks for your help with this, Boone, and apologies for the issues, Kimon!

Kimon, I haven't heard from you regarding things to check, so I'm going to go ahead and do the rollback and cross my fingers about it - the longer I leave the 3.x version in place, the more problems there may be with the rollback.

Fixed in https://github.com/castiron/cac/commit/6753dbce704491e6fc20d9982bfa5031879a0f04. Release coming momentarily.

The release is live.

Thanks, didn't see your prompt replies until just now.

Rollback worked splendidly and things are now back to normal. I will get to work on my dev side with Events 3.0 to start matching its upcoming changes and pay attention to back channels to find out when things are more stable

Here is a list of things that were broken (not just style/plugin changes) that I was on the lookout for:
- past events visible
- non-Events plugin-in pages with Events plug-in queries working properly
- category classes being properly rendered within class="" html tags
- no missing </div> tags on certain pages breaking sidebar functionality

Wonder if there might be some way of developing a calendar of when you are updating themes and plugins so that developers on the Commons can be prepared for upgrades. We had the same issue when Thematic got upgraded last year if you remember.

Either way thanks to all for such a prompt reply on this.

Rollback worked splendidly and things are now back to normal

Great, thanks - and thanks for the checklist.

Wonder if there might be some way of developing a calendar of when you are updating themes and plugins so that developers on the Commons can be prepared for upgrades. We had the same issue when Thematic got upgraded last year if you remember.

We do have a release calendar: during the academic year, updates happen on the 1st, 11th, and 21st of each month. As a rule, unless there is a pressing reason not to do so, each of those releases will include all available plugin and theme updates. I know that this is kind of a pain for sites like yours. In an ideal world, we'd be able to review individual plugin and theme updates manually before each release, but that'd require resources an order of magnitude greater than what we currently have available to us. Alternatively, we could spread our releases out more, so that we only upgraded plugins and themes every, say, 2 or 3 months, which would give both our dev team and our partner developers (like you) plenty of time to review changes. However, Matt and others in our community have been reluctant to adopt shorter release cycles.

That said, I'm happy to hear any suggestions you might have for a better system. It might be, for example, that we could set up a list of plugins that are of particular importance to particular members of the community, and we could try to devote more resources only to those plugins.

I'd also be interested in hearing your suggestions, Kimon. One thing you might do is subscribe to the feed of our Dev blog - http://dev.commons.gc.cuny.edu , where site update details are posted with every release

I think that what Boone is suggesting is i part what I am thinking of. I certainly am sympathetic that this could become a hairy management mess if it became more finely detailed. But I think there is a way so stagger and release a little information to the dev community ahead of time that would make it easier for us. What happens now is that a massive change happens and the live site is broken for the time it takes me to adapt to the new release of the plugin. You guys have been great with that (both this time and last year when Thematic went wonky) but it is stressful for all when things have to be submitted really quickly or plugins have to be rolled back. As a developer I am also completely happy to follow the schedule, and understand the change over the summer, it is just that in those panic situations it can be an entire month when a site is down and that can be really problematic, especially when you have upcoming events or are planning on releasing a journal issue (BTW I have no complaints about the current schedule and think that it is working really well) How is this:

1. When the possibility of a new plugin or theme update becomes apparent (I take it you are following the alerts on the multi-site admin side) you can tag that plugin for updating in two to three months time on the regular Commons update schedule rather than updating it right away.
2. Then the list and schedule of what plugins and themes will be updated in future releases (and on which particular releases) could be put on the Dev Blog and developers can follow the feed to see what updates are coming down the pike and are relevant to their sites. (The problem with the Dev feed as is is that I only see the updates after the fact.)
3. Then developers would have at least a month to tweak their sites on our own dev sites and get you updates in plenty of time for the subsequent update in the Commons schedule, even if it is summer time.

I think that that would require minimum management on the Commons team side. Just keeping a schedule of what prolonging of plugin updates is happening when, and one more announcement on the Dev Blog for each release that announces which plugins are newly pending for updates. One caveat to this would be that you guys of course have ability to override the waiting period if an update patches a security hole or anything vital like that. This puts the onus on developers to follow the progress of the Commons's update schedule and still allows us to do the checks with our themes, etc. so that you don't have to take on the resources issue Boone importantly mentions above.

How does that sound?

Hmmm. I like the idea of notifying people ahead of time about changes that are coming, but I'm uncomfortable with delaying updates for 2-3 months.

One thought I have is to create a group on the Commons that is geared especially towards people who are actively developing sites on the Commons (ie, doing things that are more complicated than simply enabling a theme -- making child themes, etc.), and to give that group warnings a week or so ahead of site updates.

That might work for us, but it would place an additional onus on our development team that could become significant with our current schedule of 3 updates a month. So, even better than that might be some kind of system that is integrated into Redmine, since we're already using that. Perhaps there is some way of creating a special category or tag and then subscribing interested people to that tag so that they'll be notified of forthcoming changes.

Whether or not that works, I'd be most in favor of finding a way to give people warnings of forthcoming changes without adding a whole bunch of new steps to our workflow.

I'm just curious as to why you would think updating themes and plugins on a month+ delay is that much of a problem? I would imagine most Commons users don't even notice upgrades, while the gains for giving developers a little longer can be substantial. This is particularly true when an upgrade is a significant change and since we don't have a choice to stay with a legacy plugin on the Commons have to perform what may be considerable changes to site functionality.

I'm just curious as to why you would think updating themes and plugins on a month+ delay is that much of a problem?

A couple reasons. One, if I say today that I'm going to upgrade plugin X to version Y a month from now, it's likely that that the plugin will be at version Y.Z when the time actually comes. This is a lot to keep track of. The more important reason is that the WP plugin ecosystem is set up in such a way that plugin upgrades are frequent and small rather than infrequent and large. Most plugin upgrades do not represent major rewrites (like this version of Events Calendar) or even new features, but instead fix bugs. Some bugs are serious or security-related; some are not. You're probably that few users notice when some of the minor bugs are fixed. However, over the long run, a strategy of upgrading plugins to the latest versions as soon as they're available will fix more small issues and add up to more general benefit than holding off all plugin upgrades by a few months.

I recognize that there is a spectrum between the "upgrade everything immediately" strategy and the "upgrade everything later" strategy. I think we can find someplace on that spectrum that'll represent a good balance of the concerns at hand. What I am definitely saying, though, is that the "upgrade everything later" strategy seems like it goes way too far.

Here are a couple of ideas about how to strike the balance:

- Upgrade immediately for minor plugin releases (x.y.z). For major releases (x.y), do some sort of advance warning of the type Kimon suggests.
- When we set up relationships with devs like Kimon, allow them to create a list of mission-critical plugins. Compile those lists, and do some sort of advance warning before upgrading any of those plugins.

Happy to hear other thoughts.

Boone / Kimon: I am really liking the thought processes in this issue thread about release strategies for plugin/theme releases.

I see us talking about two separate, but intimately linked issues - release/roadmap definition and verification process.

As articulated, plugins and themes can really complicate a 'minor' (or 'major') release if they are unwieldy and tangle up the code - on the other hand, many of the theme and plugin releases don't pose these kinds of complications to releases, and to impose a standing monthly delay on their implementation could pose more problems (dealing with a month of potential obsolescence, etc.) than they could solve.

It strikes me that, as Boone suggests, if we can put together a list of mission-critical plugins and themes, indicating the pages and code snippets they are likely to affect, we can use such a verification checklist as a pre-deployment QA phase for each release. However, we will need to keep it optimized to avoid burning cycles on each release. To minimize the time soak, we might want to look at deprecating other problematic plugins and/or themes if they are largely unused... as we add more and more themes and plugins, such a list will likely increase in scope and releases will be prone to greater disruption.

Also, we may want to consider formalizing our schedule with another 'type' of release - perhaps one of each three monthly 'minors' - one that is dedicated to introducing and testing specific plugin and/or themes that we know to be complex. Such a monthly 'plugin/theme only' release could take place in one of the minor release slots, but limited in scope to verifying new/updated plugins and/or themes. This would ensure no more than 10-20 days between the introduction of an unexpected issue and its resolution (and would not preclude emergency rollbacks and fixes of the kind Boone conducted on the Calendar plugin).

Thanks, Boone and Micki. The steps Boone laid out seem like reasonable steps:

- Upgrade immediately for minor plugin releases (x.y.z). For major releases (x.y), do some sort of advance warning of the type Kimon suggests.
- When we set up relationships with devs like Kimon, allow them to create a list of mission-critical plugins. Compile those lists, and do some sort of advance warning before upgrading any of those plugins.

Micki, I'm not sure about your last suggestion; if we implement Boone's steps above, I think it won't be needed, and I'd like to keep our workflow as simple as possible. But Boone, if that sounds like it would be useful to you, please say so.

Matt: Agreed - the reason for my suggestion was that, given the significan timpact of the upgraded events calendar on this minor release, we may not be able to differentiate plugin/theme releases as 'minor' v. 'major'. By segregating out plugin/theme upgrades to dedicated 'minor' releases, we may better be able to contain/respond to cascading issues from plugin and theme upgrades.

And you're absolutely right - Boone's recommendation will give us a sense of the best process 'on-the-ground' to handle these kinds of issues moving forward.

Boone, any thoughts about the usefulness of having an occasional 'plugin/theme'-only minor (or indeed, major) release?

if we implement Boone's steps above

I'd proposed these two items as two separate suggestions, not as two parts of a single solution. IMO, in our current situation, doing both of the things I suggested is too much overhead. We should start simple and implement a single piece of additional workflow now, and we can always refine later on.

On that note, I'm going to suggest the following. (Dates are off the top of my head; feel free to suggest something else.)

- Continue to upgrade minor dot-dot releases for plugins and themes immediately, as we've already been doing. Skip major dot releases for plugins and themes for our thrice-monthly releases.
- On the 5th of each month, a member of our team will review all available plugin upgrades and compile a list of major dot releases.
- We'll post that list to the dev blog, with a warning that they will be upgraded for the release on the 21st.
- For the 21st release, we will upgrade the major release plugins to the versions announced on the 5th, or any minor versions in that series that may have been subsequently released. In other words, if on the 5th there is a 1.7.3->1.8.1 upgrade waiting for a given plugin (and that's what we announce in our dev blog post), on the 21st we will upgrade to the latest 1.8.x of that plugin (but not 1.9.x, if it's come out in the interim).

That'll give our partner devs 16 days' notice, ensures that plugins are never more than a month-and-a-half out of date (as when a major upgrade is released on the 6th of a month, when we'd announce it on the 5th of the following month and upgrade on the 21st), and adds relatively little overhead to our dev process (the process will be self-documenting; the blog post on the 5th will also serve as an upgrade checklist for the release manager on the 21st).

Does this seem like a reasonable compromise? Keeping in mind that we can always add more overhead later on.

Sounds fine to me, with the caveat that plugin updates that address critical security issues may necessitate reduced warning periods. Aside from that, though, this sounds like a good compromise. Thanks, Boone.

Thanks so much for this great conversation and for the accommodation to developers in the new plan. It really is only major plugin and theme upgrades that cause the big problems (such as the Event Calendar step up to 3.0). It would be great if the plugin could be the 1st rather than the 21st to give three rather than two weeks, but understandable at this point if you want to keep the delay on the version plugins to a shorter time period.

Thanks for this! We remain CUNY Academic Commons proud over at the MESTC, especially as we are about to deploy three new themes for three new sites including two OA journals (European Stages and Journal of American Drama and Theatre) and the site for Prelude '13.

Fantastic, Kimon. When you debut the new journals, please start a new ticket under the Publicity tracker to remind us to create some homepage slides that highlight the new publications.

Great, didn't even know there was a publicity tracker!

Saw this on the Modern Tribe (Events Calendar developer) website today: RELEASE: EVENTS 3.0.2 / EVENTS PRO 3.0.4 / ADD-ONS 3.0.1 http://tri.be/release-events-3-0-2-events-pro-3-0-4-add-ons-3-0-1/

Apparently they hadn't even officially launched the damn thing. No wonder it was so buggy and everyone was so pissed at them.

This does do two things however. First of all it reinforces that this plan we are instituting is a good one that will protect us from unscrupulous developer product releases. So thankful for the conversation we've been having. Secondly, since this popped up in my new RSS category "Important" (which now includes the Commons Dev blog, Modern Tribe's Feed, Thematic's Feed, and the feed from the wiki service the BGC uses) I feel better about staying abreast of when these changes happen and how MESTC can respond to any potential problems ahead of time. RSS feeds FTW

Once again thanks to all.

Cool - thanks, Kimon.