Scripts for quicker provisioning/updating of development environments
As the Commons team grows, more people are faced with the task of creating a local enviroment that mirrors the Commons for their development work. This is currently not an easy task. At a high level, you have to:
- make sure that a *AMP environment is running on your computer, and is roughly compatible with the server software running on the Commons production server
- clone the Commons repo
- download and import some version of the Commons database into your own local database
- do a bunch of wizardry to ensure that either you can switch your local environment to self-identify as 'commons.gc.cuny.edu', or to swap out with a local dev URL like 'commons.local.dev' or 'local.cicdev.com'
- maybe get some representative piece of the data at wp-content/blogs.dir/
Each of these steps is fraught with issues about data integrity, implementation details, and various, unpleasant gotchas.
Let's start to work toward something a bit more organized. I'm going to suggest the following as a starting point:
- Let's use Vagrant for VM management, since it's pretty widely used in the WP community
- Pick a provisioning tool - bash, Puppet, Salt, Chef, etc - for doing environment provisioning. I don't have a ton of experience here, so if anyone has thoughts, please jump in
- Write the necessary provisioning scripts to mirror the Commons fairly well (keeping in mind that we don't control the Commons environment directly, and that the Commons runs RHEL, which means we'll need to use a pretty-close free alternative OS)
- Write some scripts to handle fetching recent versions of Commons data, including a "cleaned-up" version of production data (I'll open a separate ticket for this)
I think this would be a cool summer project. Dom, I'm going to assign to you for the moment, as it seems up your alley, and ask you to give your initial thoughts.
#1 Updated by Matt Gold over 4 years ago
Thanks. FWIW, I can tell you that Zach used Chef for Debates in DH, so you could check in with him on that. And my DH students used Ansible for DH Box (http://twitter.com/dh_box). Here's what they wrote about that:
Stephen then turned to a set of tools explicitly designed for mass provisioning of cloud servers. He evaluated several different possibilities: Chef, Puppet, Salt, and Ansible. Each is an ‘IT automation’ library/toolset. Eventually Stephen chose Ansible as his solution, for both objective and subjective reasons: Ansible allows for SSH-ing into a client computer (a DH Box instance in this case) and watching the provisioning as it happens, live. It is comparatively easy to see when a build fails, and to see what failed in the install script. And Ansible was written in Python, which Stephen felt comfortable with.
#3 Updated by Dominic Giglio over 4 years ago
I would actually enjoy working on this. Boone and I have had a few conversations about the intricacies of our local setups and how an automated process would remove some of the initial pain in getting a development env up and running.
Not sure chef/puppet style tools are the right way to go since they are designed to provision remote cloud based servers - this ticket is talking about provisioning local environments. But I will research them, I'm thinking bash might be the simplest solution. Although using tools like Vagrant or Docker might lend themselves to working more efficiently with a chef/puppet style provisioning setup.
Will report back when I know a little more.
#4 Updated by Boone Gorges over 4 years ago
Great, Dom. Glad you're interested.
Bit more background on my motivation here: there's a lot to be gained from doing development in a VM, which will allow for greater sandboxing, better compatibility with the production server, etc. And using local VMs is not all that different from provisioning remote servers.
See https://github.com/Varying-Vagrant-Vagrants/VVV for an example of a project that uses Vagrant for provisioning local WP dev environments. They use Bash for the provisioning process itself, but not for a principled reason. Here's another similar project using Salty https://github.com/humanmade/Salty-WordPress
#5 Updated by Dominic Giglio over 4 years ago
I agree, a local VM setup would be ideal. I'll check out those links as soon as I can.
I mentioned Docker in my last reply only because I've heard and read that developers who used to use Vagrant find Docker to be a better alternative. I do not know if this is true - I will read up on both (Docker and Vagrant).
I have experimented with Vagrant in the past. I've built custom VM OS images using both Ubuntu LTS and Arch linux. I know nothing about RHEL, I think using Ubuntu wouldn't pose a problem, but if anyone monitoring this ticket wants me to investigate another FOOS OS please chime in.
#6 Updated by Boone Gorges over 4 years ago
CentOS is based on RHEL, and is no-cost, so it's probably the best place to start in terms of OS. A little Github poking shows that there might be plenty of starting places: https://github.com/search?q=centos+vagrant&ref=cmdform
#7 Updated by Dominic Giglio over 4 years ago
I've started researching this and looking into the links you provided above. Would you let me know when you have a few minutes to talk? I'd just like to make sure that you and I are on the same page regarding this project since this is not just a feature that is being added to the commons in general; this is going to be a very interactive project that all of the commons developers are going to use. I don't want to start going off in one direction while you are thinking in another. The call (or Skype chat?) doesn't have to be long, just a quick couple of minutes to make sure I'm understanding and researching correctly to help build what you have in mind.
#9 Updated by Dominic Giglio over 4 years ago
sounds good, I've got Skype open now and will leave it open all day. I'm going to try and split today between some commons and littlstar stuff so will be in front of my laptop almost continually.
Am I one of your Skype contacts? You are in my list. If I'm not in yours let me know and I'll send a contact request.
#10 Updated by Chris Stein over 4 years ago
This sounds like a great project. One other thing you might want to check out is Boxen, by the folks at Github. They wrote Boxen to help set up development environments for people. It uses Puppet. I think the downside is it's Mac only right now but it is probably at least worth a look.
Here is someone who did something similar to Boxen with and Ansible he calls Battleschool (works on Linux too)
and another similar idea with Vagrant and Ansible
I don't really know much about this stuff but I'm interested because I want the IT folks at BMCC to do something similar to set up our lab computers and servers for the students, so I'll be following along what you come up with. I did also talk with the DH Box folks and based on that conversation and some blog posts I've ready Ansible does seem to be more approachable and easier to get into than Chef or Puppet.
#12 Updated by Dominic Giglio over 4 years ago
Chris, thanks for the links and thoughts.
My plan right now is to keep this whole process as simple and plain vanilla as possible. To that end I envision a first attempt simply using a custom VirtualBox Vagrant "Base Box" built on CentOS 6.5 per Boone's suggestion to stay as close to the RHEL environment used on the GC's production servers. Any additional custom config will probably come right out of Bash so no other developer (or non-developer) need learn any crazy new provisioning tool. The whole point of this thing is to make development easier across the group.
My belief is always start simple and add complexity only when needed. If the building of this virtualized container requires advanced provisioning i will probably look at Ansible first. Earlier comments plus your links seem to shed a favorable light on it. I've tried Puppet and Chef and always found them to be more work to configure than writing a few lines of Bash script.
I'll keep in touch as to my progress.
#13 Updated by Dominic Giglio over 4 years ago
Just wanted to let you know that I'll be working from the GC tomorrow. I've got some local stuff going in a topic branch that I'll try and push up to our repo at some point today/tonight. If your schedule permits maybe we can discuss on the dev call or afterwards on Skype.
#15 Updated by Dominic Giglio over 4 years ago
Boone (and anyone else interested in this issue),
I've push the topic branch with the start to the Vagrant work. Here's a summary:
1.) base CentOS 6.5 64-bit box created from clean install. Currently hosted in my public Dropbox folder for now.
2.) basic Vagrantfile generated to use affore-mentioned box
3.) vagrant-config folder to store custom configurations and a basic Bash provisioning script
As I mentioned on the call I am currently working out networking and config files so I can keep adding more to the provisioning script. In its current state it's still pretty useless but I wanted to get you something to look at incase you've got feedback.
git checkout -b vagrant origin/vagrant
That's about all you can do right now. You can run
vagrant ssh after provisioning completes to ssh in. Here are the user details:
System user has passwordless sudo access and you can
su to the superuser using the password: vagrant
I rescheduled my GC development to tomorrow so I'll be there for 5-6 hours tomorrow working on this and any other issues I can get to.
#18 Updated by Dominic Giglio over 4 years ago
I've pushed up some new code to the vagrant topic branch
If you'd like to test ensure you've installed VirtualBox/Vagrant which I linked to in comment 15 above and then pull the latest code from the vagrant branch.
I was having trouble with the networking in the base Cent box I created so for now I am just using an official "clean and minimal" CentOS 6.5 64-bit from the official Vagrant box repositories. I don't think this is a problem for now since eventually we want to install the whole thing from scratch during provisioning anyways.
All you need to do is run
vagrant up from the root of our repo and it will go get the base box, launch it and then provision it. Right now the provisioning just installs the LAMP components and I've set Apache's DocumentRoot to point to the
index.php in the vagrant-config directory I've added to the repo. Once the box is up and running you can visit 192.168.167.9 in your browser and you should see output from
It's still very basic but coming along. I decided to stop here because it's time to configure MySql and that's going to be a decent hurdle to get over. If you'd like to set aside some time to go over the current state of the project let me know and we'll pick a time to chat. If you're not ready to test it yet, then I will update here once I've got new info regarding MySql. As soon as the DB is properly setup and installed it shouldn't be too hard to actually load our site; probably just a few config tweaks.