I’ve worked on a variety of continuous integration and continuous delivery projects. We frequently use Jenkins as our platform for driving all build, provisioning, and deployment activities. As part of this, we have a lot of fairly complicated jobs in Jenkins that we are constantly modifying. As a version control pundit, one of my goals is to “version control everything.” This includes, of course, our Jenkins configuration. By version controlling Jenkins, we have a change history and safety net to fall back on if we break something. Using some slightly more advanced techniques, it also allow us to maintain various Jenkins servers on different branches for different groups of people.
In my situation, we’re using Git. The same could be done with Subversion or any other VCS with very little difference. We’re also running our server on Windows, but again the techniques apply almost directly to any Unix-based environment. I’ll assume you know the basics of Git and Jenkins for the rest of the discussion.
Intial Setup
Version controlling Jenkins is actually fairly simple. It involves maintaining the various config.xml files, the plugin.jpi files, and various other XML files that maintain overall configuration of the various aspects of Jenkins. There are some files and patterns, however, that you generally don’t want to version control. We use a .gitignore file to do that.
To get started, go to your JENKINS_HOME directory. This might be ~/.jenkins in default configurations. In our case, lets assume it’s /tomcat/jenkins. Create a .gitignore file with the following content.
| .m2/ .ivy2/ .sbt/ .ssh/ .emacs.d/ logs/ nodes/ .gitconfig .viminfo .bash_history log *.log *.tmp *.old *.bak *.jar *.swp .owner terracotta/ updates/ fingerprints/ config-history/ # Only get the config.xml from any Jobs directory jobs/*/builds jobs/*/modules jobs/*/htmlreports jobs/*/last* jobs/*/next* jobs/*/*.csv jobs/*/*.txt jobs/*/*.log jobs/*/workspace jobs/*/modules # Process jobs that exist in folders jobs/**/jobs/*/builds jobs/**/jobs/*/modules jobs/**/jobs/*/htmlreports jobs/**/jobs/*/last* jobs/**/jobs/*/next* jobs/**/jobs/*/*.csv jobs/**/jobs/*/*.txt jobs/**/jobs/*/*.log jobs/**/jobs/*/workspace jobs/**/jobs/*/modules # More files war workspace secrets users secret.key* identity.key plugins/*/ # Contains local info and secrets; don’t want this in Git jenkins.model.JenkinsLocationConfiguration.xml jenkins.mvn.GlobalMavenConfig.xml credentials.xml hudson.scm.CVSSCM.xml
|
|
Tuning the .gitignore file takes a bit of trial and error. Basically, you need to keep an eye on files that appear when you look at ‘git status’ for the files in your hierarchy of JENKINS_HOME.
A lot of these are JPI plugin files that tend to be packaged with Jenkins. You could version-control these, but we chose not to in our case. The JenkinsLocationConfiguration file is interesting because it can be edited on different servers to provide the Jenkins URL and admin email addresses in a way that won’t conflict every time you edit it on different servers.
Next, run git init to initialize a local git repository in your current directory.
> git init
Initialized empty Git repository in /tomcat/jenkins/.git
Now, start adding your files to the repository. I start with the important ones first:
> git add config.xml
> git add jobs/*/config.xml
> git commit
That grabs the critical files. Frankly, if this was ALL you did, it would provide a good change history of what you do with jobs, but would skip a lot of the actual Jenkins configuration.
Next, you get a little bold and add everything else that isn’t ignored.
> git add .
> git status
This should give you a reasonably long list of lots of JPI and XML files in the current top level directory. Review it and commit the files as long as it didn’t do something completely bizarre in your environment.
The next step is to push your local Git repository up to a remote location so you can share it and maintain backup copies, but I’ll leave that as an Git exercise for the reader.
Recurring Workflow
Now that you’ve setup Git for Jenkins, you’ll want to begin committing and synchronizing changes on a regular basis. If you are in a very localized environment and only have a single server, you just need to make sure you commit the changes. In our case, we started by just logging onto the server every so often and running a manual commit.
> cd /tomcat/jenkins
> git status
> git add .
> git commit
> git log -5 –pretty=oneline –stat
The trickier part comes if you are running multiple Jenkins servers and need to share your changes in multiple environments. In that case, you’ll need to pull and push changes upstream, then do similar things on any other server you might be using. One note: if you pull changes down to a local Jenkins server, you will likely have to restart Jenkins to get it to recognize the changes to the local config files. In my case, I find it easiest to just restart Tomcat.
After a while, I got sick of remote logging into the server all the time, so I created a simple Job in Jenkins to do the commit for me. Specifically, I created 3 jobs:
- GitStatus – run Git status in the .jenkins directory to see what has changed. I even tacked on ‘git diff’ to the end of the job so I could see the specifics.
- GitCommit – run the commits. I take a “COMMENT” parameter for the job so that I can record a message about what I changed.
- GitSync – pull/push changes from the upstream server.
This allowed me to use the browser to “Build Now” or “Build with Parameters” whenever I wanted to see and/or commit my changes. Slightly easier than having to remote into the server and run from the command line. Eventually, we got really brave and set Jenkins up to do this automatically. Once per hour, it would run the sync/commit jobs to make sure our changes are always automatically captured. The good part of this is that you’ll always have a version control path of your code. The down side is that you start losing context because there are no human-authored commit message about what changed and why. I’ll leave the details of the jobs to another post.
