Inevitably on any agile project with a good degree of DevOps maturity, engineers start looking less at the application efficiencies and begin to look at addressing issues with their pipelines and frameworks to deliver faster or more effectively. At Coveros, we call this “DevOps’ing your DevOps.” It is a silly sentence but it speaks to the challenge of starting to automate development of the automation framework itself. I recommend a simple workflow for Jenkins, but still most of the time I come to Continuous Integration (CI) shops whose Jenkins’ pipelines are quite complicated. The more complicated the process, the more Jenkins code itself becomes fragile. On top of that, as the organization grows and becomes more reliant on Jenkins, the cost of your DevOps services being down or unstable is so large that we require the same type of performance and reliability testing (that we want for our apps) for Jenkins. We need to keep our CI pipeline stable, but still be able to make changes to it without fear of a major impact on development.
Testing Jenkins Jobs
One simple way to start testing our pipeline is by testing our Jenkins jobs before they go into our “Production Jenkins.” Jenkins jobs themselves are XML files contained in the directory tree $JENKINS_HOME/jobs. I back mine up every night to a Git repository; I’ve been doing this for a while for backups initially. Once the Git repository has been initialized, I just have a Jenkins job that. The job code:
cd $JENKINS_HOME/jobs git add . git add -u git commit -m "$COMMENT" git push
Git Tip: Use .netrc file for hands off authorization if you have to use http[s]
I put some thought into what I wouldn’t want to include in version control
# .gitignore file for Jenkins "jobs" directory # Only get the config.xml from any Jobs directory /*/builds /*/modules /*/htmlreports /*/last* /*/next* /*/*.csv /*/*.txt /*/*.log /*/workspace /*/modules # Process jobs that exist in folders **/jobs/*/builds **/jobs/*/modules **/jobs/*/htmlreports **/jobs/*/last* **/jobs/*/next* **/jobs/*/*.csv **/jobs/*/*.txt **/jobs/*/*.log **/jobs/*/workspace **/jobs/*/modules
So it is all in Git, where is the clever part?
If we make a testing branch for our “Test Jenkins”, we can do code reviews and merges/pull-requests to our production branch for our production Jenkins! This can really make your life far easier and you’ve managed to backup all your Jenkins jobs along the way!
Finally, to keep our Jenkins on the production server in sync with our git repo, we create a job that reset’s any manual changes (which are about to be wiped out), pulls, and reload the job configuration from disk to get the new change.
git reset --hard git pull
There is a web-ui button that does this, but for the advanced: you can use the jenkins cli to reload the jobs from disk
java -jar jenkins-cli.jar -s http://localhost/ reload
Until the monkey gets the banana,
Jonathan Malachowski
