Hotfixes within DevOps Pipelines

I recently wrote several blog posts about setting up a DevOps pipeline, and it was working great for our code. However, recently, I ran into an issue. My perfectly written and tested code somehow introduced a bug into our production environment! Luckily, we caught this issue quickly, and it was a relatively easy fix. So, I fixed the code, but the problem then became getting this code into production. I had no simple way to get the code into production in a quick fashion, and I realized I needed to create a hotfix pipeline, separate from my main DevOps pipeline.

First thing I needed to do was update the build job to tag each of our builds. I used the Jenkins git plugin, as this made it easier to tie the builds to our git repos – more traceability is never a bad thing. A added a step in the Jenkins build job to tag the code in the develop branch with the Jenkins build number. The only other change that I wanted to make was to provide an additional tag to our git code when we deployed to production. The master branch (remember, our develop code was merged into master) was checked out to the build tag when deploying to production, and after a success, an additional tag was applied indicating a successful production deploy, with that same build number.

This was important to do, because when we needed to initiate our hotfix, we don’t want to use our develop branch. There is potentially code in the develop branch that isn’t tested or ready to go into production quite yet. So, instead of branching hotfix off of develop, we can simply branch it off of master, where the tag matches our latest production deploy

Next thing I did was copy the main build job (the one I labeled Developer Tests), but had it build over that new hotfix branch, instead of the develop branch. The entire ‘CI’ process remained the same, running our smoke and acceptance tests, simply over the fixed code. This code was then packaged, and pushed into Nexus.

Instead of our old QA deploys, a QA environment is immediately spun up, with an email sent out to the team, notifying them that the environment is up, and that it is ready for some manual testing. Instead of running all of the regression and in depth security tests, we want some manual testing to at least spot check out application. This is all in the interest of cutting some of our testing time short, to fast track out application to production.

While the QA job does the normal deploy, the Jenkins promotion plugin was setup to manage everything downstream. Two triggers were setup: promotion and rejection

The Promotion trigger destroys the QA environment, and then triggers the deploy to production job. We also merge hotfix back into master

The Rejection trigger simply destroys the QA environment.

That is it. The one last manual step left, is to manually merge master back into develop in order to capture that hotfix for future development work. This was purposefully set as a manual job instead of part of the automated process, due to concerns with merge conflicts.