I’m currently working on a DevOps project, heavily centered around AWS GovCloud. It’s important to point out I’m working in GovCloud, as opposed to AWS, as this means several key tools are missing. My colleague, Alan Crouch, recently pointed out how NAT Gateways are missing from the offered infrastructure. Another tool we found missing was Route 53, that lovely guy who does all of our DNS management. Without it, there are several things we found lacking, but the biggest one for me actually has to do with ease of use for Testers. Because we stand-up fresh application instances to perform all of our testing on (both automated and manual) and then tear them down when we are done. Unfortunately, we can’t provide static locations for our testers to point to. We resolved this by having Jenkins, our lovely build tool managing the creation of these instances (via Chef), send the tester an email with a link to the machine they requested, once it is done provisioning. Simply enough, right? Not quite, we still had a few requests from the QA team.
Because our application has a separate database (which we stood up using RDS) aside from the app server, they also need access to this virtual machine as well. The testers use SQL Workbench to verify data and connections, and they didn’t want to have to create a new connection each time a new version was deployed and tested, and this could be multiple a day. A few other requests/requirements we got them:
- Is it possible to use ‘non-clean’ databases?
- Is it possible to use a database from a prior deploy?
- Is it possible to setup static endpoints for mysql connections?
At face value, none of these seemed too challenging. The biggest issue being the static endpoints, seeing as no R53 exists in GovCloud. Our general naming convention for our deployed applications is as follows: [Application]-[Profile]-[Environment]-[Build Number]. i.e.
Application-ARD-DevInt-1.0.0.102 Application-FHB-Test-1.0.1.450 Application-GJN-QA-2.1.0.204
Our EC2 and RDS instances shared the same name (other than RDS replaced ‘.’ with ‘-‘ due to forced naming conventions). Initially, I tried to solve bullets one and two, allowing the Testers to simply deploy a new version of the application onto an existing app server. While this was technically simple (re-run our chef scripts with a newer application version passed in), our naming convention soon got all messed up. In order to stay consistent, I needed to update our EC2 machine name, our RDS machine name (which meant a new endpoint), AND tell Chef to update the node name on the server. This was quickly becoming a nightmare.
What I realized I needed was to combine all three goals into one over-arching solution. I realized that if instead of naming QA application instances after the version number, but instead on the tester performing the testing, I a) wouldn’t need to rename instances for upgrades, and b) would provide a static endpoint. Because these QA instances were only created by Jenkins when a tester requested one, I decided to use the Jenkins Build User Vars Plugin. This allowed me to inject in the user to the build variables. So my application naming convention ended up looking like the below:
machine_name="Application-${profile}-${environment}-${release}";
if [ "$environment" == "QA" ]; then
machine_name="Application-${profile}-${environment}-${BUILD_USER_ID}";
fi
This simple solution allowed me to hit all requests from our QA team and allowed for a simple ease of use. The main goal that I always push for is to ensure that our DevOps solution fits into the current development/testing process, and doesn’t create more work for anyone. Initially there may be some additional work to get up to speed/set things up, but ultimately if more work is created people won’t adopt these changes, and DevOps will eventually fail. This solution was able to hit exactly that point!
