Welcome to the third part in my series, DevOps in a Regulated and Embedded Environment. In this part, we’re going to look at doing deployments in a restricted environment.
What exactly am I talking about? Remember that the target device was missing a whole lot of tooling but was ultimately rather complex. In particular, while we could communicate with the device using telnet and FTP, we didn’t have the luxury of easily scriptable services like ssh and rsync. The shell we were dropped into was bare sh without bash or anything more modern available. Moreover, we didn’t have a readily available build of python or ruby. So there was no access to the normally extremely useful frameworks I’d normally lean on like Chef or Ansible.
Yes, we could have tried to beat down that wall. We could have spent an unbounded amount of time trying to produce working builds of python and ssh. Perhaps we’d even have gotten lucky and done it in a relatively short amount of time, and then sorted out any dependency issues. However, we also might have spent weeks trying to get something working and wound up with nothing to show for my time. It’s important to note that part of the problem here was that the DevOps team (by which I mean all three of me, myself, and I) was entirely distinct from the development team. With support from the team, we may have been able to get builds of these tools produced relatively quickly. Moreover, the product team was understandably risk averse. The product is a medical device; it goes in somebody. Software that isn’t strictly necessary can’t wind up on the device because it increases risk. Whether this reasoning is correct or not is besides the point. It was one more hurdle that would have to be fought.
There were good reasons why things happened this way. For one, we didn’t yet have buy-in that the DevOps effort was worth the development cost. That’s why the initial team was so limited. I was conducting the pilot project to show that there was real value in a delivery pipeline for the organization. But it does highlight an important lesson: your development and DevOps teams shouldn’t be distinct.
So I needed another option. Luckily, a little Googling led me to a wonderful little tool called expect. For those not familiar with it, expect is a scripting language that’s used to interact with textual programs that are normally not amenable to scripting. For example, here’s an expect script that connects, over telnet, to a particular target, logs in with the provided user name when it’s requested, and then the password when it is requested, and finally returns the prompt to the user so that they can work on the target.
$ cat login.expect
#!/usr/bin/expect
set timeout 20
set addr [lindex $argv 0]
set user [lindex $argv 1]
set pass [lindex $argv 2]
spawn telnet $addr
expect "login:"
send "$user\r"
expect "Password:"
send "$pass\r"
expect "#"
interact
And of course, we could of course use the proc keyword to define this as a function instead of it’s own separate script:
proc login { addr user pass } {
spawn telnet $addr
expect {
timeout { send_user "Could not connect\n"; exit 1 }
eof { send_user "Connection refused\n"; exit 1 }
"login:"
}
send "$user\r"
expect "Password:"
send "$pass\r"
expect {
timeout { send_user "Failed to login.\n"; exit 1 }
"#"
}
}
Notice that we’re now using the expect block to handle conditional failure modes. Later, we can use this function to attempt to log into the target environment and be certain that anything that follows the call to login will only execute if we’ve logged in and have shell access. Instead of returning control to the user, we return it to whatever expect script called this login function.
We can even use similar logic to communicate with the FTP server with this same approach. That’s wonderful!
Separation of Concerns
Don’t see it? That’s okay, it took me a long time to see it too. For a long time, I considered these scripts to be fairly hacky. It’s only on review that I’m realizing how nice they are. What we’ve gained now is the ability to completely separate “talking to the target” from “doing stuff on the target”. I can use FTP to load scripts, deployment artifacts, etc. onto the target, and I can use telnet to run those scripts on those artifacts or things I know to be present in the environment. Notice that this is the first benefit you normally get from a deployment framework: all the stuff you have to do to get stuff onto the deployment target is totally separate from the framework-specific DSL scripts you actually execute on the deployment target to set it up and bring it up to date.
This separation of concerns is critical. It means I can worry about one thing at a time: how do I get the target environment into the right state to start a deployment, and only then worry about how do I do a deployment once it’s in that state. If I were trying to do this another way, I might accidentally mix those things up and end up with an unmaintainable mess. Instead, I get a nice, clean separation that allows for refactoring useful bits of code.
Yes, those deployment frameworks give us other benefits as well: most of what we do is guaranteed to be idempotent, and the DSL is much nicer to write than straight sh. But something is better than nothing. In fact, without idempotence, this separation is far more important. With idempotent resources like Chef’s package, we can guarantee that the version of a dependency we want is installed and at the right version before we ever attempt to deploy and run our application code. In the embedded environment, we don’t have that luxury. We don’t have any analogue for Chef’s package; we don’t have anything resembling packages, except maybe .tar.gz and we all know how useful that is.
So here’s a goal for the future: Let us create a shell based deployment framework with really minimal dependencies. Maybe something like that would actually be useful. I don’t know. I do know I’ll start building it if I ever end up needing to support an embedded environment like this again. Given time and resources, I’d probably start refactoring what I already have into one. It’s a difficult problem and it’s not clear that there’s a market that needs this. For one thing, the actual shell deployment functions would need to do a whole bunch of work to be idempotent. For another, it’s going to obnoxious to test properly. Even the company that is ultimately using all this work should be looking to put this stuff into maintenance-only mode while they refocus development onto a newer and more modern platform.
But who knows? Tell me what you think! Are you working on an embedded project and looking for a deployment or configuration management framework?
And join me next time for the fourth and final part of this series where we’ll look into the various scalability and resource concerns that very quickly pop up as you try to go from supporting a single branch and small team of developers to multiple streams of work and multiple teams.
