I came into the software industry in 1994, having just graduated from George Mason University. One of my classes was a “Software Engineering” class that taught us about the Software Development Life Cycle (SDLC), which was a nice way to say waterfall. It supposed that SDLC was the pinnacle of software development methodologies because it was what we had. What I didn’t know was that the fight over middleweight methodologies like the Rational Unified Process (RUP) was at a fever pitch, with RUP getting ready to emerge as the victor. At the time lightweight methodologies were just beginning and included things like Spiral, Rapid Prototyping, and Rapid Application Development (RAD).
Back then, software projects had a bad reputation of taking a very long time, usually delivering late and over budget. Failed projects were at least as common a delivered projects. Many delivered projects were disappointing, and a good number quickly became shelfware.
Each phase in SDLC was considered its own thing and one phase could be successful on its own even if subsequent phases were failures. I worked with people that specialized in writing requirements and specifications. They were often praised for their work in getting that phase done in an efficient manner. Their reputation as miracle workers was never questioned if a later phase was a complete failure and resulted in a canceled project. Given the incentive structure in place and the lack of control or even insight into later phases, I am not sure if what happened after they left a project was reasonable to hold against them.
The Spiral Model
Spiral, Rapid Prototyping, and RAD had recently been introduced. Spiral had the insight that software had to be built with feedback as it was being built. A visual representation of the development process was a spiral where along certain points of the software would be released with each large spiral arc representing more and more functionality in the software. It was an iterative approach that used feedback from each release to help guide future work.
Rapid Prototyping
Rapid prototyping was what it sounds like, quickly creating a prototype that could be used to collect data and get feedback. Sometimes that prototype was used to iron out the details of a final product. The final product was sometimes a rewrite of the prototype. Sometimes the prototype was put into production. Sometimes a new codebase was created to replace the prototype in production, Sometimes that prototype was enhanced and remained in production. Sometimes the prototype was declared good enough and was left in production.
RAD—Rapid Application Development
RAD, as it was explained to me, was based on the idea that most enterprise software had a short useable lifetime, maybe only a few years. That means that it doesn’t make sense to spend years creating it. The goal was to spend as little time as possible, like 6 weeks to 3 months, using a 4GL (Fourth Generation Language) like Visual Basic to slap something together.
If it had bugs, that is fine, all software had bugs. If there was a part of the requirements it didn’t do but could be done anyway, that is fine, that happens all of the time. Given that most of the software was used by small teams of people, it didn’t need to be the most efficient. Since most software lived on a LAN without Internet access, it didn’t need to be secure. Sure some of those assumptions would be wrong for some applications but the risks were considered small enough to not matter. Maybe we would take that into consideration in a few years when the software would be re-created again.
This was the world Agile was brought into. You can see that Agile was created to enhance the things that worked and counter the things that didn’t work.
Extreme Programming
Extreme Programming was the first Agile methodology that I was taught. As it was explained to me the goal was to do all of the things that developers thought were good to do—and do them to the extreme. Design and code reviews are powerful tools to improve the quality of the codebase. Most programmers have experienced the value of getting someone else to look over your code when there is a mistake in it you can’t figure out, usually because you are so close to the code you can’t see the problem. Pair and mob programming are getting two or more people working together to create code so that there is constant design and code review. You also have someone that can help you think through your design and coding issues.
From Spiral, we got feedback loops that help us triangulate into creating software that is best fit for purpose. To me, agile methodologies were the first to embrace the idea that we don’t really know what the software should do or look like until we have something we can see and use.
Agile embraces the idea that change is a part, not only of software development, but of life.
Other methodologies put barriers up to changing the software. Other methodologies didn’t accept the fact that the needs of the users or marketplace the software is created to serve, often changes. If you take two years or more to create a new application, there is a good chance that enough of the requirements that were captured years ago are now work, rendering the software suboptimal or worse unusable. Agile embraces the idea that change is a part, not only of software development, but of life. Agile uses test automation to help keep the software flexible and open to change while making sure we aren’t breaking that things need to continue to work.
RAD helped us see that delivering working software quickly has business value. Agile takes it a step further and has us deliver working software that is a better fit for purpose and with more functionality on a regular basis. Unlike RAD, Agile uses what we learn from customer feedback right away to make the software better within days, weeks, or at most, months.
Agile takes the idea of building software quickly like Rapid Prototyping but emphasizes production-ready code that can be put into production on a regular basis with practices that give us the confidence that it will not fall over when people start using it.
To me, Agile was the first software development methodology that accepted the real-world concerns of change, user feedback, and the need to have working code in production as guiding principles. It acknowledges that software development is hard and that our first idea of how the software should work is often wrong and needs to be refined as we go. Agile embraces informed change by using the information we gain from feedback to course-correct as we go. Agile thinking also accepts that we know the least about a software project at the beginning so that creating all of the requirements and specifications and freezing them is basing important decision from a place of ignorance.
[Agile] acknowledges that software development is hard and that our first idea of how the software should work is often wrong and needs to be refined as we go.
Successful software is software that delights the users of that software. Agile gives you the best possible chance of creating software that delights your users by allowing the teams to learn as much as possible before committing to a specific detail of the software.
So where did Agile come from? Like much of the software industry, Agile came from the knowledge we have gained from a series of past attempts to make things better. Some of there were successful, some were failures. All of them have helped us see the possibility of a better way to create software. I don’t think we are finished with that journey. I do think we are on the right path.
One thought to “Where did Agile come from?”
I started my software career in 1972, even then, the ideas that (finally) led to Agile can be seen in the of Dr. W. Edwards Deming (https://en.wikipedia.org/wiki/W._Edwards_Deming) with, (or example, Toyota post WWII which led to the emergence of Lean in the 1960s. Lean hit the US in the 80s, but mainly in manufacturing. It’s ideas and Deming’s writings (e.g., Out of the Crisis), however, can be seen in the Agile Manifesto’s Values and Principles and, before that, in the quality movement represented by the work of Juran, Crosby, and others (not to mention the many Japanese quality “gurus” preceding them).