TITLE: Barbarians at the Agile Gate AUTHOR: Eugene Wallingford DATE: June 15, 2011 7:53 AM DESC: ----- BODY: For the last year or so, there seems to have been something of a backlash against agile software development. A lot of conference speakers and bloggers have been telling us about the failures of the move to agile approaches, both the misconceptions at its base and the misdirections in its evolution. I have been especially uncomfortable watching writers and consultants who have teaching the world the ways of agile development join the bandwagon. Don't get me wrong. I am aware that agile development has its weaknesses; all things do. I'm also aware that the social and technical movement leading to its adoption across our industry has had its problems; all movements do. I do believe that we should understand the weaknesses in our practices and our methods for teaching the world about them, so that we can learn how to do those things better. Still, it's been a little disconcerting to watch the backlash proceed. Then again, in my short career, I've seen something similar happen to design patterns, object-oriented programming, and structured programming. It seems the natural order of history in our business. I've long wondered why. Then I came across The Return of the Barbarian, which tells the story of humans history as a cycle between barbarian culture and civilization. If we think of the software world in similar terms, our own history makes sense. Agile software development was once the barbarian. Now its the civilized culture, ripe to be conquered by a new, hungry barbarian. How so? Think back to before the days of agile. The software world, broadly speaking, had an entrenched culture that we all recognize. Developers and clients were different breeds. Developers and users, too. We documented stable requirements, designed up front a software system to deliver them, and passed the design onto programmers, wrote code. When the process didn't go as well as planned, programmers worked longer and harder to meet their deadlines. This was an idealized world, to be sure, but we strove hard to meet the ideal. People who had made this style of development work -- smart people, energetic people -- codified their knowledge in textbooks and training courses, so that the rest of us could learn how to duplicate their successes. Anyone willing to study could learn the techniques and become part of civilized society. Collectively, we were wise, but individually, we were weak, really, going through the motions. Along came agile. It turns out that there were world-class developers -- smart people, energetic people -- who did things differently. They worked closely with their clients and users. They collaborated heavily among themselves. They took small steps, refactored their code, and grew their software, rather than erecting it. To many in the industry, this was a romantic way to live and work, and they wanted to join in the fun. The dominant software culture didn't see it that way, though. Agile looked, well, barbaric. What happens when a cowboy programmer does something to break the system? How can we write programs if we don't know everything about what it should do? Where are the systematic controls on the process? Those entrenched in civilized software development culture saw agile methods as a step backward, to a less advanced time. But they were wrong, as we read in "The Return of the Barbarian":
The reason this seems like a strange phenomenon is that we confuse refinement with advancement. Finely-crafted jewelry is not more advanced than roughly-hewn jewelry. A Boeing 747 is about a million times more capable than the Wright Flyer I, but it does not contain a million times as much intelligence. It is merely more refined.... The difference between advancement and refinement is clearest in disruption. A beautifully-crafted sword is not more advanced than a crude gun. It is merely more refined.
The problem was, the software civilization built around structured programming was more than refined than the agile approaches, but not necessarily more advanced. The system itself was quite intelligent, with much wisdom and knowledge encoded in its practices, its textbooks, and its other literature. But individually, developers did not need to be as sharp, because the system guided them to success (as much as it could). The above passage is soon followed by a stark distinction:
The intelligence manifest in an artifact is simply the amount of human thought that has been externalized into it. Refinement on the other hand, is a measure of the amount of work that has gone into it. In Hegelian terms, intelligence in design is fundamentally a predatory quality put in by barbarian-Masters. Refinement in design is a non-predatory quality put in by civilized-Slaves.
It was in this context that agile -- a new barbarian culture -- swooped in and made inroads. The existing culture derided it as a fad, but it was in fact a set of advanced values, principles, and practices, less polished than the refined extant culture but full of deep thought and human experience. Over time, we saw the inevitable cultural evolution. In order to teach agile values, principles, and practices to a wider audience, barbarian-masters wrote down their wisdom in the form of books and conference talks and podcasts and index cards. In order to reach the managerial class of the corporations that build and buy software, first- and second- generation agilista created management seminars and certification programs and the trappings of institutional respectability. The practices were packaged, adjusted... refined. Soon everyone could "do" agile, or at least pretend to, by following the rules. Some folks got it, though not everyone, but most everyone tried to do it. The agile barbarian has become the civilized Organization Man. Where is the romance in that? Now new barbarians are at the gate, ready to shove the once-romantic revolutionaries of OO and patterns and agile development out into the streets. It is the natural order. Here's the thing. Agile approaches still work as well as they did back when they were the knowledge in the minds of great programmers, who were unhindered by the rules that they follow instinctively and break when necessary in the pursuit of a great program. That knowledge was born out of experience and embodied wisdom gained from living in the civilization that came before. If we want to become like the agile masters, we need to do more than imitate them. We need to live the values, principles, and practices -- to make them our own, not something written in a book or a blog. Perhaps this is just a fanciful retelling of our history to explain away the agile movement's shortcomings and failures. But I think it carries a grain of truth. A new barbarian knocks down the door as soon as we become too civilized, when we convert our deep, compiled, pragmatic, contextual knowledge into handbooks and notecards and consultants' packaged talks and on-site training courses. To the extent that my tale is true, I think it offers us a path forward: a reminder to embrace our barbarian past. Each of us can develop his or her own individual intelligence, rather than relying on the external trappings of a civilized movement. Don't let the next revolution cause you to abandon what works well. Look for useful knowledge and practice in the ways of the next wave of barbarians, and use it to get better. Instead of arguing with others about test-first design, pair programming and refactoring, about collaboration with users, collective code ownership, and sustainable pace, we should simply do them, in the way that fits best our context. This leads by example. It may not scale as fast as the civilized approach, but as we have seen, that approach is fraught with its own dangers. Besides, the goal isn't to scale or to create a movement. It's to write great programs. -----