TITLE: OOPSLA Day 2: Jim Waldo "On System Design" AUTHOR: Eugene Wallingford DATE: October 26, 2006 4:28 PM DESC: ----- BODY: Last year, OOPSLA introduced another new track called Essays. This track shared a motivation from another recent introduction, Onward!, in providing an avenue for people to write and present important ideas that cannot find a home in the research-oriented technical program of the usual academic conference. But not all advances are in the form of novelty, of results from narrowly-defined scientific experiments. Some empirical results are the fruit of experience, reflection, and writing. The Essays track offers an avenue for sharing this sort of learning. The author writes an essay in the spirit of Montaigne and Bacon, using the writing to work out an understanding of his experience. He then presents the essay to the OOPSLA audience, followed by a thoughtful response from a knowledge person who has read the essay and thought about the ideas. (Essays crossed my path in a different arena last week, when I blogged a bit on the idea of blog as essay.) Jim Waldo, a distinguished engineer at Sun, presented the first essay of OOPSLA 2006, titled "On System Design". He reflected on his many years as a software developer and lead, trying to get a handle on what he now believes about the design of software. What is a "system"? To Waldo, it is not "just a program" in the sense he thinks meant by the postmodern programming crowd, but a collection of programs. It exists at many levels of scale that must be created and related; hence the need for us to define and manage abstractions. Software design is a craft, in the classical sense. Many of Waldo's engineer friends are appalled at what software engineers think of as engineering (essentially the application of patterns to roll out product in a reliable, replicable way) because what engineers really do involves a lot of intuition and craft. Software design is about technique, not facts. Learning technique takes time, because it requires trial and error and criticism. It requires patience -- and faith. Waldo paraphrase Grady Booch as having said that the best benefit of the Rational toolset was that it gives developers "a way to look like they were doing something while they had time to think". The traditional way to learn system design is via apprenticeship. Waldo usually asks designers he respects who they apprenticed with. At first he feared that at least a few would look at him oddly, not understanding the question. But he was surprised to find that every person answered without batting an eye. They all not only understood the question but had an immediate answer. He was also surprised to hear the same few names over and over. This may reflect Waldo moving in particular circles, or only a small set of master software developers out there! In recent years, Waldo has despaired of the lack of time, patience, and faith shown in industry for developing developers. Is all lost? No. In reflecting on this topic and discussing with readers of his early drafts, Waldo sees hope in two parts of the software world: open source and extreme programming. Consider open source. It has a built-in meritocracy, with masters at the top of the pyramid, controlling the growth and design of their systems. New developers learn from example -- the full source of the system being built. Developers face real criticism and have the time and opportunity to learn and improve. Consider extreme programming. Waldo is not a fan of the agile approaches and doesn't think that the features on which they are sold are where they offer most. It isn't the illusion of short-term cycles or of the incrementalism that grows a big ball of mud which give him hope. In reality, the agile approaches are based in a communal process that builds systems over time, giving people the time to think and share, mentor and learn. Criticism is built into the process. The system is an open, growing example. Waldo concludes that we can't teach system design in a class. As an adjunct professor, he believes that system design skills aren't a curriculum component but a curricular outcome. Brian Marick, the discussant on Waldo's took a cynical turn: No one should be allowed to teach students system design if they haven't been a committer to a large open-source project. (Presumably, having had experience building "real" big systems in another context would suffice.) More seriously, Marick suggested that it is only for recent historical reasons that we would turn to academia to solve the problem of producing software designers. I've long been a proponent of apprenticeship as a way of learning to program, but Waldo is right that doing this as a part of the typical university structure is hard, if not impossible. We heard about a short-lived attempt to do this at last year's OOPSLA, but a lot of work remains. Perhaps if more people like Waldo, not just the more provocative folks at OOPSLA, start talking openly about this we might be able to make some progress. Bonus reading reference: Waldo is trained more broadly as a philosopher, and made perhaps a surprising recommendation for a great document on the act of designing a new sort of large system: The Federalist Papers. This recommendation is a beautiful example of the value in a broad education. The Federalist Papers are often taught in American political science courses, but from a different perspective. A computer scientist or other thinker about the design of systems can open a whole new vista on this sort of document. Here's a neat idea: a system design course team taught by a computer scientist and a political scientist, with The Federalist Papers as a major reading! Now, how to make system design as a skill an inextricable element of all our courses, so that an outcome of our major is that students know the technique? ("Know how", not "know that".) -----