TITLE: Software Case Studies AUTHOR: Eugene Wallingford DATE: May 05, 2005 11:16 AM DESC: Like patterns, the software case study could become an extremely valuable genre of the literature of computing. ----- BODY: (I wrote a complete essay on this topic over the course of several hours on Monday and then, in a few swift seconds of haste, I did the unthinkable: I rmed it. Sigh. I don't know if I can reproduce that wondrous work of art, but I still have something I want to say, so here goes.) Last time, I wrote about how having a standard literary form can ease the task faced by writers and readers, including reviewers. I found this to be true of submissions to the OOPSLA Educators Symposium, and my colleague Robert Biddle found it to be true of submissions to the conference's practitioner reports track. From this starting point, Robert and I moved onto an ambitious idea. I closed my last entry with a cliffhanger: The unsupported claim that "the lack of a standard form is especially acute in the area of practitioner reports, which have the potential to be one most important contributions OOPSLA makes in the software world." This time, I'd like to talk about both parts of this claim. First, why is the lack of a standard literary form especially acute in the area of practitioner reports? Keep in mind that the authors of practitioner reports are software practitioners, folks in the trenches solving real problems. Most of these folks are not in the habit of writing expository prose to teach others or to share experiences -- their jobs are primarily to create software. Unlike most submitters to the Educators Symposium, the authors of practitioner reports are usually not academics, who typically have experience with expository writing and and can at least fall back on the literary forms they learned in publishing academic papers. (Sadly, those forms tend not to work all that well for sharing pedagogical experience.) Not having a standard literary form makes writing and reading practitioner reports that much harder. How can authors best communicate what they learned while building a software project? What ideas do readers expect to find in the paper? What's worse, because many software developers don't have much experience writing for a broad audience, not knowing how to go about writing a paper can create a considerable amount of fear -- and the result is that many practitioners won't even try to write a paper in the first place. It turns out that a standard literary form has another benefit: it provides comfort to potential authors, lowering the entry barrier to new writers. A great example of this effect is the software patterns community. Its popularization of a common and powerful literary form made it both possible and attractive to many practicing software developers to record and disseminate what they had learned writing programs. The software community as a whole owes an extraordinary debt for this contribution. So, I contend that the practitioner's track would benefit even more than the educators' track from the creation and widespread adoption of a standard literary form. (But I still hope that we educators will take steps to improve our lot in this regard.) Second, how do practitioner reports have the potential to be one most important contributions that OOPSLA or any other conference makes in the software world? Keep in mind that the authors of practitioner reports are software practitioners, folks in the trenches solving real problems. Researchers and methodologists can propose ideas that sound great in theory, but practitioners find out how well they work in the real world, where academic abstractions take a back seat to the messiness of real businesses and real people and real hardware. Even when researchers and methodologists make a good-faith effort to vette their ideas outside of their labs, it is difficult to recreate all the complexities that can arise when new adopters try implement the ideas in their organizations. The result is, we don't really know how well ideas work until they have been tried in the trenches. And practitioner reports can tell us. Sharing knowledge of a practical sort used to lie outside the domain of computer science and even software engineering, but the software patterns movement showed us that it could be done, that it should be done, and gave us some hints for how to do it. The potential benefit to practitioners is immeasurable. Before trying out XP, or migrating from Visual Basic to VB.Net, or integrating automated acceptance tests into the build cycle, a practitioner can read about what happened when other folks in the trenches tried it out. Usually, we expect that the ideas worked out okay in practice, but a good report can point out potential pitfalls in implementation and describe opportunities to streamline the process. Of course, academics can benefit from good practitioner reports, too, because they close the loop between theory and practice and point out new questions to be answered and new opportunities to exploit. Robert and I didn't talk much about what I've written so far in this entry, because we rather quickly moved on to Robert's bigger vision of what practitioner reports can be, one that presupposes the untapped value buried in this resource: the software case study. As Robert pitched it, consider NCSA Mosaic. Here is a program that changed the world. How was it built? What technical and non-technical problems arose while it was being written, and how did the development team solve them? Did serendipity ever strike, and how did they gain leverage? We can find the answers to all of these questions and more -- the creators are still around to tell the story! Case studies are a standard part of many disciplines. In business schools, students learn the how companies work by reading case studies. I remember well a management course I took as an undergraduate in which we studied the development of particular companies and industries through case studies. (Most of what I know about the soft drink industry came from that case book!) Of course, the law itself is structured around cases and webs of facts and relationships. We are fortunate to have some very nice case studies in computing. Knuth has written widely about his own programs such as Tex and his literate programming tools. Because Smalltalk is as much system as language, Alan Kay's The Early History of Smalltalk from History of Programming Languages II qualifies as a software case study. Other papers in the HOPL volumes probably do, too. I have read some good papers on Unix that qualify as case studies, too. Two of my favorite textbooks, Peter Norvig's Paradigms of AI Programming and Clancy and Linn's Designing Pascal Solutions, are built around sets of case studies. In the latter, the cases were constructed for the book; in the former, Norvig analyzes programs from the history of AI and reimplements them in Common Lisp. (You really *must* study this book.) I even used a case study book as a CS undergrad: Case Studies in Business Data Bases, by James Bradley. It's still on my bookshelf. More recently, the College Board's Marine Biology Case Study has received considerable attention in the AP and CS1 communities. So, if we believe that software case studies have merit, we find ourselves back in the trenches... How do we write them? Using an excellent case study as an exemplar would be a start. I suggest that any case study of value probably must tell us at least three things:
  1. In what context did we operate?
  2. What did we do?
  3. What did we learn?
These elements apply to case studies of whole systems as well as to case studies of incremental changes to systems or process improvements. They are almost certainly part of the better practitioner reports presented at OOPSLA. Robert and I will likely work on this idea further. If you have any ideas, please share them. In particular, I am interested in hearing about existing and potential software case studies. Which of the case studies you've read do you think are the best exemplars of the genre? Which programs would you like to see written up in case studies? -----