TITLE: Shocking News: Design Technique Not Helpful When Design Is Already Done AUTHOR: Eugene Wallingford DATE: October 13, 2016 4:29 PM DESC: ----- BODY: Thanks to Greg Wilson for a pointer to this paper, which reports the result of an empirical evaluation of the effects of test-driven development on software quality and programmer productivity. In a blog entry about the paper, Wilson writes:
This painstaking study is the latest in a long line to find that test-driven development (TDD) has little or no impact on development time or code quality.
He is surprised, because he feels more productive when he writes tests up-front.
I'd like to be able to say that these researchers and others must be measuring the wrong things, or measuring things the wrong way, but after so many years and so many different studies, those of us who believe might just have to accept that our self-assessment is wrong.
Never fear! Avdi Grimm points the way toward resolution. Section 3.3 of the research paper describes the task that was given to the programmers in the experiment:
The task was divided into 13 user stories of incremental difficulty, each building on the results of the previous one. An example, in terms of input and expected output, accompanied the description of each user story. An Eclipse project, containing a stub of the expected API signature, (51 Java SLOC); also an example JUnit test (9 Java SLOC) was provided together with the task description.
Notice what this says: The paper reports that previous studies of this sort have also set tasks before programmers with similar initial conditions. Grimm identifies a key feature of all these experiments: The problem given to the test subjects has already been defined in great detail. Shocking news: test-driven design is not helpful when the design is already done! As Grimm says, TDD helps the programmer think about how to start writing code and when to stop. For me, the most greatest value in doing TDD is that it helps me think about what I need to build, and why. That is design. If the problem is already defined to the point of implementation, most of the essential design thinking has been done. In that context, the results reported in this research paper are thoroughly unsurprising. Like Grimm, I sympathize with the challenge of doing empirical research on the practices of programmers. I am glad that people such as the paper's authors are doing such research and that people such as Wilson discuss the results. But when we wonder why some research results conflict with the personal assessments of real programmers and our assessments of our own experiences, I think I know what one of the problems might be: Evaluating the efficacy of a design methodology properly requires that we observe people doing, um, design. -----