TITLE: Shocking News: Design Technique Not Helpful When Design Is Already Done
AUTHOR: Eugene Wallingford
DATE: October 13, 2016 4:29 PM
Thanks to Greg Wilson for a pointer to
which reports the result of an empirical evaluation of the effects
of test-driven development on software quality and programmer
a blog entry about the paper,
This painstaking study is the latest in a long line to find
that test-driven development (TDD) has little or no impact
on development time or code quality.
He is surprised, because he feels more productive when he writes
I'd like to be able to say that these researchers and
others must be measuring the wrong things, or measuring
things the wrong way, but after so many years and so
many different studies, those of us who believe might
just have to accept that our self-assessment is wrong.
Never fear! Avdi Grimm
points the way
toward resolution. Section 3.3 of the research paper describes
the task that was given to the programmers in the experiment:
The task was divided into 13 user stories of incremental
difficulty, each building on the results of the previous
one. An example, in terms of input and expected output,
accompanied the description of each user story. An
Eclipse project, containing a stub of the expected API
signature, (51 Java SLOC); also an example JUnit test
(9 Java SLOC) was provided together with the task
Notice what this says:
The paper reports that previous studies of this sort have also
set tasks before programmers with similar initial conditions.
Grimm identifies a key feature of all these experiments: The
problem given to the test subjects has already been defined in
Shocking news: test-driven design is not helpful when the
design is already done!
As Grimm says, TDD helps the programmer think about how to start
writing code and when to stop. For me, the most greatest value
in doing TDD is that it helps me think about what I need to
build, and why. That is design. If the problem is already
defined to the point of implementation, most of the essential
design thinking has been done. In that context, the results
reported in this research paper are thoroughly unsurprising.
Like Grimm, I sympathize with the challenge of doing empirical
research on the practices of programmers. I am glad that people
such as the paper's authors are doing such research and that
people such as Wilson discuss the results. But when we wonder
why some research results conflict with the personal assessments
of real programmers and our assessments of our own experiences,
I think I know what one of the problems might be:
Evaluating the efficacy of a design methodology properly requires
that we observe people doing, um, design.
- Test subjects were given the user stories.
- The stories were sequenced to assist the subjects.
- Each story was accompanied by an example.
- Subjects were given code stubs with method signatures