TITLE: The Value of Standard Form in Evaluating Technical Papers
AUTHOR: Eugene Wallingford
DATE: May 01, 2005  7:27 PM
DESC: Freedom of form in writing papers is no gift for authors who are trying to communicate their ideas to a community of researchers, but it also createsproblems for the reviewers charegd with determining the value of papers.
-----
BODY:
Over lunch today, Robert Biddle and I discussed some
thoughts we had after reviewing submissions to the
OOPSLA
<A HREF="http://www.oopsla.org/2005/ShowPage.do?id=CallEducatorsSymposium">
   Educators Symposium</A>
and practitioners reports track this year.  Both of
these tracks suffer from a problem that had never
really occurred to me before this year: not having
an accepted form for telling stories.

If you read the proceedings of most computer science
conferences, you will recognize that all the papers
have a similar look and feel.  This is the way that
scientists in the domain communicate with one another.
One function of a common form is to ensure an efficient
exchange of information; having a common style means
that readers immediately feel at home when they come
to a paper.  But there is a subtle secondary function,
too.  When you see a paper, you can tell whether the
author is a part of the community or not.

One of the the things that makes reviewing papers for
the Educators Symposium tough is the variety of papers.
With no standard, authors are left to mimic other
papers or invent new forms that fit the story they
want to tell.  But every new form makes the program
committee's job harder, too.  How do we evaluate the
contribution of a paper that looks and sounds different
than any we've seen?  What role should experimental
validation of a teaching technique play?  What role
should an explanation of lessons learned, or a
discussion of how to implement the technique at a
different institution?

Most frustrating for me as program chair were situations
in which two reviewers evaluated the same paper in
essentially complementary ways.  This isn't a fault of
the reviewers, because the community as a whole has not
reached a consensus for what papers should be like.  I
suppose that one of my jobs as program chair is to guide
this process closely, working with the program committee
to give authors and reviewers alike a better sense of
what we are looking for in submissions.  The nifty
assignment track that I introduced to the symposium last
year was an attempt in this direction.  I borrowed from
a 
<A HREF="http://nifty.stanford.edu/">
   successful form</A>
used at recent SIGCSEs to encourage OO educators to
tell the stories of their coolest and most engaging
programming assignments.  Having an expectation of what
a nifty assignment should look like has led, I think,
to a more satisfying evaluation process for these
submissions.

Maybe we as a community of educators need to work
together to develop a standard way to tell teaching
stories.  Perhaps the greatest contribution of the
software patterns community was to standardize the
way practitioners and academics discuss the elements
of program design and construction at a level above
algorithms and data structures.  As linguistic form,
it enables communication in a way that was heretofore
impossible.

Notice here that a common literary <EM>form</EM> is
more than a literary format.  The patterns community
is a good example of this.  Even after a decade there
are a number of accepted formats for writing patterns,
some favored by one group of writers, some viewed as
especially effective for patterns of a particular sort.
The key to the pattern community's contribution is
that it establishes expectations for the
<STRONG>content</STRONG> of software patterns.  I can
comfortably read a pattern in almost any format, but
if I can't find the context, the problem, the forces,
and the solution, then I know the pattern has problems.
The PLoP conferences play the role of enculturating
pattern writers by helping them to learn the standard
form and how to use it.

The lack of a standard form is especially acute in the
area of practitioners reports, which have the potential
to be one most important contributions OOPSLA makes in
the software world.  I'll have more to say about this
tomorrow.
-----