TITLE: The Value of Standard Form in Evaluating Technical Papers AUTHOR: Eugene Wallingford DATE: May 01, 2005 7:27 PM DESC: Freedom of form in writing papers is no gift for authors who are trying to communicate their ideas to a community of researchers, but it also createsproblems for the reviewers charegd with determining the value of papers. ----- BODY: Over lunch today, Robert Biddle and I discussed some thoughts we had after reviewing submissions to the OOPSLA Educators Symposium and practitioners reports track this year. Both of these tracks suffer from a problem that had never really occurred to me before this year: not having an accepted form for telling stories. If you read the proceedings of most computer science conferences, you will recognize that all the papers have a similar look and feel. This is the way that scientists in the domain communicate with one another. One function of a common form is to ensure an efficient exchange of information; having a common style means that readers immediately feel at home when they come to a paper. But there is a subtle secondary function, too. When you see a paper, you can tell whether the author is a part of the community or not. One of the the things that makes reviewing papers for the Educators Symposium tough is the variety of papers. With no standard, authors are left to mimic other papers or invent new forms that fit the story they want to tell. But every new form makes the program committee's job harder, too. How do we evaluate the contribution of a paper that looks and sounds different than any we've seen? What role should experimental validation of a teaching technique play? What role should an explanation of lessons learned, or a discussion of how to implement the technique at a different institution? Most frustrating for me as program chair were situations in which two reviewers evaluated the same paper in essentially complementary ways. This isn't a fault of the reviewers, because the community as a whole has not reached a consensus for what papers should be like. I suppose that one of my jobs as program chair is to guide this process closely, working with the program committee to give authors and reviewers alike a better sense of what we are looking for in submissions. The nifty assignment track that I introduced to the symposium last year was an attempt in this direction. I borrowed from a successful form used at recent SIGCSEs to encourage OO educators to tell the stories of their coolest and most engaging programming assignments. Having an expectation of what a nifty assignment should look like has led, I think, to a more satisfying evaluation process for these submissions. Maybe we as a community of educators need to work together to develop a standard way to tell teaching stories. Perhaps the greatest contribution of the software patterns community was to standardize the way practitioners and academics discuss the elements of program design and construction at a level above algorithms and data structures. As linguistic form, it enables communication in a way that was heretofore impossible. Notice here that a common literary form is more than a literary format. The patterns community is a good example of this. Even after a decade there are a number of accepted formats for writing patterns, some favored by one group of writers, some viewed as especially effective for patterns of a particular sort. The key to the pattern community's contribution is that it establishes expectations for the content of software patterns. I can comfortably read a pattern in almost any format, but if I can't find the context, the problem, the forces, and the solution, then I know the pattern has problems. The PLoP conferences play the role of enculturating pattern writers by helping them to learn the standard form and how to use it. The lack of a standard form is especially acute in the area of practitioners reports, which have the potential to be one most important contributions OOPSLA makes in the software world. I'll have more to say about this tomorrow. -----