Adv. OS and Distributed Systems (15-172)
What makes a good systems paper
(summary for Monday, Oct. 15)
by Vahe Poladian


What makes a good systems paper?

In this summary I describe what good systems papers have 
in common and exactly what features make them good.

In general, a good research paper has the following 
ingredients:

1.  A concise introduction, which up-front states the problem 
being tackled, as well as any previous research in order to 
establish the freshness of the ideas or implementation 
presented.  The freshness criteria is really important in any
line of research, and without establishing freshness, the 
paper loses merit.   This is especially true in a fast moving 
discipline such as computer science,

2.  The very good papers usually provide a fairly thorough
overview of all related work, emphasizing how the work in the 
paper differs from the previous work.  Good papers do a good 
job of explaining the necessary background in high level terms.  
For example, the NASD paper explains the various ways of 
implementing file systems, from the point of view of computers 
and network interfaces involved.  This may be simple from an 
advanced readers point of view, but usually enhances the 
credibility of the authors.  Examples of papers that do a good 
job of explaining the related material and emphasizing how this 
work is different from all of those are: Traxtents, Abacus, 
NASD, etc,

3.  In computer systems, good papers usually present an 
implementation of the idea, in one of the three forms: (a) 
simulation, (b) prototype, (c) full-scale implementation that 
is ready for production use.  In some instances, really good 
ideas that are potentially break-through material do not 
present any implementations,

4.  Implementation is good, but it is also important to state
up front all the choices available and the design decisions that
went into making specific choices.  For example, the RPC paper
presents a variety of design decisions.  It then states one
particular implementation.  Apparently, Nelson's thesis on 
RPCs, the original work in the area, did even a better job of 
stating the choices available.

Papers that are follow-up on earlier implementation usually
have to stand to higher standards.  For example, the AFS paper
is a follow-up to one that described the prototype 
implementation, and as such, the shortcomings of the prototype
design are revelead, and alternative, better designs are presented,

5.  The results, usually in the form of experiments, 
comparisons with similar, related systems, benchmarks, etc, are 
clearly stated.  It is inevitable, that some of the results are 
worse than previous or related work.  However, in most cases in 
order to justify a good, publishable paper, the results have to 
be positive.  In order to convince the readers of the validity 
of the results, the paper needs to make an effort to use already 
established metrics, and compare the results to other systems 
that are similar.  In rare cases, the paper may introduce new 
metrics.  This is typically true of work that is in very new 
areas.  For example, the AFS paper introduces metrics that are 
used for distributed file system performance measurement.

A good systems paper should honestly present the results without 
attempting to bias the user's attention towards test cases that 
make the paper look good.  It is ok to mention not so good 
results, and explain why that is so.  Typically, the credibility 
of the paper is enhanced when authors can explain analytically 
the results,

6.  Esp. in those cases, when paper is a follow-up on earlier
paper or prototype work, it is important to have a lessons
learned section.  Typically, some of the best lessons are
learned when scaling from a prototype solution to a full-scale
solution.  Also, a good source of lessons could be 
implementation challenges, or even failures.

7.  Presentation is an important element in a paper in a 
fast-evolving discipline such as computer systems.  Since
there is a large body of literature:  conference-published
papers, archival journals, etc, poor presentation might
kill an otherwise good paper.  It is fair to assume that
people reading papers have little time, and it is best to
capture the readers attention and keep it while she reads
the paper.  To achieve this, paper must be well-organized,
be concisely put, and present good ideas up front.  Often 
times portions of paper devoted to certain sections do
not represent the relative time spent doing the underlying
task.  This is a difficult challenge to overcome, but in 
order to achieve a well-readable paper, it is important
to suppress details that showed progress or struggle.  
Most good papers spend very little about the process of 
implementation, and instead focus on (i) design choices,
(2) specific challenges, and (3) decisions, techniques,
or algorithms used.

While I could go into further detail in each of this 
aspects, for the sake of concision I will conclude my
summary at this point.