Review Guidelines
=================

Dick Epema

last updated: march 2009

See also: A.J. Smith, The Task of the Referee, IEEE Computer, 
          Vol. 23(4), pp. 65-71, 1990.


Structure of a review:

1. An abstract in your own words of the paper, including a concise problem
   statement and a summary of the results, which serves three purposes:
   a. it forces you to formulate what the actual topic and 
      contribution of the paper is
   b. it is a signal to the author(s) that you have actually read
      and understood the paper
   c. it can help the editor of the journal or the program committee to
      better understand the review and to reach a decision

2. A list of the main strengths of the paper. Examples are:
   a. "this is a very important and timely topic to work on"
   b. "this paper has a very strong and thorough experimental section"
   c. "this paper is really very well written"

3. A list of the main weaknesses of the paper. Examples are:
   a. "this result was already published in the CACM in 1961"
   b. "there is no motivation at all for the problem investigated"
   c. "this paper is very poorly written "
   d. "the paper is not very well structured in that ..."

Don't overdo things in 2. and 3. above, these are the main points! So
lists of 2-5 items are reasonable. Also try to be concise in these points,
you can explain them in more detail under 4. below (e.g., examples
of poor writing or structure).

4. Detailed comments. Examples are:
   a. "the abstract should include an indication of the results of the paper"
   b. "the legends in Figures 2 and 5 are too small to be read"
   c. "the authors may check and refer to the paper [authors/title/journal]"
   d. "formula (3) is incorrect because ..."
   e. "the paper is not very well structured in that Section N should be
       subdivided into subsections and that ..."

You can be pretty elaborate in the detailed comments (much more so than in the
list above).

5. Numerical assessment. Usually, conferences ask for numerical assessments
of papers for such aspects as originality, technical content, and readability,
and for overall quality. If they don't, you may give such assessments anyway.
Some journals have lists of questions for reviewers that are usually of a 
more qualitative character.

In general, the better the conference or journal, the more thorough
and elaborate a review should be. But try to be concise and to the point
(and constructive) an editor or program committee member has to go through 
lots of these things!