How to Write Review a Paper

Why do you care?

• Your advisor will one day ask you to review a paper.
• You will graduate and one day be asked to review a paper.
• Your papers will be reviewed like this... get into your reviewer's head!

My Process

Make the paper scribblable

I load the paper into my tablet or print it out. It's important that (a) I'm comfortable, (b) distraction-free, and (c) can make unrestricted markings.

Read-Through

I generally read the paper in a single pass. But I write down everything.

Review notes

Back at a text editor/CMT/etc... I go through the notes and ensure that each note I made is (a) addressed in the paper, and/or (b) appears in my review.

Criterion

• Motivation
• Completeness
• Validity
• Readability

A good paper doesn't need to win in all categories.

Motivation

Does the paper address a relevant problem or provide useful insights?

• The paper identifies a new problem.
• The paper provides new insights on an old problem.
• The paper identifies new applications of old techniques.

Why would someone want to read this paper?

Completeness

Does the paper solve the problem it set out to solve?

• Validate the list of claims (if present).
• Does the paper have a broad/narrow enough scope?
• Is the initial motivation addressed?
  • ... or has a sufficient milestone been met?

Is the paper clickbait?

Validity

Are statements in the paper correct?

• System design/Algorithms
  • Unexpected runtime costs.
  • Unhandled corner cases.
• Experiments
  • Experimental design doesn't test what the authors claim.
  • Results don't agree with author's claims.
• Proofs
  • Sanity check

How would you solve the problem the authors pose yourself?

Readability

Is the paper written clearly

• There is a clearly stated problem / objective.
• All background topics not covered in [grad level class] are outlined.
• System design and formalisms are clear, precise, and complete.
• The paper is free of English bugs, grammar bugs, typos, etc...

Do you understand the paper?

Reading the Paper: Milestones

Introduction

Do I understand the problem the paper is solving?

How would I go about solving the problem the paper outlines?

How would I go about measuring a solution to this problem?

Background

Do I have a reasonable understanding of the techniques the paper plans to use?

Reading the Paper: Milestones

Algorithm/Data Structures

Do I understand the approach the authors are taking?

If the approach doesn't line up with my own, why?

Experiments

Are all of my expected experiments from earlier addressed?

Do the experiments measure what the authors want to measure?

Are the datasets reasonable/representative of the motivating workloads?

Do the graphs support the paper's claims?

Feedback

Be Specific

• Communicate to the authors at least one way to address your concern.
• Establish a clear metric that can test whether the concern is addressed.
• Include citations where possible.
• Differentiate suggestions from criticism.
• Refer to specific lines of text.

Receiving Feedback

Why did the reviewer write it?

• Did the reviewer misunderstand what you wrote?
  What can you do to make it clearer?
• Did the reviewer ignore an important point?
  What can you do to make it more visible?
• Did the reviewer not have the right background?
  What can you do to make it more accessible?
• Did the reviewer disagree with the motivation?
  "Pitch" the motivation to your fellow students/faculty.

The reviewer is a sample of the people who will be reviewing your paper. They may be wrong, but you still need to communicate to others like them to get an accept!