How to Review a Paper

- Your advisor will one day ask you to review a paper.
- You will graduate and one day be asked to review a paper.
- Your papers will be reviewed like this... get into your reviewer's head!

- Make the paper scribblable
- I load the paper into my tablet or print it out. It's important that (a) I'm comfortable, (b) distraction-free, and (c) can make unrestricted markings.
- Read-Through
- I generally read the paper in a single pass. But I write down
**everything**. - Review notes
- Back at a text editor/CMT/etc... I go through the notes and ensure that each note I made is (a) addressed in the paper, and/or (b) appears in my review.

- Motivation
- Completeness
- Validity
- Readability

A good paper doesn't need to win in all categories.

Does the paper address a relevant problem or provide useful insights?

- The paper identifies a new problem.
- The paper provides new insights on an old problem.
- The paper identifies new applications of old techniques.

Why would someone want to read this paper?

Does the paper solve the problem it set out to solve?

- Validate the list of claims (if present).
- Does the paper have a broad/narrow enough scope?
- Is the initial motivation addressed?
- ... or has a sufficient milestone been met?

Is the paper clickbait?

Are statements in the paper correct?

- System design/Algorithms
- Unexpected runtime costs.
- Unhandled corner cases.

- Experiments
- Experimental design doesn't test what the authors claim.
- Results don't agree with author's claims.

- Proofs
- Sanity check

How would you solve the problem the authors pose yourself?

Is the paper written clearly

- There is a clearly stated problem / objective.
- All background topics not covered in [grad level class] are outlined.
- System design and formalisms are clear, precise, and complete.
- The paper is free of English bugs, grammar bugs, typos, etc...

Do you understand the paper?

- Introduction
- Do I understand the problem the paper is solving?
- How would I go about solving the problem the paper outlines?
- How would I go about measuring a solution to this problem?
- Background
- Do I have a reasonable understanding of the techniques the paper plans to use?

- Algorithm/Data Structures
- Do I understand the approach the authors are taking?
- If the approach doesn't line up with my own, why?
- Experiments
- Are all of my expected experiments from earlier addressed?
- Do the experiments measure what the authors want to measure?
- Are the datasets reasonable/representative of the motivating workloads?
- Do the graphs support the paper's claims?

Be Specific

- Communicate to the authors at least one way to address your concern.
- Establish a clear metric that can test whether the concern is addressed.
- Include citations where possible.
- Differentiate suggestions from criticism.
- Refer to specific lines of text.

Why did the reviewer write it?

- Did the reviewer misunderstand what you wrote?
What can you do to make it clearer?
- Did the reviewer ignore an important point?
What can you do to make it more visible?
- Did the reviewer not have the right background?
What can you do to make it more accessible?
- Did the reviewer disagree with the motivation?
"Pitch" the motivation to your fellow students/faculty.

The reviewer is a sample of the people who will be reviewing your paper. They may be wrong, but you still need to communicate to others like them to get an accept!