Write Review a Paper
Why do you care?
- Your advisor will one day ask you to review a paper.
- You will graduate and one day be asked to review a paper.
- Your papers will be reviewed like this... get into your reviewer's head!
- Make the paper scribblable
- I load the paper into my tablet or print it out. It's important that (a) I'm comfortable, (b) distraction-free, and (c) can make unrestricted markings.
- I generally read the paper in a single pass. But I write down everything.
- Review notes
- Back at a text editor/CMT/etc... I go through the notes and ensure that each note I made is (a) addressed in the paper, and/or (b) appears in my review.
A good paper doesn't need to win in all categories.
Does the paper address a relevant problem or provide useful insights?
- The paper identifies a new problem.
- The paper provides new insights on an old problem.
- The paper identifies new applications of old techniques.
Why would someone want to read this paper?
Does the paper solve the problem it set out to solve?
- Validate the list of claims (if present).
- Does the paper have a broad/narrow enough scope?
- Is the initial motivation addressed?
- ... or has a sufficient milestone been met?
Is the paper clickbait?
Are statements in the paper correct?
- System design/Algorithms
- Unexpected runtime costs.
- Unhandled corner cases.
- Experimental design doesn't test what the authors claim.
- Results don't agree with author's claims.
How would you solve the problem the authors pose yourself?
Is the paper written clearly
- There is a clearly stated problem / objective.
- All background topics not covered in [grad level class] are outlined.
- System design and formalisms are clear, precise, and complete.
- The paper is free of English bugs, grammar bugs, typos, etc...
Do you understand the paper?
Reading the Paper: Milestones
- Do I understand the problem the paper is solving?
- How would I go about solving the problem the paper outlines?
- How would I go about measuring a solution to this problem?
- Do I have a reasonable understanding of the techniques the paper plans to use?
Reading the Paper: Milestones
- Algorithm/Data Structures
- Do I understand the approach the authors are taking?
- If the approach doesn't line up with my own, why?
- Are all of my expected experiments from earlier addressed?
- Do the experiments measure what the authors want to measure?
- Are the datasets reasonable/representative of the motivating workloads?
- Do the graphs support the paper's claims?
- Communicate to the authors at least one way to address your concern.
- Establish a clear metric that can test whether the concern is addressed.
- Include citations where possible.
- Differentiate suggestions from criticism.
- Refer to specific lines of text.
Why did the reviewer write it?
- Did the reviewer misunderstand what you wrote?
What can you do to make it clearer?
- Did the reviewer ignore an important point?
What can you do to make it more visible?
- Did the reviewer not have the right background?
What can you do to make it more accessible?
- Did the reviewer disagree with the motivation?
"Pitch" the motivation to your fellow students/faculty.
The reviewer is a sample of the people who will be reviewing your paper. They may be wrong, but you still need to communicate to others like them to get an accept!