Pamitc

Reviewer Guidelines
Thank you for agreeing to review for IEEE CVPR14. The quality of the conference depends to a large extent on the willingness of reviewers like you to provide thoughtful, high-quality reviews. Please read through the rest of this document that provides details on what is expected of you as a member of the Reviewing Committee for CVPR14.


The CVPR14 Reviewing Timeline

• October 20: Reviewer registration deadline
• November 1: Paper submission deadline
• December 1: Papers assigned to reviewers
• January 13: Reviews due
• January 27: Author rebuttal due
• Jan 27 - Feb 3: Paper discussion with Area Chairs and fellow reviewers
• February 3: Final reviewer recommendation due
• March 1: Final decision to authors


The CVPR14 Reviewer Guidelines

Check your papers

As soon as you get your reviewing assignment, please go through all the papers to make sure that (a) there is NO obvious conflict with you (e.g., a paper authored by your recent collaborator from a different institution) and (b) you feel comfortable to review the paper assigned. If these issues arise, please respond right away in CMT or by emailing the Program Chairs.

What to Look For

Look for what's good or stimulating in the paper. Minor flaws can be corrected and shouldn't be a reason to reject a paper. CVPR as a conference is looking for new ideas. We recommend that you embrace novel, brave concepts, even if they have not been tested on many datasets. Each paper that is accepted should, however, be technically sound and make some contribution to the field.

Blind Reviews

Authors were asked to take reasonable efforts to hide their identities, including not listing their names or affiliations and omitting acknowledgments. This information will of course be included in the published version. Reviewers should also make all efforts to keep their identity invisible to the authors.

Be Specific

Please be specific and detailed in your reviews. In the discussion of related work and references, simply saying "this is well known" or "this has been common practice in the industry for years" is not sufficient: cite specific publications or public disclosures of techniques.

Your main critique of the paper should be written as answers to question 2 and 3 in the review form, which ask for the paper’s strength and weaknesses. Use bullet points here, and explain your arguments. Your discussion, sometimes more than your score, will help the authors, fellow reviewers, and Area Chairs understand the basis of your opinions, so please be thorough. Your reviews will be returned to the authors, so you should include any specific feedback on ways the authors can use to improve their papers. For more suggestions on writing your reviews, read the section below on Writing Technical Reviews.

When You're Done

When you have finished with your review, you should destroy any paper manuscript and/or supporting material you received. See the Ethics guidelines below.


Writing Technical Reviews

Here are some recommendations that may help you as you do this very valuable task. In many professions, people give back to their community by doing volunteer work. In technical fields, we volunteer our time by reviewing papers that are written by other researchers in our field. We recommend that you approach your reviews in this spirit of volunteerism. Sure, your reviews make you a gatekeeper in helping decide which papers are ready for publication. Just as important, however, is to provide feedback to the authors so that they may improve their work. Try to write your review in a way that the authors can benefit from. We suggest reading a paper and then thinking about it over the course of several days before you write your review. The tone of your review is important. A harshly written review will be disregarded by the authors, regardless of whether your criticisms are true. If you take care, it is always possible to word your review diplomatically while staying true to your thoughts about the paper. Put yourself in the mindset of writing to someone you wish to help, such as a respected colleague who wants your opinion on a concept or a project. Here are some specific issues to keep in mind as you write your reviews:
  • Short reviews are unhelpful to the authors and to other reviewers. If you have agreed to review a paper, you should take enough time to write a thoughtful and detailed review.
  • Be specific when you suggest that the writing needs to be improved. If there is a particular section that is unclear, point it out and give suggestions for how it can be clarified.
  • Don't give away your identity by asking the authors to cite several of your own papers.
  • If you don't think the paper is right for the CVPR program, suggest other publication possibilities (journals, conferences, workshops) that would be a better match for the paper.
  • Avoid referring to the authors by using the phrase "you" or "the authors." These phrases should be replaced by "the paper." Directly talking about the authors can be perceived as being confrontational, even though you do not mean it this way.
Be generous about giving the authors new ideas for how they can improve their work. Your suggestions may be very specific (for example, "this numerical solver would be better for your application") or may be more general in nature. You might suggest a new dataset that could be tried, or a new application area that might benefit from their tool. You may tell them how their idea can be generalized beyond what they have already considered. A thoughtful review not only benefits the authors, but may well benefit you, too. Remember that your reviews are read by other reviewers and especially the Area Chairs, in addition to the authors. Being a helpful reviewer will generate good will toward you in the research community.


Ethics for Reviewing Papers

1. Protect Ideas
As a reviewer for CVPR, you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. CVPR submissions are not published documents. The work is considered new or proprietary by the authors; otherwise they would not have submitted it. Of course, their intent is to ultimately publish to the world, but most of the submitted papers will not appear in the CVPR proceedings. Thus, it is likely that the paper you have in your hands will be refined further and submitted to some other journal or conference, or even to CVPR next year. Sometimes the work is still considered confidential by the author's employers. These organizations do not consider sending a paper to CVPR for review to constitute a public disclosure. Protection of the ideas in the papers you receive means:
  • Do not show the paper to anyone else, including colleagues or students, unless you have asked them to write a review, or to help with your review.
  • Do not show any results or videos/images or any of the supplementary material to non-reviewers.
  • Do not use ideas from papers you review to develop new ones.
  • After the review process, destroy all copies of papers and videos that are not returned to the senior reviewer and erase any implementations you have written to evaluate the ideas in the papers, as well as any results of those implementations.
2. Avoid Conflict of Interest
As a reviewer of a CVPR paper, you have a certain power over the reviewing process. It is important for you to avoid any conflict of interest. Even though you would, of course, act impartially on any paper, there should be absolutely no question about the impartiality of review. Thus, if you are assigned a paper where your review would create a possible conflict of interest, you should return the paper and not submit a review. Conflicts of interest include (but are not limited to) situations in which:
  • You work at the same institution as one of the authors.
  • You have been directly involved in the work and will be receiving credit in some way. If you're a member of the author's thesis committee, and the paper is about his or her thesis work, then you were involved.
  • You suspect that others might see a conflict of interest in your involvement. For example, even though Microsoft Research in Seattle and Beijing are in some ways more distant than Berkeley and MIT, there is likely to be a perception that they are "both Microsoft," so folks from one should not review papers from the other.
  • You have collaborated with one of the authors in the past three years (more or less). Collaboration is usually defined as having written a paper or grant proposal together, although you should use your judgment.
  • You were the MS/PhD advisor of one of the authors or the MS/PhD advisee of one of the authors. Most funding agencies and publications typically consider advisees to represent a lifetime conflict of interest. CVPR has traditionally been more flexible than this, but you should think carefully before reviewing a paper you know to be written by a former advisee.
If you recognize the work or the author and feel it could present a conflict of interest, send the paper back to the Area Chair as soon as possible so he or she can find someone else to review it.

3. Be Professional
Belittling or sarcastic comments have no place in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. Write a courteous, informative, incisive, and helpful review that you would be proud to add your name to (were it not anonymous).

The best review is the one you wish someone else had written when reviewing one of your papers.


Reviewer Instructions
The following provides further details for reviewing papers using the CVPR14 submission and reviewer system. In addition, please see the Reviewer Guidelines above and the FAQs below.
  • The submission/review site is /CVPR14/ (bookmark or save this URL!)
  • Please make sure that your browser has cookies and Javascript enabled.
  • Please add to your list of safe senders to prevent important email announcements from being blocked by spam filters.
  • Do not create a new account! You will have the opportunity to change your contact email/account name after you log in using the email address we used to contact you.
Discussions with Area Chairs via CMT

In our invitation, we mentioned that we expect reviewers to work with area chairs to clear up confusions and reach consensus on papers. This new site has an electronic bulletin board feature that allows area chairs to contact reviewers anonymously. Once the area chair posts a note, reviewers will be notified and asked to log in to see the post and respond. The identities of the reviewers will be hidden from each other.

Once you've been notified that the papers have been assigned to you, please log in to the site and follow these steps:

1. Download papers and check for possible conflict or submission rule violations:
  • Click on "Paper Reviews and Discussions".
  • In the "Paper Reviews and Discussions" page, click on "Download Assigned Papers". This allows you to download a zip file containing all the papers plus supplementary files (if available).
    1. You think you are in any way conflicted with the paper.
    2. There is a violation of the stated paper submission rules.
    3. Such a violation includes:
      • Over 8 pages,
      • Constitutes a double submission,
      • Supplementary material includes a newer version of the paper.
      • Please specify the exact nature of the violation.
  • For your reference, the paper submission guidelines can be found here.
2. Familiarize yourself with the "Paper Reviews and Discussions" page:
  • "Paper Summary" label: next to it, you'll see the icons "+" and "-". Clicking on "+" shows you all the abstracts; clicking on "-" collapses all them back.
  • At the end of each paper title, you'll see "+" as well. This has the same function of showing the abstract for that paper, toggling to "-" at the same time, which collapses it when selected.
  • Please take the time to familiarize yourself with the table entries; clicking on any of the column heading (e.g., "Paper ID" or "Rank") sorts according to its description.
3. Review papers:
  • For a paper, under the review column, click "Add" (to the right of the "Review" line) to review. Please read instructions carefully. Please see the Reviewer Guidelines AND take each review seriously. Authors are counting on you for a fair and thorough review.
  • Currently, CMT does not allow users to type certain characters into a text box that could be interpreted as html tags (for example, "y<x") or a malicious script. As a workaround, introducing spaces between these characters (for example, "y < x") will allow you to submit the text since this can no longer be interpreted as an html tag.
  • If you save your review as a draft, it is visible only to you. You can access your draft review form by clicking on the same "Add" link. To make the review visible to the area chair, click on the "Submit" button in the review form. "Submit" won't work if any of the required items is not filled.
4. (Optional) Review papers offline:
  • You have two options to access the "Offline Reviewing" page: (1) In the "Paper Reviews and Discussions" page, click on "Review papers offline" link near the top of the page, or (2) In the "View/Edit Review" page, click on "offline reviewing" link.
  • In the "Offline Reviewing" page, you can download one review template file for a single paper, several papers, or all the papers. We suggest that you download a review template file for each paper to avoid confusion.
  • Please read instructions on how to modify the file to incorporate your responses. Note that you must not add certain characters in your responses that could be interpreted as html tags or a malicious script. See item 3 above.
  • You can upload the completed file using the "Upload" interface at the bottom of the page. The new uploaded version will (destructively) overwrite the current review.
  • We suggest that you try downloading a review template file for one paper, enter test responses, and upload to get a sense of how it works.
  • You should always verify the review after uploading (by inspecting it online).
  • We suggest that you use an XML editor to edit the file, for example: EditiX (Windows, Unix/Linux, Mac OS X) or XML Notepad (Windows only). (Remember to edit only fields currently filled with the phrase "REPLACE THIS WITH YOUR ANSWER".)
5. CMT Review Form

Paper Summary. Please summarize in your own words what the paper is about.

Paper Strengths. Please discuss the positive aspects of the paper. Be sure to comment on the paper's novelty, technical correctness, clarity and experimental evaluation. Notice that different papers may need different levels of evaluation: a theoretical paper may need no experiments, while a paper presenting a new approach to a known problem may require thorough comparisons to existing methods. Also, please make sure to justify your comments in great detail. For example, if you think the paper is novel, not only say so, but also explain in detail why you think this is the case.

Paper Weaknesses. Please discuss the negative aspects of the paper: lack of novelty, technical errors, lack of clarity, insufficient experimental evaluation, etc. Please justify your comments in great detail. If you think the paper is not novel, explain why and give a reference to prior work. Do not ask the authors to cite your own work. If you think this is essential, write it in the confidential comments. The AC will pass it to the authors if appropriate. If you think there is an error in the paper, provide details of where the error is and why it is an error. If you think the experimental evaluation is insufficient, remember that theoretical results/ideas are essential to the CVPR community and that a theoretical paper need not have experiments. It is not appropriate to reject a paper simply because it did not outperform other existing algorithms, especially if the theory is novel, interesting or presents some value to the community. It is also not okay to ask for comparisons with unpublished papers and papers published after the CVPR deadline. Last but not least, remember to be polite and constructive in your comments.

Preliminary Rating. Please rate the paper according to the following choices.

Oral: these are papers whose quality is in the top 10% of the papers at CVPR. Examples include a theoretical breakthrough with no experiments; an interesting solution to a new problem; a novel solution to an existing problem with solid experiments; or an incremental paper that leads to dramatic improvements in performance.

Oral/Poster: these are very strong papers, which may have one weakness that makes you unsure as to whether they should be oral or poster.

Poster: these are strong papers, which have more than one weakness. For example, a well-written paper with solid experiments, but incremental; a paper on a well-studied problem with solid theory, but weak experiments; or a novel paper with good experiments, but poorly written.

Weak Reject: these are papers that have some promise, but they would be better off by being revised and resubmitted.

Strong Reject: these are papers that have a major flaw, or have been done before.

Preliminary Evaluation. Please indicate to the AC, your fellow reviewers, and the authors your current opinion on the paper. Please summarize the key things you would like the authors to include in their rebuttals to facilitate your decision making. There is no need to summarize the paper or reviews. If you have additional concerns that were not included in the reviews, please be sure and include them as well.

Confidence. Write "Very Confident" to stress that you are absolutely sure about your conclusions (e.g., you are an expert who works in the paper's area), "Confident" to stress that you are mostly sure about your conclusions (e.g., you are not an expert but can distinguish good work from bad work in that area), and "Not Confident" to stress that that you feel some doubt about your conclusions. In the latter case, please provide details.

Confidential Comments to the PC/AC Chairs (will not be seen by authors)

Reviewed by. The assigned reviewer must complete his review form and write his/here name here. Under very unusual circumstances, someone else designated by the assigned reviewer might do the review. In that case, please write the name of the alternate reviewer and explain why the review was reassigned. In either case, by submitting this form, the assigned reviewer states that he/she agrees with the content of the review and is responsible for it.

The following questions should be answered after the rebuttal.

Final Rating. After reading the author's rebuttal, please rate the paper according to the following choices.

Final Recommendation. After reading the author's rebuttal and the discussion, please explain your final recommendation. Your explanation will be of highest importance for making acceptance decisions and for deciding between posters and orals. Include suggestions for improvement and publication alternatives, if appropriate.

Finally




Reviewer FAQs
  • What is the first thing I should do after the papers are assigned to me?
    Go through the assigned papers quickly to check on two things. (1) Confirm that there is NO obvious CONFLICT of INTEREST with this paper, and (2) check if this paper is REALLY OUTSIDE your range of expertise (remember, we are matching to subject areas you told us about).
  • Is there another review form format available for easier editing?
    No. The review format was designed to make parsing unambiguous. We recommend saving one file for each paper being reviewed. In the file, simply replace with "Response to comment question."
  • Is there a minimum number of papers I should accept or reject?
    No. Each paper should be evaluated in its own right. If you feel that most of the papers assigned to you have value, you should accept them. It is unlikely that most papers are bad enough to justify rejecting them all. If that were the case, provide clear and very specific comments in each review. Area Chairs are suspicious when they see reviewers that reject too many papers.