Pamitc

Area Chair Guidelines
This page provides detailed guidelines to Area Chairs of CVPR14 as to how the process will unfold and what their responsibilities are. Please note that there are some changes in the review process compared to previous CVPRs.

Role of the Area Chairs

Prior to the AC meeting, the ACs help identify suitable reviewers for each paper, write initial consolidation reports for papers with diverging reviews, evaluate the reviews as well as the authors' responses, write a final consolidation report and suggest acceptance/reject decisions for each paper, and check the reject decisions of a buddy AC. ACs are expected to play an active role in ensuring higher quality and more timely reviews, and in initiating discussions amongst reviewers (anonymously), and in looking at authors' responses to the reviews. At the AC meeting in February, ACs will work with other ACs (in triplets and then in panels) to decide which papers to accept and finalize the consolidation reports to explain the decisions.


Best Practices of Being an Area Chair

We want to remind all ACs of some of the basic (and obvious) practices of being an AC.
  • You have a strong influence on the decision for a paper. Take your job very seriously and be fair.
  • If you identify any paper that you believe may be written by a current or former student, post-doc or supervisor, or by a collaborator in the last two years, please notify the ACs immediately so that the paper can be assigned to another AC.
  • The Program Co-Chairs did not submit any papers, so they are not in direct conflict with any papers and they know all authors of all papers and also all reviewers/ACs. They will NOT be involved with any decisions of papers from institutions with which they are in conflict. You can send them a question about any paper and if one of them is conflicted, another one will respond to you without including the conflicted PC Chair.
  • DO NOT talk to any other AC about papers assigned to you without prior approval from the PCs as there may be several other ACs conflicted with the paper. You will only be allowed to talk to other ACs in your AC triplet or panel, which will be designed to be conflict free.
  • DO NOT talk to any other AC about your own paper (the paper you are an author on) or a paper you have some conflict with, during this whole process.
  • Be professional and willing to listen to other reviewers and ACs. Do not give in to undue influence from anyone.
  • Remember, it is not fair to dismiss any review without looking at other reviews and reading the paper yourself to make an evaluation. Outright rejecting a review is not acceptable without a clearly articulated supporting argument. Area chairs who wish to make a decision that is not clearly supported by the reviews will have to present the paper to their panel at the meeting.
  • Senior members of the committee should guide and advise the younger and first-time members. First-time members, please ask for help as needed. Note that next time you may be the senior member and questions could flow to you. This is an opportunity to learn about the process.

Consolidation Reports

Area Chair Consolidation Reports are the MOST CRUCIAL aspect of the review process of CVPR. This is where the Area Chair justifies his/her recommendation to accept/reject a paper. The Program Chairs will read all Consolidation Reports and we have committed ourselves to ensure that these reports are representative of a good quality review process. These reports should highlight why the decision was reached. If all reviewers agree on a paper, this consolidation report can be simple, but feel free to encourage authors and provide constructive feedback. If there is even a slight disagreement on the reviews, it is your job to clarify why and how the disagreement was resolved. Again, just saying "reviewer 1 was wrong" without any reasons is not acceptable. Explain why. For the most divergent reviews, explain the process you went through to reach a decision and if you and others read the paper, try to add some details from your and others reviews with the consolidation reports. Overruling all three reviewers to make a decision on a paper is not acceptable. If you strongly feel that the paper deserves that, make sure to discuss it in detail with your AC Buddy and also in your AC Panel. In such cases, (a) you should read the paper completely, (b) provide a detailed consolidation report, and (c) solicit confirmation of the decision from your panel. The program chairs will review and confirm or reverse any such decisions that do not have the consensus support of the panel. Please use the format provided in the meta-review form to write your final consolidation report, which includes 1-2 sentences on each one of the following points a) initial ratings, b) main reviewer critiques, c) author's rebuttal, d) AC discussion with reviewer's, e) initial AC's recommendation, f) discussion at the AC triplet and panel, and g) final recommendation.


Important Dates:

November 1: Paper submission deadline
November 15: Papers assigned to area chairs
November 24: Reviewer suggestions due
December 1: Papers assigned to reviewers
January 13: Reviews due
January 20: Initial consolidation report due for papers with diverging reviews
January 27: Rebuttal due
Jan 27-Feb 3: Discussion between area chairs and reviewers
February 3: Final reviewer recommendations due
February 12: Final consolidation reports due
February 14: Reject papers assigned to shadow ACs
February 18: Shadow AC decisions on reject papers due
February 22-23: Area Chair meeting


Meta-Review Form

Pre-Rebuttal Feedback to Authors

For papers with diverging reviews or papers where you disagree with the reviewers, please summarize the key things you would like the authors to include in their rebuttals to facilitate your decision-making. There is no need to summarize the paper or reviews. If you have additional concerns that were not included in the reviews, please be sure and include them as well.

Preliminary Rating

Please rate the paper according to the following choices.
  • Oral: These are papers whose quality is in the top 10% of the papers at CVPR. Examples include a theoretical breakthrough with no experiments; an interesting solution to a new problem; a novel solution to an existing problem with solid experiments; or an incremental paper that leads to dramatic improvements in performance.
  • Oral/Poster: These are very strong papers, which may have one weakness that makes you unsure as to whether they should be oral or poster.
  • Poster: These are strong papers, which have more than one weakness. For example, a well-written paper with solid experiments, but incremental; a paper on a well-studied problem with solid theory, but weak experiments; or a novel paper with good experiments, but poorly written.
  • Reject: These are papers that have some promise, but they would be better off by being revised and resubmitted, or have a major flaw, or have been done before. Reject decisions will be reviewed by an AC.

Final Consolidation Report

Please summarize the decision process for the paper, including the reviews, rebuttal, discussions, and AC/panel recommendations. For example: "Initially, the paper received mixed ratings: 2 weak rejects and 1 poster. The main critiques were that the proof of Theorem 1 was unclear and that experimental comparison with [1] was missing. The authors' rebuttal explained the proof of Theorem 1 in more detail and argued that comparison with [1] was inappropriate. During discussion, R2 was convinced that the experiments were sufficient and upgraded the rating to poster. However, R3 was still unsatisfied with the proof of Theorem 1. The AC agreed that the proof of Theorem 1 is unconvincing, found it essential for the method's correctness, and recommended rejection in spite of the 2 poster recommendations. The secondary AC recommended that the paper be discussed at the AC meeting. The AC panel discussed the paper extensively and agreed that the proof of Theorem 1 is essential. Therefore, the panel recommends that the paper be rejected."


Detailed Area Chair Instructions

During the reviewing phase, please periodically log in to the CMT site and monitor the review progress.

Phase 1 - Assigning Reviewers

1) Reviewer Assignment: Submitted papers will be assigned to ACs around November 15. Each AC will receive between 30 and 35 papers. Papers will be assigned to you based on (a) papers from authors not in conflict with you (see conflict of interest in the Reviewer Instructions), (b) results from the "Toronto" document matching system, (c) subject areas you chose when you registered on the review site, and (d) requests by the authors. As it can be imagined, this will not be perfect as sometimes the Program Chairs have to do some load-balancing. Area Chairs should look through all the papers assigned to them and ensure that they are (1) not conflicted and (2) are knowledgeable in the subject area.

For each paper you are assigned you must initially select 7 or more candidate reviewers during the reviewer assignment step. November 15. You will have until November 24 to suggest reviewers. The CMT system will then select 3 from each candidate list, subject to necessary constraints on reviewer load. If it is not able to assign 3 from your list, it will assign fewer than 3, and you will be required to iterate again suggesting additional reviewers. Several iterations may be necessary if you only suggest the minimal number of candidate reviewers, and those will occur in a relatively short cycle (24 or 48 hours). If you plan to only assign the minimal number of reviewers, please expect to be available on short notice to add a few more to the papers which are not satisfied on the first round. If you will be unavailable for several days, it is better to assign 10 or more candidate reviewers. Please assign as many candidate reviewers as possible to each paper, as long as you are comfortable getting any subset of 3 actually assigned. You must assign at least 7, and the more you assign, the greater the chances you will not be required to suggest additional reviewer names for that paper. To the extent possible the CMT system will only assign papers to reviewers you suggest. While CMT will not automatically do so, the program chairs may personally assign reviewers based on preference keyword match or any other scheme necessary to papers of ACs who do not respond in a timely manner.

Here are some suggestions on how to proceed. It can take anywhere from 4-8 minutes for a paper to do this, if you look at the paper and other info to help make a decision. Considering a load of 30 papers, you can see how this may add up quickly. You have been chosen as an Area Chair because of your specific expertise in certain areas of computer vision and also for your general expertise in other areas of computer vision. When you look at a paper, you may just come up with a name or three of ideal reviewers. Consider these people as the reviewers for this, but do also consider things like, (a) are they current in the field and (b) will they do a good job in a timely manner.
  • Look at the paper, especially the Introduction, Related Work, and Citations at the end. See who they refer to and whose work they are building on. Often the cited people would be ideal reviewers, but not exclusively.
  • Remember that the system will give you some recommendations, based on the score of the Toronto matching system and the score obtained from matching subject areas. This may provide a good starting point, but will not be perfect.
  • Usually, we suggest that you choose a reviewer you know and trust. But the vision community is large. If the system recommends someone you do not know, look at their webpage and Google Scholar profile or at their papers on ACM or IEEE digital libraries.
  • Be sure to spend a decent amount of time doing this. The quality of the reviewer assignments has a tremendous impact on the quality of the reviews and the final decisions.
  • You may find it helpful to maintain your own spreadsheet with your list of preferred reviewers for each paper and the final paper assignments.

Phase 2 - Checking Incoming Reviews

After you have ranked reviewers, the Program Chairs, with the review system's help will distribute papers to the reviewers. During this phase, we recommend that you log into the system to check if any reviews are being entered. If reviews are showing up, look at them and see if (a) the reviewer has added sufficient detail and (b) has used appropriate tone in the review, and most importantly (c) provided a knowledgeable review.

After the review deadline has passed, check if all reviews are in. If not, send them a note and nicely demand that they finish the reviews ASAP. The system will also send reminders to them, but a personal note is often more effective.
  • In "Area Chair" console, select "Consolidation Reports".
  • You should be able to see all the reviewers' names and the review status right next to each. For this incarnation of CMT, we just highlight cases where reviews have not been submitted (in red).
  • For each paper, you can:
    • View all the reviews ("View All").
    • Email the reviewers anonymously via CMT ("Email Reviewers"). You can select all the assigned reviewers for that paper (default), or just those not done with reviewing, or specific reviewer(s). For the last item, you will be able to edit the recipient list. Remember: Don't identify yourself or the other reviewers in the email.
  • You can also choose to bulk email all reviewers or all reviewers who have not completed their reviews (for all the papers in your stack). See below the title "Consolidation Reports" ("Email Reviewer"). Mouse over the "Email Reviewer" box and you will see the two options (mentioned earlier) from which to choose.
  • You can view all reviews for all papers in a single webpage. Just select "View All Reviews" in the line right below the title "Consolidation Reports".
  • Important note on emailing via CMT: Since any email sent on your behalf is anonymously sent through CMT, the communication is one way only. Please tell your email recipients not to respond to the email, because there is no automatic routing of email to you. So, be very specific about what you would like the reviewer(s) to do, e.g., improve their review in a certain way.
  • It is the area chair's responsibility to contact and urge the reviewers to finish their reviews in a timely manner.

Phase 3 - Initial Consolidation Report

Reviewers will have until January 13 to complete their reviews. You will then have till January 20 to look through the reviews write an initial consolidation report for papers with diverging reviews. This is your opportunity to tell the authors your views on the paper if different from what was expressed by the authors. You may also let the authors know which ones are the most important issues that need to be addressed in the rebuttal.

Please look through ALL the reviews. If any review appears unreasonable (e.g., one-liner comments or overly harsh), please contact the reviewer and ask for clarification. Remember that communication is only one-way, so be specific about what you need from the reviewer. Check again in one or two days to see if there is any update to that review. Since we will not be keeping track of your emails, so please do the book-keeping yourself (best to make a note of what you've done). Again, you may find it helpful to create and maintain your own spreadsheet to keep track of the reviews, scores, and any high-level discussion items or to-do lists.


Phase 4 - Rebuttal Period

During the author rebuttal period reviewers are no longer allowed to modify their reviews.


Phase 5 - Final Consolidation Report and Discussion with the Reviewers

Rebuttals will be due on January 27. You will then initiate a discussion with the reviewers for papers with diverging reviews and make sure that the reviewers submit their post-rebuttal decision by February 3rd. You will have until February 12 to finalize your consolidation reports. At this point you will need to enter a tentative decision for each paper. Papers that you mark for rejection will be checked by a fellow AC and if he/she agrees with your decision, these papers will not be discussed at the AC meeting. All other decisions are just proposed and will be finalized at the AC Meeting. All consolidation reports will be released to the authors immediately after the AC meeting.

5.1. When to Initiate a Discussion with Reviewers

Please initiate a discussion any time you feel that the overall scores and/or the comments are insufficient to allow you to make a decision (e.g., all papers with conflicting reviews). At the very least, initiate a discussion if the difference between the maximum and minimum scores is greater than 2 levels. In addition, please initiate a discussion in order to understand if the rebuttal responds properly to the reviewers' concerns. You can take the input from the reviewers to help in your proposed decision and the consolidation report that you draft.

5.2. During the Discussion

You play the role of a moderator in the discussion. While it is ideal to reach a consensus, don't feel obligated to force one - the discussions are mainly to allow you to make more informed decisions. It is ok if the reviewers end up not changing their overall scores as long as you've gained a better understanding of how they arrived at their scores. Stick to the facts, and do not influence the outcome of the discussion by imposing your views on the paper. However, if you disagree with the majority view, say so, provide compelling reasons, and solicit feedback from the reviewers. If the discussion looks like it is getting out of hand (e.g., when a reviewer makes inappropriate remarks or a battle of words erupts)

5.3. Anonymous Discussion Feature

This feature can be accessed from the "Detailed Meta-Reviews and Discussions" page:
  • You must do the following steps to enable the reviewers to participate in the discussion. Emailing the reviewers via CMT is NOT the right way to initiate the discussion.
  • To initiate a discussion for a specific paper, first enable the discussion and then select "View/Post Message" under the "Discussion" column for that paper.
  • In the "Paper Discussion" page, click on "Start A New Topic" to initiate the discussion (unless a reviewer took the initiative of starting one, in which case you reply; see the next bullet item). We suggest that your first post starts with a very brief summary of the reviews, a request to look at the other reviews for details, followed by specific things you want the reviewers to address.
  • Once a discussion has been initiated, click on "Reply" on the far right to continue posting on the same discussion thread.
  • Please identify the reviewers by the review number. Never identify other reviewers by name, so that their identities are kept from the other reviewers.
  • Anytime a post is made (either by the area chair or a reviewer), the area chair and reviewers will receive an email notification from CMT with the subject that looks like "CVPR14: New reviewer discussion posted for Paper ID XXX". There is a link in the email you can use to join the discussion (after logging in, you will be routed directly to the discussion page). Alternatively, you can just log in to CMT as usual.
  • After you've posted, do not refresh the page (e.g., by hitting F5)! This will generate another post with the exact same message!
  • In the discussion, you (as AC) can see the identities of the posters. However, each reviewer will NOT know the identities of the other posters.
  • Once the author rebuttal period is over, you and the reviewers will be able to see the author rebuttal.
  • Because of the frank nature of the discussions, the authors will not see them at any time. The consolidation reports will be visible to authors only after the AC meeting, when decisions are made known.
  • Reviewer instructions for the discussion feature are given here . Note that reviewers can revise their reviews until the review deadline.

5.4. Writing the Final Consolidation Report

Final consolidation reports are due on Feb 12 and they should summarize the decision process for the paper, including the reviews, rebuttal, discussions, and AC/panel recommendations. For example: "Initially, the paper received mixed ratings: 2 weak rejects and 1 poster. The main critiques were that the proof of Theorem 1 was unclear and that experimental comparison with [1] was missing. The authors' rebuttal explained the proof of Theorem 1 in more detail and argued that comparison with [1] was inappropriate. During discussion, R2 was convinced that the experiments were sufficient and upgraded the rating to poster. However, R3 was still unsatisfied with the proof of Theorem 1. The AC agreed that the proof of Theorem 1 is unconvincing, found it essential for the method's correctness, and recommended rejection in spite of the 2 poster recommendations. The secondary AC recommended that the paper be discussed at the AC meeting. The AC panel discussed the paper extensively and agreed that the proof of Theorem 1 is essential. Therefore, the panel recommends that the paper be rejected."


Phase 6 - Early Rejections

On February 14 you will receive a small set of papers that other ACs recommend for rejection. Your job is to make sure that the reject decision and the consolidation report is appropriate. If you disagree with the decision, you need to inform the PCs and these papers will be discussed at the AC meeting. If you agree with the decision, the paper will be rejected with no further discussion. Your decisions are anonymous. If you decide that a paper deserves further discussion, the panel will not know who made that recommendation, only that the shadow AC felt the paper deserves further discussion.


Consolidation Scores on the AC Website

To allow you to see the reviewer scores on the consolidation page, in the Area Chair "Consolidation Report" page, you can customize the page by selecting "Customize The View"->"Edit Review Columns". Select Q2 ("Overall Rating") and click on "OK" to see the updated table.


Phase 7 - Prepare for AC Meeting Discuss with AC Buddies, Propose Decisions on Paper

The Area Chair Meeting will occur February 22-23 in College Park MD. This will require you to travel to College Park. We need everyone to be present, since there will be a panel discussion (more extensive than in recent years).


Area Chair FAQs

Conflicts with Papers Assigned to Area Chairs
  • I recognize a piece of work (i.e., I know the authors) should I recuse myself from the paper? They are not collaborators and I have no financial ties or other connections with them. My feeling is that I can be impartial and handle the review but it is your call.
    If you think you can do the review impartially and fairly, and there is no direct conflict, we think you should review this paper, as we feel you are best qualified for it. Thanks for letting us know.
  • I have been assigned a paper that might be pretty close to things I am working on at present (I don't know for sure since I didn't want to look at the full paper). I would like to decline handling that paper, if possible, on grounds of potential conflict of interest.
    Thanks for letting us know. Yes, this may qualify as a conflict of interest so we will assign to some other Area Chair.
  • I see a paper that I know is authored by a recent collaborator of mine (collaborated in last 3 years). I am reasonably confident I can still render a fair and an impartial review. But wanted to let you know.
    This appears to be a conflict. While we do trust you to be fair and impartial, it is best to have someone else take over this paper. Thanks for letting us know. We will swap it.
  • I have a paper assigned to me that appears to be very similar to one that I have just co-authored for CVPR14. I can be still objective in review of this paper, irrespective of my submission.
    Thanks for telling us. We feel you are the expert in the area and trust you will do a good job. If you feel you cannot, then we will re-assign to another area chair.

Assigning Reviewers
  • What should I do with a paper with the names of the authors explicitly mentioned?
    Papers with author names explicitly violate the anonymity requirement of CVPR and may be rejected. Let the Program Chairs know the paper id.
  • What should I do with a paper that did not exactly follow the required format?
    If the paper is still 2 column style CVPR format (missing the line numbers, paper id on every page, etc.) and not longer than 8 pages (font not too small, margins ok, etc.), then it is fine to assign reviewers. (We don't want to be too rigid.)
  • What about papers over 8 pages long?
    Overall, our hardline policy is (as the authors were warned!) that the paper will be rejected. If you see a paper with just a citation or 1-2 lines on page 9, then we are letting it go. But if the paper is grossly over onto page 9, then it is an "administrative" reject.
  • Is there a race to assign reviewers? If I do not assign reviewers right away, will I lose all the good reviewers for papers assigned to me?
    No, there is NO race. You are recommending 7 reviewers for each paper and on Nov 24 we will optimize the assignments based on (a) reviewer load, (b) your choice/rank of a reviewer, and (c) subject area match to reviewer. We hope that this way, good reviewers per your choice will be assigned the paper for which you are an AC. However, we do not want to see all of you waiting until the last week to assign papers.
  • I have scanned a paper and it seems very weak and will be rejected, what should I do?
    Unfortunately, if the paper has been submitted and does not meet the criteria for an administrative reject (non-anonymous, too long, dual-submission, etc.), the paper has to be reviewed. Give the paper a fair chance and have it reviewed. We agree that sometimes this is a waste of reviewing resources, but we feel the authors deserve the best from us.

Assigning New Reviewers
  • When should I decide to request a new reviewer that is not in the database of reviewers?
    After you scan a paper to see its content, some obvious names of reviewers will come to your mind. Look for them in the reviewer lists. All reviewers should be visible to you (there are over 700 of them, so it is a bit slow). They are sorted by their relevance. If you do not see the person you wish to review the paper, consider adding them as an additional reviewer. Consider however that (a) this person may already have been asked to review for CVPR14 and may have declined and (b) this person may not have the time to review this time. So do not assume that the new reviewer will agree to do this review. Add an extra reviewer as a back-up for this one.

Area Chairs Meeting
  • When and where is the Area Chairs' Meeting?
    AC Meeting will be held on Feb 22-23. All ACs are expected to arrive on Feb 21 (Fri), and be there until the session ends on the evening of Feb 23 (Sun). All ACs are invited to stay an extra day, Feb 24 (Mon) to participate in an informal workshop on computer vision and related topics.
  • When should we make reservations for the AC meeting?
    You will be getting instructions from us about how to plan for your travel and how you will be reimbursed in a separate e-mail.