|Area Chair Guidelines|
This page provides detailed guidelines to Area Chairs of CVPR 2014 as to how the process will unfold and what their responsibilities are. Please note that there are some changes in the review process compared to previous CVPRs.|
Role of the Area Chairs
Prior to the AC meeting, the ACs help identify suitable reviewers for each paper, write initial consolidation reports for papers with diverging reviews, evaluate the reviews as well as the authors' responses, write a final consolidation report and suggest acceptance/reject decisions for each paper, and check the reject decisions of a buddy AC. ACs are expected to play an active role in ensuring higher quality and more timely reviews, and in initiating discussions amongst reviewers (anonymously), and in looking at authors' responses to the reviews. At the AC meeting in February 2014, ACs will work with other ACs (in triplets and then in panels) to decide which papers to accept and finalize the consolidation reports to explain the decisions.
Best Practices of Being an Area Chair
We want to remind all ACs of some of the basic (and obvious) practices of being an AC.
Area Chair Consolidation Reports are the MOST CRUCIAL aspect of the review process of CVPR. This is where the Area Chair justifies his/her recommendation to accept/reject a paper. The Program Chairs will read all Consolidation Reports and we have committed ourselves to ensure that these reports are representative of a good quality review process. These reports should highlight why the decision was reached. If all reviewers agree on a paper, this consolidation report can be simple, but feel free to encourage authors and provide constructive feedback. If there is even a slight disagreement on the reviews, it is your job to clarify why and how the disagreement was resolved. Again, just saying "reviewer 1 was wrong" without any reasons is not acceptable. Explain why. For the most divergent reviews, explain the process you went through to reach a decision and if you and others read the paper, try to add some details from your and others reviews with the consolidation reports. Overruling all three reviewers to make a decision on a paper is not acceptable. If you strongly feel that the paper deserves that, make sure to discuss it in detail with your AC Buddy and also in your AC Panel. In such cases, (a) you should read the paper completely, (b) provide a detailed consolidation report, and (c) solicit confirmation of the decision from your panel. The program chairs will review and confirm or reverse any such decisions that do not have the consensus support of the panel. Please use the format provided in the meta-review form to write your final consolidation report, which includes 1-2 sentences on each one of the following points a) initial ratings, b) main reviewer critiques, c) author's rebuttal, d) AC discussion with reviewer's, e) initial AC's recommendation, f) discussion at the AC triplet and panel, and g) final recommendation.
November 1: Paper submission deadline
November 15: Papers assigned to area chairs
November 24: Reviewer suggestions due
December 1: Papers assigned to reviewers
January 13: Reviews due
January 20: Initial consolidation report due for papers with diverging reviews
January 27: Rebuttal due
Jan 27-Feb 3: Discussion between area chairs and reviewers
February 3: Final reviewer recommendations due
February 12: Final consolidation reports due
February 14: Reject papers assigned to shadow ACs
February 18: Shadow AC decisions on reject papers due
February 22-23: Area Chair meeting
Pre-Rebuttal Feedback to Authors
For papers with diverging reviews or papers where you disagree with the reviewers, please summarize the key things you would like the authors to include in their rebuttals to facilitate your decision-making. There is no need to summarize the paper or reviews. If you have additional concerns that were not included in the reviews, please be sure and include them as well.
Please rate the paper according to the following choices.
Final Consolidation Report
Please summarize the decision process for the paper, including the reviews, rebuttal, discussions, and AC/panel recommendations. For example: "Initially, the paper received mixed ratings: 2 weak rejects and 1 poster. The main critiques were that the proof of Theorem 1 was unclear and that experimental comparison with  was missing. The authors' rebuttal explained the proof of Theorem 1 in more detail and argued that comparison with  was inappropriate. During discussion, R2 was convinced that the experiments were sufficient and upgraded the rating to poster. However, R3 was still unsatisfied with the proof of Theorem 1. The AC agreed that the proof of Theorem 1 is unconvincing, found it essential for the method's correctness, and recommended rejection in spite of the 2 poster recommendations. The secondary AC recommended that the paper be discussed at the AC meeting. The AC panel discussed the paper extensively and agreed that the proof of Theorem 1 is essential. Therefore, the panel recommends that the paper be rejected."
Detailed Area Chair Instructions
During the reviewing phase, please periodically log in to the CMT site and monitor the review progress.
Phase 1 - Assigning Reviewers
1) Reviewer Assignment: Submitted papers will be assigned to ACs around November 15, 2013. Each AC will receive between 30 and 35 papers. Papers will be assigned to you based on (a) papers from authors not in conflict with you (see conflict of interest in the Reviewer Instructions), (b) results from the "Toronto" document matching system, (c) subject areas you chose when you registered on the review site, and (d) requests by the authors. As it can be imagined, this will not be perfect as sometimes the Program Chairs have to do some load-balancing. Area Chairs should look through all the papers assigned to them and ensure that they are (1) not conflicted and (2) are knowledgeable in the subject area.
For each paper you are assigned you must initially select 7 or more candidate reviewers during the reviewer assignment step. November 15, 2013. You will have until November 24 to suggest reviewers. The CMT system will then select 3 from each candidate list, subject to necessary constraints on reviewer load. If it is not able to assign 3 from your list, it will assign fewer than 3, and you will be required to iterate again suggesting additional reviewers. Several iterations may be necessary if you only suggest the minimal number of candidate reviewers, and those will occur in a relatively short cycle (24 or 48 hours). If you plan to only assign the minimal number of reviewers, please expect to be available on short notice to add a few more to the papers which are not satisfied on the first round. If you will be unavailable for several days, it is better to assign 10 or more candidate reviewers. Please assign as many candidate reviewers as possible to each paper, as long as you are comfortable getting any subset of 3 actually assigned. You must assign at least 7, and the more you assign, the greater the chances you will not be required to suggest additional reviewer names for that paper. To the extent possible the CMT system will only assign papers to reviewers you suggest. While CMT will not automatically do so, the program chairs may personally assign reviewers based on preference keyword match or any other scheme necessary to papers of ACs who do not respond in a timely manner.
Here are some suggestions on how to proceed. It can take anywhere from 4-8 minutes for a paper to do this, if you look at the paper and other info to help make a decision. Considering a load of 30 papers, you can see how this may add up quickly. You have been chosen as an Area Chair because of your specific expertise in certain areas of computer vision and also for your general expertise in other areas of computer vision. When you look at a paper, you may just come up with a name or three of ideal reviewers. Consider these people as the reviewers for this, but do also consider things like, (a) are they current in the field and (b) will they do a good job in a timely manner.
Phase 2 - Checking Incoming Reviews
After you have ranked reviewers, the Program Chairs, with the review system's help will distribute papers to the reviewers. During this phase, we recommend that you log into the system to check if any reviews are being entered. If reviews are showing up, look at them and see if (a) the reviewer has added sufficient detail and (b) has used appropriate tone in the review, and most importantly (c) provided a knowledgeable review.
After the review deadline has passed, check if all reviews are in. If not, send them a note and nicely demand that they finish the reviews ASAP. The system will also send reminders to them, but a personal note is often more effective.
Phase 3 - Initial Consolidation Report
Reviewers will have until January 13 to complete their reviews. You will then have till January 20 to look through the reviews write an initial consolidation report for papers with diverging reviews. This is your opportunity to tell the authors your views on the paper if different from what was expressed by the authors. You may also let the authors know which ones are the most important issues that need to be addressed in the rebuttal.
Please look through ALL the reviews. If any review appears unreasonable (e.g., one-liner comments or overly harsh), please contact the reviewer and ask for clarification. Remember that communication is only one-way, so be specific about what you need from the reviewer. Check again in one or two days to see if there is any update to that review. Since we will not be keeping track of your emails, so please do the book-keeping yourself (best to make a note of what you've done). Again, you may find it helpful to create and maintain your own spreadsheet to keep track of the reviews, scores, and any high-level discussion items or to-do lists.
Phase 4 - Rebuttal Period
During the author rebuttal period reviewers are no longer allowed to modify their reviews.
Phase 5 - Final Consolidation Report and Discussion with the Reviewers
Rebuttals will be due on January 27. You will then initiate a discussion with the reviewers for papers with diverging reviews and make sure that the reviewers submit their post-rebuttal decision by February 3rd. You will have until February 12 to finalize your consolidation reports. At this point you will need to enter a tentative decision for each paper. Papers that you mark for rejection will be checked by a fellow AC and if he/she agrees with your decision, these papers will not be discussed at the AC meeting. All other decisions are just proposed and will be finalized at the AC Meeting. All consolidation reports will be released to the authors immediately after the AC meeting.
5.1. When to Initiate a Discussion with Reviewers
Please initiate a discussion any time you feel that the overall scores and/or the comments are insufficient to allow you to make a decision (e.g., all papers with conflicting reviews). At the very least, initiate a discussion if the difference between the maximum and minimum scores is greater than 2 levels. In addition, please initiate a discussion in order to understand if the rebuttal responds properly to the reviewers' concerns. You can take the input from the reviewers to help in your proposed decision and the consolidation report that you draft.
5.2. During the Discussion
You play the role of a moderator in the discussion. While it is ideal to reach a consensus, don't feel obligated to force one - the discussions are mainly to allow you to make more informed decisions. It is ok if the reviewers end up not changing their overall scores as long as you've gained a better understanding of how they arrived at their scores. Stick to the facts, and do not influence the outcome of the discussion by imposing your views on the paper. However, if you disagree with the majority view, say so, provide compelling reasons, and solicit feedback from the reviewers. If the discussion looks like it is getting out of hand (e.g., when a reviewer makes inappropriate remarks or a battle of words erupts), please contact us (cvpr...@pamitc.org) immediately. Don't forget to let us know the paper number.
5.3. Anonymous Discussion Feature
This feature can be accessed from the "Detailed Meta-Reviews and Discussions" page:
5.4. Writing the Final Consolidation Report
Final consolidation reports are due on Feb 12 and they should summarize the decision process for the paper, including the reviews, rebuttal, discussions, and AC/panel recommendations. For example: "Initially, the paper received mixed ratings: 2 weak rejects and 1 poster. The main critiques were that the proof of Theorem 1 was unclear and that experimental comparison with  was missing. The authors' rebuttal explained the proof of Theorem 1 in more detail and argued that comparison with  was inappropriate. During discussion, R2 was convinced that the experiments were sufficient and upgraded the rating to poster. However, R3 was still unsatisfied with the proof of Theorem 1. The AC agreed that the proof of Theorem 1 is unconvincing, found it essential for the method's correctness, and recommended rejection in spite of the 2 poster recommendations. The secondary AC recommended that the paper be discussed at the AC meeting. The AC panel discussed the paper extensively and agreed that the proof of Theorem 1 is essential. Therefore, the panel recommends that the paper be rejected."
Phase 6 - Early Rejections
On February 14 you will receive a small set of papers that other ACs recommend for rejection. Your job is to make sure that the reject decision and the consolidation report is appropriate. If you disagree with the decision, you need to inform the PCs and these papers will be discussed at the AC meeting. If you agree with the decision, the paper will be rejected with no further discussion. Your decisions are anonymous. If you decide that a paper deserves further discussion, the panel will not know who made that recommendation, only that the shadow AC felt the paper deserves further discussion.
Consolidation Scores on the AC Website
To allow you to see the reviewer scores on the consolidation page, in the Area Chair "Consolidation Report" page, you can customize the page by selecting "Customize The View"->"Edit Review Columns". Select Q2 ("Overall Rating") and click on "OK" to see the updated table.
Phase 7 - Prepare for AC Meeting Discuss with AC Buddies, Propose Decisions on Paper
The Area Chair Meeting will occur February 22-23, 2013 in College Park MD. This will require you to travel to College Park. We need everyone to be present, since there will be a panel discussion (more extensive than in recent years).
Area Chair FAQs
Conflicts with Papers Assigned to Area Chairs
Assigning New Reviewers
Area Chairs Meeting