Guidelines

Short paper track aka. Track 1

We wish to acknowledge the following as sources of inspiration for the following guidelines:

• CVPR 2018 guidelines for reviewers.

Important dates

April 26th, 11:59PM ET: Deadline for submission
April 23rd: Area Chairs and reviewers bidding periods begin
April 25th: Area Chairs and reviewers bidding periods end
April 26th: Short Papers Track Chairs assign short papers to Area Chairs
April 27th: Area Chairs assign short papers to their reviewers
May 7th: Review period ends
May 7th: Area Chairs meta-review starts
May 14th: Area Chairs give back their meta-reviews and decisions
May 18th: Final decision, notification of acceptance

Contact information

aisg2019.icml.contact@gmail.com

Blind Reviews

Authors were asked to take reasonable efforts to hide their identities, including not listing their names or affiliations and omitting acknowledgments. This information will of course be included in the final version. Reviewers should also make all efforts to keep their identity invisible to the authors. ArXiv papers are not considered prior work since they have not been peer reviewed. Therefore, you should review your short papers independently as if the arXiv papers didn't exist. Citations to these papers are not required and failing to cite or beat performance of arXiv papers are not grounds for rejection. An important general guideline is to make every effort to treat papers fairly whether or not they know (or suspect) who wrote them. Reviewers should not search for the authors of a paper, and complain that the paper is not anonymous if they happen to find them.

Check Your Papers

As soon as you get your reviewing assignment, please go through all the short papers to make sure that (a) there is no obvious conflict with you (e.g., a short paper authored by your recent collaborator from a different institution) and (b) you feel comfortable to review the short paper assigned. If either of these issues arise, please let us know right away by contacting your the Area Chair using CMT.

What to Look For

Minor flaws can be corrected and shouldn't be a reason to reject a short paper.
Acceptance and rejection decisions should not be determined solely by the method's raw performance. Rather, it is important to weigh both the novelty and potential impact of the work on the society alongside the reported performance. Impact can be on the research community, but also on the specific problem being addressed.

Each short paper that is accepted should be technically sound and make a contribution to the field.

It is important to note that the potential impact of a proposition is one of the most important criteria. Some problems do not require technical novelty, but instead thoughtful application of existing methods. Papers in that category (i.e. impactful but not necessarily novel) should not be rejected for this reason only.

Be Careful

Please think carefully about your reviews. In particular, it's a good idea to avoid ad-hoc policy innovations, which can occur with the best of intentions. Here is an example. Author submits a short paper relying on a dataset that cannot be public. Reviewer takes the position that this means the results cannot be trusted, and rejects on these grounds. The problem with this review is that it's clearly about a matter of policy, rather than about the short paper's content. We don't have a policy about non-public datasets, and it's unfair for reviewers to invent one of their own.

When You're Done

When you have finished with your review, you should destroy any short paper manuscript and/or supporting material you received. See the Ethics guidelines below.

Writing Technical Reviews

Here are some specific issues to keep in mind as you write your reviews:

• Extremely short reviews are unhelpful to authors, other reviewers, and Area Chairs. If you have agreed to review a short paper, you should take enough time to write a thoughtful and detailed review.
• Be specific when you suggest that the writing needs to be improved. If there is a particular section that is unclear, point it out and give suggestions for how it can be clarified.
• Don't give away your identity by asking the authors to cite several of your own papers.
• Be specific about novelty. Claims in a review that the submitted work "has been done before" MUST be backed up with specific references and an explanation of how closely they are related. At the same time, for a positive review, be sure to summarize what novel aspects are most interesting in the strengths.
• Citations to papers that have only been published without review (e.g. ArXiv or Technical reports) are not required. Therefore, missing these citations is not grounds for rejecting a paper.
• If you think the short paper is out of scope for the workshop subject areas, clearly explain why in the review.
• Avoid referring to the authors by using the phrase "you". These phrases should be replaced by "the authors" or "the short paper." Referring to the authors as "you" can be perceived as being confrontational, even though you do not mean it this way.

Ethics for Reviewing Papers

1. Protect Ideas
You have the responsibility to protect the confidentiality of the ideas represented in the short papers you review. Protection of the ideas in the short papers you receive means:

• You should not show the short paper to anyone else, including colleagues or students, unless you have asked them to write a review, or to help with your review.
• You should not show any results or any of the supplementary material to non-reviewers.
• You should not use ideas from short papers you review to develop new ones.
• After the review process, you should destroy all copies of short papers and erase any implementations you have written to evaluate the ideas in the short papers, as well as any results of those implementations.

2. Avoid Conflict of Interest
There should be absolutely no question about the impartiality of any review. Thus, if you are assigned a paper where your review would create a possible conflict of interest, you should return the short paper and not submit a review. Conflicts of interest include (but are not limited to) situations in which:

• You work at the same institution as one of the authors.
• You have been directly involved in the work and will be receiving credit in some way. If you're a member of the author's thesis committee, and the paper is about his or her thesis work, then you were involved.
• You suspect that others might see a conflict of interest in your involvement.
• You have collaborated with one of the authors in the past three years (more or less). Collaboration is usually defined as having written a paper or grant proposal together, although you should use your judgment.
• You were the MS/PhD advisor of one of the authors or the MS/PhD advisee of one of the authors. Most funding agencies and publications typically consider advisees to represent a lifetime conflict of interest.

While the organizers make every effort to avoid such conflicts in the review assignments, they may nonetheless occasionally arise. If you recognize the work or the author and feel it could present a conflict of interest, contact your Area Chair using CMT as soon as possible so he or she can find someone else to review it.

3. Be Professional
Belittling or sarcastic comments have no place in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. Write a courteous, informative, incisive, and helpful review that you would be proud to add your name to (were it not anonymous).

Reviewer Instructions

The following provides further details for reviewing papers:

• Please make sure that your browser has cookies and Javascript enabled.
• Please add "cmt@microsoft.com" to your list of safe senders in your own email client to prevent important email announcements from being blocked by spam filters.

Once you've been notified that the papers have been assigned to you, please log in to the CMT site and look at the papers.

Reviewer FAQs

Is there a minimum number of papers I should accept or reject?

No. Each paper should be evaluated in its own right. If you feel that most of the papers assigned to you have value, you should accept them. It is unlikely that most papers are bad enough to justify rejecting them all. However, if that is the case, provide clear and very specific comments in each review. Do NOT assume that your stack of papers necessarily should have the same acceptance rate as the entire conference ultimately will.

Can I review a paper I already saw on arXiv and hence know who the authors are?

Yes. See next bullet below for guidelines.

How should I treat papers for which I know the authors?

Reviewers should make every effort to treat each paper fairly, whether or not they know who wrote the paper. For example: It is Not OK for a reviewer to read a paper, think "I know who wrote this; it's on arXiv; they're usually quite good" and accept paper based on that reasoning. Conversely, it is also Not OK for a reviewer to read a paper, think "I know who wrote this; it's on arXiv; they're no good" and reject paper based on that reasoning.

How should I treat arXiv papers?

ArXiv papers are not considered prior work since they have not been peer reviewed. Therefore, you should review your AISG workshop short papers independently as if the ArXiv papers didn't exist. Citations to these papers are not required and failing to cite or beat performance of arXiv papers are not grounds for rejection. For example:

• It is Not OK for a reviewer to suggest rejection for not citing an arXiv paper or not being better than something on arXiv.
• It is Not OK to accept a paper solely because it performs better than something on arXiv.
• It is Not OK to reject a paper solely because it performs worse than something on arXiv.
• It is Not OK to regard arXiv as a standard for the state of the art, because it is not reviewed. This applies *whoever* wrote the arXiv paper.
• It is Not OK for a reviewer to reject a paper solely because another paper with a similar idea has already appeared on arXiv. If the reviewer is worried about plagiarism they should bring this up in confidential comments to the AC.
• It is OK for a reviewer to suggest an author should acknowledge and be aware of something on arXiv.
• It is OK for an author to decline to acknowledge something on arXiv (because it has not been reviewed and so may not be right).

Plagiarism: Plagiarism is a regular nuisance in publication. Plagiarism consists of appropriating the words or results of another, without credit. Generally, reviewers can recognize plagiarism when they see it; it is unlikely that a reviewer will be uncertain whether plagiarism has occurred.

Project Proposals track aka. Track 2

Though the content of the submissions differ (proposal vs. paper), the reviewer guidelines for track 1 continue to apply. We make some additions to those guidelines due to the different form of track 2 reviews.

Important Dates

April 26th, 11:59PM ET : Deadline for submission
April 23rd: Open Proposal Reviews
April 29th: Screening reviews Due
April 30th: Proposal Pre-Acceptance
May 1st: Project Mentoring start
May 14th: Selection Reviews Due
May 18th: Notification of Acceptance

Writing Proposal Reviews

• We emphasize that a carefully crafted proposal that has not necessarily been developed previously is more valuable to the community than work that has been started but which doesn't hold as much long-term potential.
• Highlight points that are important for the proposal's evaluation, but which are ambiguous from the submission alone. These comments will be shared with interactive reviewers.
• Provide constructive feedback. If something seems unimpactful or infeasible, provide recommendations about how some of the limitations might be addressed.
• As a general suggestion, consider including the following in your review,
  • Summary: An objective description of the proposal (without your personal commentary) is useful for your area chair.
  • Clarity: Are sufficient details provided in the proposal for you to understand how the project could be developed? This does not mean that all details of implementation have been specified, but all next steps should be transparent.
  • Potential Value: How much real-world value does the proposed project add? How much value does it add to the community's thinking about AI for Good?
  • Decision Justification: Describe the arguments for and against passing the proposal to the next phase, and how you arrived at your final conclusion.