How to Organize a Conference Peer Review – Checklist

• What is a peer review and why is it essential for a scientific conference?
• Planning the peer review: How to establish a process
  • ✔️Checklist: How to find the best process for your peer review
• The external paper review
  • ✔️Checklist: Organizing an external review
    • Preliminary considerations
    • Review process
    • Influence the review process
• The final paper review
  • ✔️Checklist: Planning the final review
• Communicating with authors during the peer review process
  • ✔️Checklist for messages to paper authors
• 🤖How does AI change the peer review process for scientific conferences?
• ❓FAQ
  • How does peer review at conferences differ from journal reviews?
  • Can a reviewer review their own paper?
  • How do I choose the right reviewers?
  • How do I deal with late or missing reviews?
  • How much time should authors be given for revision?
• 🎯Further tips for a successful peer review

What is a peer review and why is it essential for a scientific conference?

The peer review process is the core of scientific quality assurance. It refers to the process in which experts (peers) review and evaluate conference papers submitted by researchers. The evaluation is therefore carried out by experts who are familiar with the topic because they have a profound knowledge that stems from their own research or practical experience in a specific field.

Peer review usually begins as soon as the deadline for the call for papers is over.

The peer review process for scientific conferences usually consists of two stages:

• External review by independent reviewers, who make an informed recommendation.
• The final decision made by the program committee that determines whether a paper is accepted, rejected, or needs to be revised.

A well-structured peer review process is therefore much more than just a formality. It ensures transparency, quality, and fairness. And it helps to position a scientific conference as a platform for excellent research.
In this article, we show you how to plan your peer review process, find the right reviewers, and map the whole process digitally.

Planning the peer review: How to establish a process

Before the reviewers start looking at the first abstract assigned to them, you need to answer the following question: What level of complexity do you want for your peer review?

The process varies depending on the type and size of a conference. Some organizers rely on a very simple setup in which the program committee or the person responsible for managing the conference program reviews all contributions themselves.

There’s an advantage here of course: You can move fast. It doesn’t take long to decide whether a paper will make it into the conference program or not. Only a few people are involved in the decision. The more reviewers you have, the more common are delays. In this scenario, there is no need to coordinate several people.

Other organizers prefer a multi-stage process. Each paper is reviewed by several experts.

Feedback loops are also crucial. Either you are strict and only distinguish between “paper accepted” and “paper rejected.” Or there is a third category, “To be revised,” for papers that are convincing in terms of content but could use a little fine-tuning.

✔️Checklist: How to find the best process for your peer review

• Do you prefer a simple setup for your peer review process?
  Only a few people are involved. For example, the conference president is the only person to review the submissions and decides which ones will be included in the program.
  This saves you loads of time because information does not have to be circulated among several people (or groups of people) and you don’t have to answer any questions.
• Or is you peer review process a bit more complex?
  Upon submission, a paper goes through several stages. It is reviewed by at least two experts. In the final stage, a central authority (usually a committee) decides whether and how to proceed with the paper.
  If this is the case for your conference, outline the different stages:
  • Which and how many people (groups of people) are involved in the peer review process?
  • How many and which stages does a paper go through?
  • At what points do you need to inform which people?

The external paper review

Once you’ve established the process, the next step is to hand the papers over to external experts.

This is where the reviewers come in. They evaluate the papers from their expert perspective, make recommendations, and pave the way for the program committee’s final decision.

The review usually takes place in a secured online area, which reviewers can access via a personal login. It’s up to you as the organizer to specify the questions that the reviewers must answer for each paper. These usually include the content’s relevance, the paper’s methodological quality, or how original the topic and the content of the paper are.

Whether you carry out this external review depends on the standards and scope of your conference as described above. For very small conferences with a manageable number of submissions, organizers sometimes decide to make the review an internal task.
For larger conferences, however, a multi-stage review by external experts is the standard. It ensures transparency, fairness, and scientific quality.

Speaking of fairness: A study by Pleskac, Kyung, Chapman & Urminsky found in 2024 that single-blind reviews at conferences tend to favor well-known authors and disadvantage scientists with Asian names. This suggests that reviewers judge differently when they know whose paper they are reviewing.

✔️Checklist: Organizing an external review

Preliminary considerations

First, let’s cover the basics of the review process: Who evaluates what, and how exactly does it work?

• Who are the reviewers?
  Compile a list of all reviewers as soon as you know who they are. For the first step, their names, email addresses, and the organization or institution they belong to are sufficient.
• Are these people already in your abstract management system?
  If you already have a full list of all your reviewers, you should be able to import it to the software you are using to organize the review before you start assigning the abstracts. So if your software system doesn’t know who the reviewers are, first check how you can find a way to get them there. Find a method that saves you from typing in all the names.
  Some abstract management systems offer a CSV template. This is a table that you fill in with the reviewer data and then simply upload to the system.
• Which reviewer is an expert on which topic?
  It is best to add the expertise to the list right away and assign topics to the individuals.
• How many experts do you need for each of your topics?
  Having several experts on a single topic provides different perspectives and reduces the risk of biased decisions. Also consider which individuals could potentially take on more than one topic.
• What questions do reviewers answer for an abstract?
  Sometimes it is enough for them to make an “accept” or “reject.” decision In other cases, you might want them to answer additional questions about an abstract’s quality.
• How many submissions are a reasonable amount to handle for one reviewer?
  This depends, among other things, on the number and scope of the questions. As a rule, plan for 4–8 abstracts per reviewer. If the time frame is very tight, aim for the lower end of the range; for less in-depth reviews or smaller conferences, aim for the upper end (up to ~10).
• What if not all reviewers are done with their task on time?
  Add some extra time for yourself in case you need to redistribute papers. As an alternative, consider: Are the results the other reviewers provided sufficient?
• Are reviewers allowed to review their own abstracts?
  The review should be as neutral as possible. Authors are often “blind” when it comes to their own work. External reviewers provide a more neutral judgment.
• Should a reviewer make a decision regarding the form of the presentation?
  Sometimes authors already have a preferred presentation format. Decide whether reviewers should see this. You can also let the reviewers decide.
• Should reviewers see how many versions of an abstract someone has submitted?
  This is particularly useful for multi-stage reviews. It allows reviewers to understand what an author has edited since submitting the initial version of the abstract or the previous one.
• Should reviewers be able to download the abstracts assigned to them?
  This could be an option for those who travel a lot and may only want to enter their suggestions once. However, this is now rather uncommon. If you want to offer this option, you should communicate that the results must be entered into the abstract management system.
• Should reviewers be able to see who wrote the paper?
  That’s up to you: Do you prefer a 100% anonymous review process? Or one where the author’s details are known to everyone? Or one that is limited to the author’s name only?
• Should reviewers be able to see who else reviews an abstract?
  There are two sides to transparency: on the one hand, it can lead to more consistent reviews, but sometimes reviewers also influence each other.
• Should reviewers be able to see all conference papers?
  An overview makes it easier for reviewers to identify thematic connections and better classify abstracts. However, always provide access in such a way that no confidential information is disclosed.
• Should reviewers be able to send messages to authors?
  The program committee decides whether messages can be emailed via placeholders together with the results of the review. This is definitely helpful if an author needs to revise their abstract and there is a second round of review. Then they receive feedback and can fine-tune their paper.
• Should reviewers be able to send messages to the program committee?
  This can be useful. However, the answers provided by a reviewer during the review are often sufficient.

Review process

Let’s fast forward to the actual review. There can be four options for each paper. You decide which of these will be part of your peer review process:

• Accept
  The paper is accepted and will become part of your conference program.
• Reject
  The reviewers (and the committee) consider the paper not suitable for the conference. It will not be included in the conference program.
• Waiting list
  If you have more accepted papers than there are presentation slots, a waiting list may be useful. Sometimes presenters cancel at short notice. Papers on the waiting list then automatically move up.
  
  When you inform the authors that their paper has been accepted, provide a link where they can immediately accept or decline the invitation to present it at the conference.
  
  Inform authors on the waiting list when they can expect an acceptance or rejection message from you.
• Revise
  If a feedback loop with revision is part of your peer review process, work backwards from a fixed deadline. For example, take the date on which you release the conference program. This is the best way to keep track of all the intermediate steps.
  
  Then consider who will review the revised abstracts. The very same reviewers? That’s the fastest way, because they are already familiar with the first draft and know what to look for in the next round. Once round two is finished, the program committee will make the final decision.
  

Influence the review process

This concerns the criteria which are the base for your reviewers’ work.

• What questions should reviewers answer in the peer review process?
  Does the paper fit the conference’s main topic? Is the paper based on current scientific findings?
• Mandatory information for the review
  Very important: What assessment do you absolutely need from the reviewers? In most cases, this is a recommendation to either accept or reject the paper.
  However, other questions may also be included, such as whether the chosen form of presentation is appropriate.
• What should the final review be based on?
  There are two options here: Either the individual criteria (answers) of the reviewers are taken into account, or you award points. Each answer is awarded points, which together result in an average grade an abstract receives.
• Would you like to group the criteria?
  If there are many questions, you can group them so they deal with a specific aspect. Conferences often distinguish between “content” and “form”.

The final paper review

Almost done! Now the program committee has their final say. Sometimes the committee is just one person, sometimes a group of experts. For very large conferences, there may even be several committees, which makes the review process a little more complex.

In any case, the committee already has a rough idea of what an interesting conference program looks like from their scientific point of view and how many slots are available for each topic area.

✔️Checklist: Planning the final review

• Who makes the final decision?
  Normally, this is a task for the program committee. Sometimes the final decision is made only by a single person. Prior to any kind of decision, the team usually meets to discuss the results of the review.
• What answers do program managers have to provide?
  Obviously, the end result is a final decision. If a paper is to be included in the program, the scientific committee agrees upon the topic the paper belongs to. Part of the decision is also the form of presentation: Should the author(s) present the paper? Will it become a poste? Or is it going to be turned into a workshop?

Communicating with authors during the peer review process

If a software supports you during peer review, you can send email the results directly to the authors.

✔️Checklist for messages to paper authors

• Who do you want to inform about the review results?
  Decide whether only the person who submitted the abstract should receive the email or all of the authors.
• What else do you want to communicate?
  Create different email templates in advance for each scenario – accepted, rejected, waiting list, revision required. Add any information that is important for the authors, such as deadlines for revisions or confirmation of participation.
• Do you also need different email texts for the different presentation formats?
  Here, too, it makes sense to differentiate: What information is important for speakers? What do poster authors need to know? What is only relevant to workshop leaders?
• Do you need active confirmation from the authors?
  This is very important for papers that are currently on the waiting list. Who could step in if necessary, and who will not be able to participate?

🤖How does AI change the peer review process for scientific conferences?

Don’t worry: machines won’t be taking over peer review completely anytime soon. But there is an interesting initiative coming from the Association for the Advancement of Artificial Intelligence (AAAI) that is launching an AI pilot program for its 2026 conference.

This could be groundbreaking. Large language models (LLMs) are intended to expand the review process without compromising human expertise or even replacing human reviewers.

Specifically, this involves two stages of the workflow:

• In the first phase of the review, an AI-generated review is provided for each paper. This is intended to provide an additional perspective to the human reviewers’ assessments.
• The program committee also receives assistance: AI summarizes the human reviewers’ findings. This allows the committee to quickly see where the reviews agree and where there is still room for discussion.

In the long term, the results could also influence other conferences. It is worth keeping an eye on technical innovations, especially when they make life easier for organizers.

❓FAQ

How does peer review at conferences differ from journal reviews?

At academic conferences, the review process is usually more time-sensitive and geared toward forms of presentations (oral presentations, posters, etc.), while reviews in scientific journals focus more on depth of content and readiness for publication. Conference reviews often also rate how good the paper fits the general topic of the conference and its relevance to the audience.

Can a reviewer review their own paper?

No. For reasons of independence and fairness, reviewers are not allowed to rate their own submissions. The peer review software should technically make it impossible to assign someone their own abstract for review. Otherwise, there might be conflicts of interest and that disturbs the integrity of the entire process.

How do I choose the right reviewers?

Select reviewers based on their area of expertise, experience, and availability. A structured pool of reviewers with clearly defined subject areas makes it easier for you to assign them papers. If you have a fixed list of reviewers, check it regularly. Has their current research focus shifted?

How do I deal with late or missing reviews?

Allow for buffer times. Communicate clear deadlines and send out automatic reminders.
If the reviewers are slow, you can reassign abstracts if necessary or – if there are multiple reviewers – just go with the reviews you’ve received so far.

How much time should authors be given for revision?

The deadline depends on how extensive the changes are. In practice, a period of one to two weeks has proven effective for minor adjustments. Important: Plan the deadline backwards starting from the day you want to release the program.

🎯Further tips for a successful peer review