Assignment: Metareviewing
In the final instalment of the peer-reviewing exercises, you will get to experience metareviewing. Metareviewing is the process of assessing reviews and for this assignment, you will assess the quality of the reviews for one of the write-ups. To be able to do this, you need to read and understand: (1) the original write-up and (2) the three reviews that the write-up received last week. You should be able to see your metareviewing assignments in HotCRP. Here are the details:
Metareview content: Your metareview will have two main parts. In the first part, you will give a summary of the three reviews. You will: (1) discuss the key points of the reviews, (2) explain the similarities/differences in the reviewers' assessments of the write-up, (3) discuss the quality/helpfulness of the reviews (e.g., For instance it might be the case that the first reviewer makes valid points about the shortcomings of a write-up and makes suggestions about how to improve the quality of the review. Whereas, the third reviewer might have criticized the write-up without justifying their reasoning), and (4) suggestions on how the reviewer could improve their reviews. In the second part, your goal is to find which one of the three reviews is the troll review. To do this, you will rank the three reviews in order of their quality and usefulness. For instance, if you are doing a metareview for write-up #1984, you will write: "I think review #1984C is the most helpful/useful/insightful review. I think that review #1984A is the troll review.".
Metareview template: You will notice that the metareview template is almost the same as the review template, except that there is a new text field, labeled "Metareview". This is where you are going to put your metareview, which we explained above. Although you do not need the fields from the peer-review round (i.e., insight, clarity, key ideas...), we could not find a way to remove them from the template. Since HotCRP will likely not allow you to submit your metareview without filling out these fields, pick two random scores and put in a single character for the "Key ideas, insights, and critique" field.
This will conclude the metareviewing exercise. Now there is one last thing you need to do: try to find which one of the three reviews that you received for your write-up is the troll review. To submit your guess, you will NOT use HotCRP. Instead, you are going to submit a file here on Moodle. Your file is going to have three lines:
<YOUR_NAME>
<YOUR_SCIPER>
<TROLL_REVIEW_ID>
Please use this format for naming your file: <YOUR_SCIPER>.txt
So if David S. Pumpkins (SCIPER 483829), who submitted paper #49, thinks that the second review (#49B) is the troll review, he will submit a file named 483829.txt that contains the following 3 lines:
David S. Pumpkins
483829
#49B
Since some write-ups received less than 3 reviews so it is possible that some of you did not receive a troll review for their write-up. If you believe that you didn't get a troll review, you can use NONE for the <TROLL_REVIEW_ID>.
Feel free to contact us if you have any questions - or post them to the Discussion forum if you think your classmates might have the same question.
Submission deadline is Monday, November 1, 2021 @ 23:59.