Metareview assignment
Completion requirements
Opened: Tuesday, 3 November 2020, 02:00
Due: Thursday, 12 November 2020, 00:10
In the final instalment of the peer-reviewing exercises, you will get to experience metareviewing. Metareviewing is the process of assessing reviews and in this assignment, you will assess the quality of the reviews for one of the write-ups. To be able to do this, you need to read and understand: (1) the original write-up and (2) the three reviews that the write-up received last week. You should be able to see your metareviewing assignments in HotCRP. Here are the details:
Metareview content: Your metareview will have two main parts. In the first part, you will give a summary of the three reviews. You will: (1) discuss the key points of the reviews, (2) explain the similarities/differences in the reviewers' assessments of the write-up, (3) discuss the quality/helpfulness of the reviews (e.g., For instance it might be the case that the first reviewer makes valid points about the shortcomings of a write-up and makes suggestions about how to improve the quality of the review. Whereas, the third reviewer might have criticized the write-up without justifying their reasoning), and (4) suggestions on how the reviewer could improve their reviews. In the second part, your goal is to find which one of the three reviews is the troll review. To do this, you will rank the three reviews in order of their quality and usefulness. For instance, if you are doing a metareview for write-up #1984, you will write: "I think review #1984C is the most helpful/useful/insightful review. I think that review #1984A is the troll review.".
Metareview template: You will notice that the metareview template is almost the same as the review template, except that there is a new text field, labeled "Metareview". This is where you are going to put your metareview, which we explained above. Although you do not need the fields from the peer-review round (i.e., insight, clarity, key ideas..), we could not find a way to remove them from the template. Since HotCRP will likely not allow you to submit your metareview without filling out these fields, pick two random scores and put in a single character for the "Key ideas, insights, and critique" field.
This will conclude the metareviewing exercise. Now there is one last thing you need to do: try to find which one of the three reviews that you received for your write-up is the troll review. To submit your guess, you will NOT use HotCRP. Instead, you are going to submit a file here on Moodle. Your file is going to have three lines:
<YOUR_NAME>
<YOUR_SCIPER>
<TROLL_REVIEW_ID>
and it MUST be named <YOUR_SCIPER>.txt
So if David S. Pumpkins (SCIPER 483829), who submitted paper #49, thinks that the second review he received for his write-up is the troll review, he will submit a file named 483829.txt that contains:
David S. Pumpkins
483829
#49B
Feel free to contact us if you have any questions - or post them to the Discussion forum if you think your classmates might have the same question. Submission deadline is Wednesday, November 11, 2020 @11:59:59pm CET.
Metareview content: Your metareview will have two main parts. In the first part, you will give a summary of the three reviews. You will: (1) discuss the key points of the reviews, (2) explain the similarities/differences in the reviewers' assessments of the write-up, (3) discuss the quality/helpfulness of the reviews (e.g., For instance it might be the case that the first reviewer makes valid points about the shortcomings of a write-up and makes suggestions about how to improve the quality of the review. Whereas, the third reviewer might have criticized the write-up without justifying their reasoning), and (4) suggestions on how the reviewer could improve their reviews. In the second part, your goal is to find which one of the three reviews is the troll review. To do this, you will rank the three reviews in order of their quality and usefulness. For instance, if you are doing a metareview for write-up #1984, you will write: "I think review #1984C is the most helpful/useful/insightful review. I think that review #1984A is the troll review.".
Metareview template: You will notice that the metareview template is almost the same as the review template, except that there is a new text field, labeled "Metareview". This is where you are going to put your metareview, which we explained above. Although you do not need the fields from the peer-review round (i.e., insight, clarity, key ideas..), we could not find a way to remove them from the template. Since HotCRP will likely not allow you to submit your metareview without filling out these fields, pick two random scores and put in a single character for the "Key ideas, insights, and critique" field.
This will conclude the metareviewing exercise. Now there is one last thing you need to do: try to find which one of the three reviews that you received for your write-up is the troll review. To submit your guess, you will NOT use HotCRP. Instead, you are going to submit a file here on Moodle. Your file is going to have three lines:
<YOUR_NAME>
<YOUR_SCIPER>
<TROLL_REVIEW_ID>
and it MUST be named <YOUR_SCIPER>.txt
So if David S. Pumpkins (SCIPER 483829), who submitted paper #49, thinks that the second review he received for his write-up is the troll review, he will submit a file named 483829.txt that contains:
David S. Pumpkins
483829
#49B
Feel free to contact us if you have any questions - or post them to the Discussion forum if you think your classmates might have the same question. Submission deadline is Wednesday, November 11, 2020 @11:59:59pm CET.
