Table 1 Six dimensions of the reviewing quality, their indicators, and example references
Quality Dimension | Indicators | Example References |
|---|---|---|
Reviewing process | Not a late review | Xiong & Schunn, 2021 |
Not a speeded review | ||
# of revisions to comments | ||
Rating accuracy | Expert agreement | |
Peer agreement | ||
Expert consistency/Validity | Tong et al., 2023 | |
Peer consistency/Intra-rater reliability | Zhang et al., 2020 | |
Amount | # Reviews | Zou et al., 2018 |
# Comments | Tan & Chen, 2022 | |
# of long comments (> 50 words) | Patchan et al., 2018 | |
% long comments (> 50 words) | Zong et al., 2021b | |
Total/mean length of comments | Howard et al., 2010 | |
Perceived comment quality | Feelings of comfort when evaluated | Raes et al., 2013 |
Understanding comments | Nelson & Schunn, 2009 | |
Agreement with comments | Cheng & Hou, 2015 | |
Perceived comment helpfulness | Rietsche et al., 2022 | |
Willingness to improve | Huisman et al., 2018 | |
Actual comment quality | Implementable | Cui et al., 2021 |
Processed | Wichmann et al., 2018 | |
Identification | Wu & Schunn, 2021b | |
Explanation | Leijen, 2017 | |
Suggestion/Solution | Cheng & Hou, 2015 | |
Evaluation | ||
Summary | van den Bos & Tan, 2019 | |
Localization | Patchan et al., 2018 | |
Mitigating language | Wu & Schunn, 2020a | |
Balanced tone | Ramachandran et al., 2017 | |
Feedback content | Review relevance to rubric | Darvishi et al., 2022 |
Review coverage of a submission | Ramachandran et al., 2017 | |
Address important issues in document | Gao et al., 2019 | |
Global problems or solutions | Patchan et al., 2018 | |
Focused on higher-order writing issues | Gao et al., 2019 | |
Not plagiarized | Ramachandran et al., 2017 | |
# peers referred to the same problems | Leijen, 2017 |