Table 2 Specific feedback quality measures within each of the five dimensions, including definition, the basis of the data, and data source if derived from prior studies
Dimension | Measure | Definition | Basis | Data Source |
|---|---|---|---|---|
Reviewing process | Not speeded | The percentage of normal minutes (> 10 min) between requesting a paper for review and submitting the review | Computer | Xiong & Schunn, 2021 |
Rating accuracy | Expert agreement | Mean agreement with expert ratings across rating dimensions & documents | Hand | Wu and Schunn (2021a) |
Peer agreement | Mean agreement with mean peer ratings across rating dimensions & documents | Computer | Â | |
Expert consistency | Linear correlation with expert ratings across rating dimensions & documents | Hand | Â | |
Peer consistency | Linear correlation with mean peer ratings across rating dimensions & documents | Computer | Â | |
Amount | #Comments | Mean number of comments per review | Computer | Â |
#Reviews | Total number of documents reviewed | Computer | Â | |
%Long comments | The percentage of comments that are long (#words > 50) | Computer |  | |
Perceived comment quality | Helpfulness | Mean helpfulness rating given to provided comments | Computer | Â |
Priority | Mean priority for revision given to provided comments | Computer | Â | |
Actual comment quality | %Implementable | The percentage of comments that can be incorporated in a revision | Hand | Wu and Schunn (2021b) |
%Implemented | The percentage of comments that were addressed in the submitted revision | Hand | Wu and Schunn (2020a) | |
Improvement | Mean expected impact of addressing each comment (2 = large, 1 = small, 0 = none) | Hand | Wu and Schunn (2020b) | |
%Solutions | The percentage of provided comments that include a specific bit of replacement text | Hand | Wu and Schunn (2020a) | |
%Suggestions | The percentage of provided comments that include general revision advice but not a solution | Hand | ||
%Identifications | The percentage of provided comments that describe what is problematic | Hand | ||
%Explanations | The percentage of provided comments that give an explanation for a problem | Hand | ||
#Features | The mean number of features in a comment (ident., expl., solu., sugg.) | Hand |