Enabling peer review in large University courses

Abstract

In the Austrian university education, we are often faced with large classes of 600+ students. While this situation cannot be changed due to university policies and financing, we still aim at providing the best possible teaching and learning experience for such a setting. We highly value constructivist learning, which is driven by the need for personalised feedback throughout the individual learning process. Due to the high students numbers, we lack the resources to connect and interact with every single student, however, an alternative is to encourage a reciprocal community, in which students take charge of their own learning and support each other on their journey through peer review. However, we found that the basic peer review process, as used in scientific settings or Massive Open Online Courses, did not fully meet our expectations when introduced in a first year lecture. We found that students were hesitant in providing critical feedback about their peers work; that the extra step of writing reviews often had a negative impact on their motivation to continue their own work; that they were discontent with the extra workload and that they did not fully trust their peers perspective on their work. Hence, we saw the need to redesign and adapt the basic peer review system to better fit our students needs. This thesis reports on our process and findings from using an exploratory design approach to implement peer review in the bespoke learning platform Aurora over the span of 6 years to answer the research question: "What do we need to consider when designing peer review for large university courses to meet students needs and make the process more sustainable to organise and perpetuate?". Since we are dealing with an ill-defined problem existing within a bigger technological learning environment, we took an exploratory design approach. We introduced a basic version of peer review into the lecture, observed how it was received throughout the semester and conducted a survey with students at the end of the semester towards gaining a better understanding of the context and its issues. This pointed us towards three main challenges when using peer review in large university courses: the varying quality of reviews, conveying a feeling of fairness, and keeping up student motivation in a self-directed learning environment. Based on these findings we iteratively created design artefacts to explore possible solutions to these challenges. In this work, we give an overview of such recurring design challenges and our response in the form of influential design artefacts that impacted peer review activities. Furthermore, we expand on our research practices and reflect on challenges arising from the use of design-based research and co-creation methods in our long-term, real-life context. This work contributes to the field of technology enhanced learning and teaching in that it reflects on design aspects necessary to create a better learning environment for students in the context of peer review in large university courses, such as better guiding students through the semester, visibility of every action, transparency of the whole process, or avoiding ranking amongst students. While these are aspects drawn from work in our particular context, our findings also lead us to the formulation of five transferable design considerations, suggesting to plan for scaffolding', 'contextualisation', 'framing and mindset', 'value' and 'transparency' when designing technology for educational contexts.