Large language models for sustainable assessment and feedback in higher education: Towards a Pedagogical and Technological Framework

Students need to gain, during their learning process, the assessment expertise, so the competence required for students to effectively understand assessment criteria and to be able to use the feedback received to close the gap and improve their own learning [51]. They will be supported in this process by teachers who, through continuous assessment and making judgements about the learning products of students, will develop more effective standards of judgement to define the expected competence of students themselves [16 p.76, 43]. But how to really support students’ development of assessment expertise? As Sadler pointed out [51], it is necessary to involve students in direct and authentic assessment experiences, supporting them in the acquisition of the concept of quality, and training them in order to make complex judgements according to a multiplicity of criteria [51]. What are the principles to apply in order to create sustainable assessment contexts and then scaffold authentic assessment experiences? Boud [4] drew up nine useful principles for reflecting on and designing sustainable assessment and feedback practices. For the author, it is indeed important that there is a timely sharing of clear assessment criteria for students; it is also crucial that students are seen as individuals who can achieve success and, in terms of assessment processes, these must be useful in making students confident in their success and in this sense it seems useful to consider separating feedback processes from the awarding of grades. The focus on learning during the assessment process must take priority over the focus on performance: it seems important therefore to support the development of self-assessment competencies and encourage the use of reflective peer assessment practices. One of the last fundamental aspects is related to the completion of the feedback loop as a tool for reviewing student work and finally, the importance of introducing a review process of assessment practices with the implementation of formative assessment processes is emphasised.

In terms of assessment and feedback in connection with AI systems, Swiecki and colleagues affirmed that

“AI-based techniques have been developed to fully or partially automate parts of the traditional assessment practice. AI can generate assessment tasks, find appropriate peers to grade work, and automatically score student work. These techniques offload tasks from humans to AI and help to make assessment practices more feasible to maintain” [56, p.2].

The power of AI related to the assessment and feedback processes is connected to the fact that, while traditional assessment practices could provide a partial overview about the students performance, several AI techniques thanks to their characteristics can promote a wider vision of learning process and progress. In relation to the topic of authenticity and sustainability of assessment, AI systems can help to collect, represent, and assess data in a complex way: authentic assessment processes can be very articulated in terms of task and general design, so AI can be helpful for academics to monitor learning process towards an assessment of student progress [41, 56]. In fact, authentic assessment requires students to “use the same competencies, or combinations of knowledge, skills, and attitudes, that they need to apply in the criterion situation in professional life” [17 p.69]. Authenticity has been recognised as a fundamental element of assessment design that encourages learning. Authentic assessment tries to reproduce the activities and performance criteria often encountered in the workplace, and has been shown to have a favourable influence on student learning, autonomy, motivation, self-regulation, and metacognition; qualities that are significantly associated with employability [64]. Again, international authors even suggest that the authenticity of the assessment tasks is a need for reaching the expert level of problem-solving. Likewise, strengthening the authenticity of an assessment has the potential to have an encouraging effect on student learning and motivation [17, 53].

Finally, UNESCO [38], which has been in the last years amongst the most influential and active institutions that reflect on the implications of AI in society, provided the following guidelines for AI in assessment.This study moves along those research axes.

Language Models (LMs) are defined as “computational models that have the capability to understand and generate human language. LMs have the transformative ability to predict the likelihood of word sequences or generate new text based on a given input” [6, p.39]. A language model is considered “large” (LLM) when it is scaled on the computational resources required, the number of parameters, the size of the dataset used for training [19, 27] and arguably, the capabilities.

Nowadays, the most common LLMs use the transformer architecture: "a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output [...] The Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence aligned RNNs or convolution” [63 p.2]. In fact, they defined the transformer architecture, introducing “the self-attention mechanism to determine the relevance of different parts of the input when generating predictions. This allows the model to better understand the relationships between words in a sentence, regardless of their position” [29 p.1, 63].

These models are able to learn, to infer the context, and to produce text by mapping relationships (such as influence and dependance) in series of elements, even when they are far from each other in the sequence (for example, between words that are not consecutive in a sentence). Improved parallelisation, reduced training time, and better benchmarks in several tasks compared to the once prevalent recursive and convolutional neural networks [63] have been crucial to its success.

Over the past few years, Large Language Models (LLMs) have become increasingly prevalent in society and educational settings. These AI-powered models are capable of generating, analysing, and summarising text, as well as engaging in dialogic interactions with humans [29]. One of the most well-known examples of LLMs is OpenAI’s ChatGPT (where GPT stands for Generative Pre-trained Transformer), which is based on GPT 3.5 and GPT 4 architectures. Other notable LLMs include Anthropic’s Claude (1, 2 and 3), Microsoft Copilot (another GPT4-based model), Mistral AI’s Mistral Large, and Google Gemini. While these models are extremely powerful, there are concerns about data privacy and results consistency [7]. However, there are other options available. With the release of open-source and open-access models such as Meta’s LLAMA (at the moment at its third iteration), Google Gemma, as well as TII’s Falcon (by the Technology Innovation Institute in Abu Dhabi), Mistral AI’s Mixtral 8x7B and the growth of platforms like HuggingFace, which acts as a repository and framework, there are many possibilities for local LLMs with great capabilities. These models can be customised, fine-tuned, or even trained specifically for one’s use case, allowing for greater flexibility and control [36].

To better understand the possibilities related to the use of these models, we list a series of crucial LLMs’ capabilities for education from the work of various authors:

LLMs are able to analyse massive amounts of text, aggregate it, and then offer feedback based on previously established standards [57]. The outcomes of that analysis can be applied to provide feedback to the student as well as to assist the instructor in evaluating the text. With the correct pedagogical and technical framework, LLMs can support teachers in some of the time-related sustainability difficulties, and thus break them free from the sole choice of the multiple-choice test and similar. In detail, Kasneci and colleagues [29] define the following opportunities for teachers and students regarding the implementation of AI in the teaching and learning universitycontext:

In specific relation to assessment and feedback practice, the correlation with LLM can be summarised in four different points:

This paper introduces the idea of "AI in the Constructivist Loop", a framework that integrates AI into the processes of teaching, learning and acquisition of knowledge and competences. At the core of this discussion are four concepts: technological mediators, augmented intelligence, supermind, and AI as a boundary object. This framework is based on the theories of Vygotsky [65] and Jonassen [25] regarding technological mediation, further developed through Malone’s supermind [34] and the practical use of AI as a boundary object within Activity Theory.

The constructivist theory, which highlights learning as a process of constructing knowledge in context rather than passively acquiring it, is also based on the idea that humans developed various technologies (artifacts, tools and instruments) to enhance their capabilities: both physical and cognitive. Language is one of those tools, arguably one of the most important. One can see how the creation of AIs, and especially LLMs, fits with the idea of using them as tools for expanding cognitive capability. Such tools serve as technological mediators (or mediating artifacts) between humans and the environment (including the physical environment but also abstract objects and objectives, the community and desired outcomes in general). Vygotsky’s Zone of Proximal Development (ZPD), consists of an area that separates things that learners can do autonomously and things that they can’t do at all [65]. That is the area of things that learners can do with some assistance or support, and that, eventually, they will become able to do unaided. Tools such as LLMs can help expand one’s capacity to navigate ZPD and consolidate knowledge and skills by offering support fostering critical thinking skills, and providing learners with appropriate challenges and assistance. Such kind of technological mediators are therefore considered “mindtools” [25] by acting as textcolorgreenaids and scaffolding elements, thereby bridging the gap between abstract theoretical ideas and practical understanding. This paradigm shift starts from learning technology, to learning through technology, to learning with technology.

The concept of an augmented intelligence is a well-established one. Initially referred to as “intelligence amplification”, it encapsulates the idea of merging human and machine intelligence to enhance, rather than substitute, human’s intellect. At its beginning, in the 50s the idea was still the one to use it as an amplifying tool [2], the phase, in pedagogical terms, was that of learning “through” technology, in the same way as using a ladder to reach a higher spot. The actual paradigm of learning “with” technology, influenced by the constructionist ideas of Papert & Harel [47], is nearer to the current view of augmented intelligence. It is a human-centred partnership model applicable to decision-making, research, learning and planning, where the human is in the loop with the AI [12, 71].

Human-in-the-loop concept is an extensive area of research that covers the intersection of computer science, cognitive science, and psychology [68] and can be considered a semi-supervised human-computer interaction learning method [40], which aims at achieving the accuracy of machine learning and assisting human learning. It implements the high efficiency and accuracy of AI in decision-making through scene recognition, high-quality human annotation, and active learning.

In this augmented environment Malone’s [35] idea of the supermind becomes especially pertinent. Superminds are not necessarily AI-blended, in fact, the idea is that human organisations that act together with an intelligent behaviour are superminds. For example, hierarchies, markets, communities an democracies fall inside that definition. However, focussing on the the AI possible contribution to a hybrid supermind, it would result in a blend of human and machine intelligence that leverages the strengths of both to achieve superior results. In educational settings, this translates into collaborative networks where students, educators and AI collaborate in solving problems and constructing knowledge together. These networks not only improve learning outcomes but also foster a distributed cognitive system that aligns with the constructivist belief that knowledge is socially and contextually constructed.

A practical way of viewing AI in this educational approach is possible through the lens of the Activity Theory framework [13, 28]. In this context AI can be viewed as a boundary object [23], practically, a bridge connecting individuals and groups within the education community such as students, teachers and policymakers. Its role in facilitating discussions and knowledge sharing reflects the idea that learning is a social process influenced by diverse viewpoints.

The framework of "AI in the constructivist loop" presents an approach to incorporating AI in education. By treating AI as a tool for collaboration embracing intelligence tapping into collective intelligence and recognising its role as a connecting factor this framework supports and enhances teaching methods based on constructivism. It not only paves the way for personalised and interactive learning experiences, but also lays the groundwork for future investigations into how AI can revolutionise educational practices further.

Given the growing interest and preliminary insights offered by existing international literature on the intersection of Artificial Intelligence (AI) and education (AIED), this study seeks to contribute by developing and validating a comprehensive model for integrating AI technologies in educational assessment within higher education contexts. This endeavour follows the cycle of action research and is grounded in literature and practice evidence-based design. The validation approach is the validation through comparison, that involves juxtaposing the outcomes, processes, and perceptions associated with the implementation of the new model against those derived both from more traditional or previously established methods and from other implementations of the new model. By conducting a systematic comparison, researchers can identify the relative strengths and weaknesses of the latest model and its impact on critical issues such as student learning outcomes, engagement, and satisfaction.

This study and model design work is based on one main research question:

The designed model takes into account the existing literature connected to the topic of Assessment for Learning, Authentic Assessment and Sustainable Assessment [4, 52]. Starting from these literature pieces of evidence, we are working on the development of a model for adopting AI in the assessment processes in the Higher Education context. The model takes into account the role that AI plays in the assessment and feedback practices connected to academics and students in the virtuous cycle of the learning spiral (Table 2).

The model itself will be assessed following four different levels proposed by Kaptelinin and colleagues [28] through the Activity Theory:

Every checklist is developed following four different areas:

The model will be composed by two layers of adoption:

In order to promote a first step of validation about the model, that is called AI-MAAS (AI-Mediated Assessment Academics and Students), it was decided to activate a comparison process of the model proposed according to the existing scientific literature, reports and official documents concerning the topic at international level. Starting from a more general overview also in terms of policies and institutional guidance, it is important to cite the following works:

Starting from the recent literature concerning the role and the power of AI in Education, an important study to cite is one of Kamalov and colleagues [26], who explored impacts related to the integration of AI in educational systems, in terms of benefits, challenges and application in the four spheres of (1) Personalised Learning, (2) Intelligent Tutoring Systems (ITS), (3) Assessment Automation, and (4) Teacher–Student Collaboration. Through their work, they propose an overview of practical implication about AI introduction: in fact, they assume that “by personalising learning experiences, automating administrative responsibilities, and delivering real-time feedback, AI is revolutionising the educational landscape, bridging gaps, and encouraging a more inclusive and effective learning environment. Given the importance of integrating AI in education, there is a need to reflect on its implications” [36, p.1]. The area 3 “Assessment Automation” gives us the opportunity to explore benefits and challenges connected to the implementation of AI in assessment processes and allow us to reflect on the concrete implications related to our model structure and application. Other studies reflect on the AI application also in the assessment area, like Swiecki and colleagues [56], who analyse the power and the possible applications of AI systems, assume that “ AI-based techniques have been developed to fully or partially auto-mate parts of the traditional assessment practice. AI can generate assessment tasks, find appropriate peers to grade work, and automati-cally score student work. These techniques offload tasks from humans to AI and help to make assessment practices more feasible to maintain”. In their work, they analysed the domain of Automated assessment construction; AI-assisted peer assessment; Writing analytics; the role of Electronic assessment platforms; the topic of Stealth assessment; the issue of Latent knowledge estimation; the implications related to the Learning processes; the power of AI of scaffolding the transition From inauthentic to authentic and From antiquated to modern and all the related challenges about AI intersection with educational assessment design and experiences both for academics and students. These two papers equipped us with a general, critical and depth analysis of the implications related to AI and assessment and thus the authors sustain the validation process in terms of principles and scientific evidence. In relation with the more concrete level of applications of AI in assessment, to sustain our model it is possible to explore and cite different international reports as:

These scientific products, already consolidated and recognised by the academic community, represent comparative models of design, implementation and reflection with respect to the AI-MAAS model, which, through a further process of formal validation by means of a Delphi study and thanks to direct classroom experimentation, could represent a valid tool to support the development of innovative and contemporary assessment processes.

AI can serve as a pivotal mediating artefact in summative assessment, bridging academics, students, and the assessment process with students’ work. By embedding specific AI parameters, defined by academics, to guide rubrics and assessment criteria, AI has the potential to enhance the efficiency and relevance of feedback and assessment. This model underscores the interaction between academics and AI in crafting early feedback mechanisms and refining assessment processes, including academics’ contributions to AI finetuning for continuous improvement.

The relationship between AI and students is centred on the assessment of students’ products, where AI plays a crucial role in providing early and timely feedback. This facilitates a more dynamic and responsive assessment cycle, allowing for self-assessment, adjustments and improvements in students’ work based on AI-generated insights. This brings a formative value to the summative assessment.

Moreover, the academic-student interaction is enriched through mediated assessment, where AI supports the creation of assessment materials and step-by-step evaluation processes.

The synergy among these components— academics’ expertise in defining assessment criteria, AI’s capacity to process and assess student products, and the mediated interaction between academics and students— creates a robust framework for summative assessment. This model not only accelerates the feedback loop but also ensures that assessments are more aligned with learning objectives and student needs, ultimately enhancing the educational experience and learning outcomes.

Here are presented three pieces of literature that bring research evidence supporting the structure of AI-MAAS/S model. The first one is specifically designed to act as a support and validation evidence for the model, while the second and the third fit with it and are brought as pieces of evidence as well.

Among the various experimentations that are being conducted to validate and support the AI-MAAS model, one of the most critical for the summative assessment level is about the capability of LLMs to assess students’ written products [1]. This study investigates which of the most advanced models are capable of following a rubric of assessment and evaluating as closely as possible to human expert evaluators. The products to evaluate were written educational intervention designs created by 88 students of a support teachers specilisation program. Following a rubric is of great importance for a coherent and authentic assessment, especially when assessing authentic assignments. This study involves the following axes and elements of the AI-MAAS model:

Findings of this study are that OpenAi’s ChatGPT-4 and Anthropic’s Claude 2 were the closest match to an expert human assessment. Moreover, on several occasions, they have been closer to a human evaluator than the other human evaluator. The average accuracy to human evaluation for ChatGPT-4 was 80%, and for Claude 2 was 79% (which is the same accuracy the two human expert evaluators had between them).

Only on the moist complex criteria do humans result in being more consistent between them than LLMs. Criteria such as “quality of assessment design” are something that generalist LLMs cannot yet assess properly. This emphasises the importance of oversight to ensure the reliability and educational alignment of LLMs.

Even better results on this front have been reached by Martin and colleagues [36], who worked on the summative assessment, starting from the need to assign reasoning, conceptualisation and processing tasks, and the fact that correcting large quantities of open-ended responses and tasks often proves unsustainable. They worked on the axes:

They employed machine learning (ML) to create and refine a holistic scoring rubric for evaluating students’ argument complexity when judging the plausibility of competing chemical reactions. In this case, a close match between human scores and the LLM scores was achieved, with an accuracy between 85% and 90%.

Martin and colleagues used a complex approach, which went beyond using a standard "Chat" LLM to achieve their desired outcome. They employed HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) to cluster student topics, map the results onto structures, and manually assign scores to all topics based on the rubric, creating a labeled dataset. They then compared the performance of various Bidirectional Encoder Representations from Transformers [11] pre-trained language models (BERT, RoBERTa, SciBERT) in classifying topics into the 20 rubric categories and discovered that BERT Large Uncased had the best results. Lastly, they trained a deep neural network classifier with the labeled dataset to automate the scoring of new topics based on the rubric. They validated the model using techniques such as generating artificial topics.

Models that are trained on a specific task and population are responsible for producing this excellent result. As such, it should be noted that the procedure currently in use cannot be expected to be utilised by any teacher who is not specialised in Machine Learning.

A third example from Koraishi [31] addresses the use of ChatGPT for teaching and assessing English Foreign Language (EFL) classes. His practice is along the AI-MAAS axes:

In such article, various practical applications of ChatGPT are presented and commented on, particularly concerning assessment and feedback. The use cases discussed include support for LLM in creating tests, generating feedback, personalising assessments, and producing reports. The model’s strongest affordances, which lie within the English language domain, have been wisely utilised by the author. Also this author, notes, however, that the entire process should be supervised and not rely solely on LLM evaluation.

AI could be used in an assessment formative way taking into account the virtuous connection between academics, students and AI itself. Starting from the introduction of specific train parameters for AI as academics useful to guide formative assessment experience, it could be possible to create a complex monitoring system to maximise the students’ learning experience. In fact, beginning from the direct action of the university professors whom act as experts in terms of definition of roles, criteria and rules for AI, it is possible to develop a system that could help and customise the collection process of the data produced by AI in order to monitor and redesign the teaching, learning and assessment process. In terms of students, they could participate in the prompting process of AI, in collaboration with academics: in fact, they could co-construct rules and criteria to guide the formative assessment process, in order to improve the timely feedback action produced by AI in direct connection with their work. This process might represent a valid way to strengthen the connection between the three actors of the educational practice and at the same time it can foster comparison and self assessment mechanisms in both students and teachers. Then AI as student and students as teachers in the sharing actions of construction and delivery of assessment and feedback prompts. These practices can support the development of dynamic processes of empowerment of assessment practice, not only in terms of the outputs produced by the AI, but also make the moments of university teachers’ and students’ assessment sustainable and cooperative. At the same time, these mechanisms can foster the development of critical evaluative competences in learners and accompanying academics in a critical revision of their practice and their own parameters of judgement, in order to make assessment participatory, shared and effective. These actions are placed in the domain of the model concerning educational formative assessment, going to propose a complete connection between the various elements of the model and of the educational process in general. Formative assessment, in fact, is a tool for monitoring and sequentially support the learning experience and this example can support the accompaniment of students, promoting at the same time cross-cutting competences and integrating important concepts in Higher Education with student partnership [10] in the teaching design and therefore a global approach to teaching, learning and assessment student centred [67].