RETUYT-INCO at BEA 2026 Shared Task 2: Meta-prompting in Rubric-based Scoring for German
Authors: Ignacio Sastre, Ignacio Remersaro, Facundo D'iaz et al.
Summary
The authors describe their system for automatically scoring German student short answers using rubrics. Their main contribution is "Meta-prompting": an LLM generates a custom prompt based on training examples, which is then used to grade new answers. They also tested classic ML, LLM fine-tuning, and other prompting techniques, achieving middle-tier rankings (4th-6th place out of 8-9 teams) across three shared task tracks.
Main takeaways:
- Task involves scoring short German student answers against specific rubrics.
- Meta-prompting: use an LLM to create a specialized prompt from training examples, then use that prompt to grade new answers.
- Team placed 6th/8 in Track 1 (QWK 0.729), 4th/9 in Track 3 (QWK 0.674), and 4th/8 in Track 4 (QWK 0.49).
- Also explored classic machine learning, fine-tuning open-source LLMs, and various prompting strategies.
- Focus was on handling the varying nature of rubrics and questions across the dataset.
Relevance
Not directly related to my persona installation or conditional behavior work. Meta-prompting (using an LLM to generate a task-specific prompt) is loosely connected to my question about finding prompts corresponding to fine-tuning/steering vectors, but this paper is about educational assessment, not behavioral installation.
Abstract
arXiv:2605.11242v1 Announce Type: new Abstract: In this paper, we present the RETUYT-INCO participation at the BEA 2026 shared task "Rubric-based Short Answer Scoring for German". Our team participated in track 1 (Unseen answers three-way), track 3 (Unseen answers two-way) and track 4 (Unseen questions two-way). Since these tracks required scoring short student answers using specific rubrics, we looked for ways to handle the changing nature of the task. We created a method called Meta-prompting. In this approach, an LLM creates a custom prompt based on examples from the Train set. This prompt is then used to grade new student answers. Along with this method, we also describe other approaches we used, such as classic machine learning, fine-tuning open-source LLMs, and different prompting techniques. According to the official results, our team placed 6th out of 8 participants in Track 1 with a QWK of 0.729. In Track 3, we secured 4th place out of 9 with a QWK of 0.674, and we also placed 4th out of 8 in Track 4 with a QWK of 0.49.