29/05/2026
Ever wondered whether a language model could complete a cloze test (a fill-in-the-blank test), and if so, how its performance would compare with that of humans?
On 12 May, Susan Lotz, researcher, language practitioner and content coordinator at the SU Language Centre, presented a paper asking the same question. She attended the 2026 Language Resources and Evaluation Conference (LREC) in Palma de Mallorca, Spain, and two of its co-located workshops: READIxTSAR, which dealt with readability research, and the 7th Workshop on Resources for African Indigenous Languages (RAIL), where she presented a paper.
The research is part of Susanโs joint PhD with Stellenbosch University in the Department of General Linguistics and the University of Groningen in the Center for Language and Cognition Groningen.
โMy co-authors were Rik van Noord and Gertjan van Noord, and we found that of al the encoder-only language models that we experimented with, the performance of a small language model trained on Afrikaans only, afRoBERTA, correlated best with the performance of the humans who completed the same cloze tests. We also established that the moderate correlation was not accidental, that the humans and the language model found the same word classes difficult, and that, where both the language model and humans gave wrong cloze answers, the language model was further off with its answers than the aggregated humans.โ
โWhy did we even investigate this? Weโre working towards finding a new means of readability assessment for Afrikaans โ this is also my PhD research topic. Cloze tests can be used to establish how readable a text is, so if we could find a language model that could serve as a reliable proxy for human participants, it could make a very resource intensive task easier. More work needs to be done before a language model can be used in this way, but the experiments were really interesting!โ The paper is available in the RAIL@LREC proceedings:http://lrec-conf.org/proceedings/lrec2026/workshops/rail/2026.rail-1.0.pdf
Attending such a large international conference was a bit daunting, but the spirit of the conference was one of connection and sharing. It had a multilingual focus in the sense that many smaller (often lower resourced) languages were represented in the research shared, not only the usual prominent ones. It was also interesting to see how Mallorca balances communication with its multilingual population: most notices are available in Spanish and Catalan as well as in English, and sometimes even in German, as Mallorca is a popular destination for particularly German tourists.
There wasnโt much time for sightseeing, but Susan did manage to experience the Gothic Cathedral of Santa Maria of Palma, which was magnificent, and visit a little mountain town, Valdemossa, with amazing mountain views. Valdemossa is also home to the 14th-century complex that was first a palace and later a Carthusian monastery. Chopin spent a winter there in 1838/9, composing some of his best work. After the conference she visited Barcelona for two days for a โGaudรญ immersionโ, as she put it.