General information

The Priority Area (German: Schwerpunktprogramm) “Robust Assessment & Safe Applicability of Language Modeling: Foundations for a New Field of Language Science & Technology” (acronym: LaSTing; SPP 2556) aims to advance our understanding of language technology, in particular language modeling, for safer use, especially in applications in the (computational / cognitive) language sciences. A detailed project description is here.

Aims and scope of the Priority Area LaSTing

While modern language technology increasingly permeates many areas of applications, much of its input-output behaviour and its inner mechanics remains unknown. As a result, recent years have seen a newly emerging field of interdisciplinary and methodologically diverse work at the interface between the cognitive language sciences and language technology.

Robust assessment

Given the very rapid pace of recent developments, careful reflection on standards for the methodology of testing and assessment is lagging behind. What is required is a joint effort to converge on proper standards for robust assessment of language models. Methodology is robust, in the sense intended here, if its results are generalisable (carrying over with sufficient certainty to other models and data sets), transferable (insightful beyond the purposes of understanding a single type of computational model), and reproducible (with the same or different models and data sets). Robust methodology also aspires to be as future-proof as possible, i.e. likely relevant to the next generation of models or the next set of antagonistic examples.

Safe applicability

As language technology gets applied more and more widely, concerns of safe applicability become ever more important. Safe applicability subsumes critical aspects such as being conceptually sound (e.g. anchored in “first principles” or established empirical knowledge), validated (e.g. by mathematical proof or other rigorous derivation) or at least stress-tested across a near-exhaustive traversal of possible conditions of use, ethical (e.g. bias- and harm-free, or privacy-respecting), and also economical (i.e. minimising data requirements and energy consumption). Issues of safe applicability loom particularly large in the context of high-stake implications, of which application in the scientific process is a special case. The Priority Area LaSTing therefore also particularly invites contributions on the reflection of safe applicability of language technology for knowledge gain in the cognitive language sciences.

Foundational questions

Progress on understanding the behaviour of language models and their safe applicability is inexorably tied to a better understanding of their core mechanisms and the impact of their training data or their training objectives. But just as relevant are deep foundational questions concerning the nature of language models (e.g. what are LMs models of?) and their proper role in the scientific research into human language (e.g. how could LMs be used as explanatory tools for understanding human language?). In response to these issues, the Priority Programme especially welcomes foundational work addressing general properties or potential limits of particular classes of language models, e.g. by using mathematical arguments, simulation studies, tight conceptual argumentation or a mixture of such methods.

Projects

The Priority Area LaSTing involves the following projects:

Limits and Biases in Machine and Human Language and its Learning (PIs: Artemis Alexiadou, Uli Sauerland)
The pragmatic test: how humans and LLMs decode presupposed meaning (PIs: Nadine Bade, Miriam Butt)
KIND-LM: Cognitively-inspired interaction dynamics for sample-efficient language modeling (PIs: Lisa Beinborn, Nivedita Mani)
A Resource Efficient Cross-linguistic Approach to Figurative Meaning Assessment in LLMs (PI: Maria Berger)
Unreal engines — Understanding language models through resource-optimal analysis: Implicit Bayesian pragmatic reasoning & emergent causal world models (PI: Michael Franke)
Attention in Large Language Models: Linguistic Grounding, Cognitive Modeling, and Social Application (PIs: Nicole Gotzner, Sebastian Musslick)
A multidimensional adaptive test for the psychometric assessment of LLM capabilities (PI: Fritz Günther)
Structural generalization in transformer-based LLMs (PIs: Michael Hahn, Alexander Koller)
Systemic Robustness Assessments of Language Models for Cross-Linguistic Research using Formally Related Structures (FORESTS) (PIs: Jutta-Maria Hartmann, Anke Himmelreich, Sina Zarrieß)
Moral Hallucinations in Large Language Models — Their Argumentative Structure and Ethical Implications (PIs: Annette Hautli-Janisz, Karoline Reinhardt)
The Status of Linguistic Constraints in Neural Language Models (PI: Erhard Hinrichs)
Evaluating, Explaining, and Enabling Ethical Multi-Agent Systems of Large Language Models (E4-MALM) (PIs: Anne Lauscher, Jae Hee Lee)
Interpretable Surprisal: Language Models Between Linguistic Structure and Neural Evidence (PIs: Alessandro Lopopolo, Milena Rabovsky)
Understanding the Cross-Linguistic Brain Basis of Sentence Processing through Interpretable Language Technology (PI: Lars Meyer)
Propositional Attitudes in Large Language Models (PALLM) (PI: Robert Pasternak)
Relating Probabilities of Words to Probabilities of Worlds (PI: Sean Papay)
Learning linguistic inferences and their alternatives (PIs: Jacopo Romoli, Yulia Zinova)
LLADIGA: Learning Language with Dialogue Games (PIs: David Schlangen, Raffaella Bernardi)
The evaluation of empathy-related linguistic performance in large language models: Comparing surprisal values for next-word predictions in human EEG and LLMs (PI: Markus Werning)
Gesture-Informed Language Models: Evaluating Multimodal Discourse Processing in LLMs and Humans (PI: Frances Yung)

Mercator Fellows

The Priority Area LaSTing is supported by four Mercator Fellows:

Katrin Erk: Katrin Erk is Professor at the Linguistics and Computer Science departments at the University of Massachusetts Amhers and an internationally renowned expert on computational semantics. Her work dives deep into the theoretical foundations of distributional (embedding-based) representations of meaning and compositionality
Raquel Fernández: Raquel Fernandez is Professor of Computational Linguistics & Dialogue Systems at the University of Amsterdam. She is well-known for her widely influential work on computational and data-driven approaches to dialogue modeling. Her current research investigates modern language technology from a theoretically-informed position that combines factors of individual cognition and grounding in situated interaction
Roger Levy: Roger Levy is Professor at the MIT Department of Brain and Cognitive Sciences. His seminal work in computational psycholinguistics combines (language) modeling of large data sets with experimental linguistics, increasing our understanding oflanguage processing in both machines and humans.
Christopher Potts: Christopher Potts is Professor and Chair at the Department of Linguistics at Stanford University, also associated there with the Department of Computer Science. While his early work made ground-breaking contributions to formal semantics and pragmatics, his more recent work is bridging linguistics and language technology with exemplary work on theoretically-informed NLP applications and (causal) interpretability of language models

Contact

Spokesperson
Prof. Michael Franke
michael.franke@uni-tuebingen.de

Scientific Coordination
Miriam Schiele
miriam.schiele@uni-tuebingen.de

Administrative Coordination
Fidan Can
fidan.can@semsprach.uni-tuebingen.de