General information

The Priority Area (German: Schwerpunktprogramm) “Robust Assessment & Safe Applicability of Language Modeling: Foundations for a New Field of Language Science & Technology” (acronym: LaSTing; SPP 2556) aims to advance our understanding of language technology, in particular language modeling, for safer use, especially in applications in the (computational / cognitive) language sciences. A detailed project description is here.

Projects

The Priority Area “LaSTing” involves the following projects:

• Limits and Biases in Machine and Human Language and its Learning (PIs: Artemis Alexiadou, Uli Sauerland)
• The pragmatic test: how humans and LLMs decode presupposed meaning (PIs: Nadine Bade, Miriam Butt)
• KIND-LM: Cognitively-inspired interaction dynamics for sample-efficient language modeling (PIs: Lisa Beinborn, Nivedita Mani)
• A Resource Efficient Cross-linguistic Approach to Figurative Meaning Assessment in LLMs (PI: Maria Berger)
• Unreal engines — Understanding language models through resource-optimal analysis: Implicit Bayesian pragmatic reasoning & emergent causal world models (PI: Michael Franke)
• Attention in Large Language Models: Linguistic Grounding, Cognitive Modeling, and Social Application (PIs: Nicole Gotzner, Sebastian Musslick)
• A multidimensional adaptive test for the psychometric assessment of LLM capabilities (PI: Fritz Günther)
• Structural generalization in transformer-based LLMs (PIs: Michael Hahn, Alexander Koller)
• Systemic Robustness Assessments of Language Models for Cross-Linguistic Research using Formally Related Structures (FORESTS) (PIs: Jutta-Maria Hartmann, Anke Himmelreich, Sina Zarrieß)
• Moral Hallucinations in Large Language Models — Their Argumentative Structure and Ethical Implications (PIs: Annette Hautli-Janisz, Karoline Reinhardt)
• The Status of Linguistic Constraints in Neural Language Models (PI: Erhard Hinrichs)
• Evaluating, Explaining, and Enabling Ethical Multi-Agent Systems of Large Language Models (E4-MALM) (PIs: Anne Lauscher, Jae Hee Lee)
• Interpretable Surprisal: Language Models Between Linguistic Structure and Neural Evidence (PIs: Alessandro Lopopolo, Milena Rabovsky)
• Understanding the Cross-Linguistic Brain Basis of Sentence Processing through Interpretable Language Technology (PI: Lars Meyer)
• Propositional Attitudes in Large Language Models (PALLM) (PI: Robert Pasternak)
• Relating Probabilities of Words to Probabilities of Worlds (PI: Sean Papay)
• Learning linguistic inferences and their alternatives (PIs: Jacopo Romoli, Yulia Zinova)
• LLADIGA: Learning Language with Dialogue Games (PIs: David Schlangen, Raffaella Bernardi)
• The evaluation of empathy-related linguistic performance in large language models: Comparing surprisal values for next-word predictions in human EEG and LLMs (PI: Markus Werning)
• Gesture-Informed Language Models: Evaluating Multimodal Discourse Processing in LLMs and Humans (PI: Frances Yung)

Mercator Fellows

The Priority Area “LaSTing” is supported by four Mercator Fellows:

• Katrin Erk: Katrin Erk is a professor in the Linguistics and Computer Science departments at the University of Massachusetts Amhers and an internationally renowned expert on computational semantics. Her work dives deep into the theoretical foundations of distributional (embedding-based) representations of meaning and compositionality
• Raquel Fernández: Raquel Fernandez is Professor of Computational Linguistics & Dialogue Systems at the University of Amsterdam. She is well-known for her widely influential work on computational and data-driven approaches to dialogue modeling. Her current research investigates modern language technology from a theoretically-informed position that combines factors of individual cognition and grounding in situated interaction
• Roger Levy: Roger Levy is Professor at the MIT Department of Brain and Cognitive Sciences. His seminal work in computational psycholinguistics combines (language) modeling of large data sets with experimental linguistics, increasing our understanding oflanguage processing in both machines and humans.
• Christopher Potts: Christopher Potts is Professor and Chair at the Department of Linguistics at Stanford University, also associated there with the Department of Computer Science. While his early work made ground-breaking contributions to formal semantics and pragmatics, his more recent work is bridging linguistics and language technology with exemplary work on theoretically-informed NLP applications and (causal) interpretability of language models

Aims and scope of the Priority Area “LaSTing”

While modern language technology increasingly permeates many areas of applications, much of its input-output behaviour and its inner mechanics remains unknown. As a result, recent years have seen a newly emerging field of interdisciplinary and methodologically diverse work at the interface between the cognitive language sciences (broadly construed) and language technology (focused on neural language models, but not exclusively). However, many foundational and methodological issues remain unclear. The overarching goal of this Priority Programme is therefore to channel cross-disciplinary efforts dedicated to the understanding, testing and safe application of modern language technology (with a focus on language modelling).

The Priority Programme LaSTing addresses researchers in the interdisciplinary field of the cognitive and computational language sciences (including classical disciplines such as linguistics, psychology, neuroscience, computational linguistics, artificial intelligence, philosophy, computer science and others) who seek to advance our understanding of language modelling from a theoretical or empirical point of view, or use modern language technology as a tool for innovative theoretical and empirical research in the cognitive language sciences. Individual projects are expected to relate to at least one of the Priority Area’s core issues, which are robust assessment, safe applicability and foundational questions (as detailed in the following). The Priority Programme especially encourages contributions that seek to address these core issues by bringing to bear concepts and methods from the theoretical/empirical language sciences.

Robust assessment Given the very rapid pace of recent developments, careful reflection on standards for the methodology of testing and assessment is lagging behind. What is required is a joint effort to converge on proper standards for robust assessment of language models. Methodology is robust, in the sense intended here, if its results are generalisable (carrying over with sufficient certainty to other models and data sets), transferable (insightful beyond the purposes of understanding a single type of computational model), and reproducible (with the same or different models and data sets). Robust methodology also aspires to be as future-proof as possible, i.e. likely relevant to the next generation of models or the next set of antagonistic examples.

Safe applicability As language technology gets applied more and more widely, concerns of safe applicability become ever more important. Safe applicability subsumes critical aspects such as being conceptually sound (e.g. anchored in “first principles” or established empirical knowledge), validated (e.g. by mathematical proof or other rigorous derivation) or at least stress-tested across a near-exhaustive traversal of possible conditions of use, ethical (e.g. bias- and harm-free, or privacy-respecting), and also economical (i.e. minimising data requirements and energy consumption). Issues of safe applicability loom particularly large in the context of high-stake implications, of which application in the scientific process is a special case. The Priority Area LaSTing therefore also particularly invites contributions on the reflection of safe applicability of language technology for knowledge gain in the cognitive language sciences.

Foundational questions Progress on understanding the behaviour of language models and their safe applicability is inexorably tied to a better understanding of their core mechanisms and the impact of their training data or their training objectives. But just as relevant are deep foundational questions concerning the nature of language models (e.g. what are LMs models of?) and their proper role in the scientific research into human language (e.g. how could LMs be used as explanatory tools for understanding human language?). In response to these issues, the Priority Programme especially welcomes foundational work addressing general properties or potential limits of particular classes of language models, e.g. by using mathematical arguments, simulations studies, tight conceptual argumentation or a mixture of such methods.

Examples of more concrete research questions that fit into these three core issues are:

• Behavioural Assessment: What are adequate, robust methods of experimentally assessing the (abstracted, linguistic) capability of an LM based on its input-output behaviour? What is a valid comparison of machine predictions with human behaviour?
• Representations & Mechanisms: Which information is reliably retrievable from LMs’ latent representations (embeddings) for linguistic/explanatory purposes or for understanding the inner workings of LMs? How can we distil the abstract computational processes that generate an LM’s behaviour?
• Training & Optimisation: How can we understand LMs in terms of their optimisation, e.g. in terms of properties of the training data, their internal inductive biases, the training objective etc.? How does that compare with human language learning?
• Task Decomposition Models: What are best practices for using LMs as part of a larger (theoretically informed) composition of the task to be solved (e.g. in agent models, applications such as RAG, or explanatory, neuro-symbolic (cognitive) models)?
• Resource Efficiency: How can we solve problems of data-hunger and computational costs (training and inference), e.g. by taking human-like inductive biases into account, or using more informative, curated data? How can we use synthetic data and machine judgements to solve theoretical issues?
• Alternative Models: How can language science benefit from alternative models beyond text-to-text LMs, e.g. by embracing multi-modality, interaction, dialogue or more cognitively plausible model architectures?
• Ontological Status: Are LMs models or theories of language? What exactly does an LM predict (occurrence frequencies, behaviour of an idealised speaker, aggregated behaviour of a population of speakers …)?
• Explanatory Potential: How can novel language technology be used as or in support of explanations, e.g. of linguistic phenomena, empirical or experimental data in the language sciences?
• LM capabilities: What are the limits of LM capabilities and why? How can we systematically identify them, also for future generations of language modelling/technology?

Examples of work that is outside the scope of this Priority Area are efforts geared mainly at improving system performance (e.g. based on some benchmark score). Also, projects that merely seek new areas of application with established tools, as long as there is little or no reflection on methods or concepts, or any other bearing on the knowledge-oriented cognitive language sciences.

In order to achieve its goals, LaSTing requires broad and deep interdisciplinary collaboration. The Priority Programme therefore implements an extensive suite of individual measures to support diversity, networking and dissemination, and to ensure the success of early career researchers and scholars with backgrounds underrepresented in academic research.