Author: Shaib, Chantal; Suriyakumar, Vinith M.; Sagun, Levent; Wallace, Byron C.; Ghassemi, Marzyeh
Description: For an LLM to correctly respond to an instruction it must understand both the semantics and the domain (i.e., subject area) of a given task-instruction pair. However, syntax can also convey implicit information. Recent work shows that syntactic templates — frequent sequences of Part-of-Speech (PoS) tags — are prevalent in training data and often appear in model outputs. In this work we characterize syntactic templates, domain, and semantics in task-instruction pairs. We identify cases of spurious correlations between syntax and domain, where models learn to associate a domain with syntax during training; this can sometimes override prompt semantics. Using a synthetic training dataset, we find that the syntactic-domain correlation can lower performance (mean 0.51 +/- 0.06) on entity knowledge tasks in OLMo-2 models (1B-13B). We introduce an evaluation framework to detect this phenomenon in trained models, and show that it occurs on a subset of the FlanV2 dataset in open (OLMo-2-7B; Llama-4-Maverick), and closed (GPT-4o) models. Finally, we present a case study on the implications for safety finetuning, showing that unintended syntactic-domain correlations can be used to bypass refusals in OLMo-2-7B Instruct and GPT-4o. Our findings highlight two needs: (1) to explicitly test for syntactic-domain correlations, and (2) to ensure syntactic diversity in training data, specifically within domains, to prevent such spurious correlations.
Subject headings: Computation and Language; Large language models; LLM; Syntax; Domain; Semantics; Syntactic diversity; Training data; Artificial intelligence; AI
Publication year: 2025
Journal or book title: arXiv
Pages: 2509.21155
Find the full text: https://arxiv.org/pdf/2509.21155
Find more like this one (cited by): https://scholar.google.com/scholar?cites=4415021056968916428&as_sdt=1000005&sciodt=0,16&hl=en
Serial number: 4160