A Diagnostic Framework for Compositional Generalization via NL-to-FOL Translation

Creative Commons License

Deveci İ. E.

Journal of Logic, Language and Information, 2026 (SCI-Expanded, AHCI, Scopus)

  • Publication Type: Article / Article
  • Publication Date: 2026
  • Doi Number: 10.1007/s10849-026-09480-0
  • Journal Name: Journal of Logic, Language and Information
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Arts and Humanities Citation Index (AHCI), Scopus, IBZ Online, Applied Science & Technology Source, INSPEC, Linguistic Bibliography, MathSciNet, MLA - Modern Language Association Database, Philosopher's Index, zbMATH, Philosopher's Index full-text, MLA International Bibliography, Engineering Source (EBSCO)
  • Keywords: Compositionality, Large Language Models, Neural Machine Translation
  • Open Archive Collection: AVESIS Open Access Collection
  • Middle East Technical University Affiliated: Yes

Abstract

With the emergence of large language models and their impressive performance across diverse natural language processing tasks, the question of whether connectionist models can exhibit compositionality without relying on symbolic processing has regained attention in both cognitive science and artificial intelligence. However, interpretability challenges faced by neural networks make it difficult to determine whether they genuinely generalize compositional structures. In this paper, we introduce a targeted evaluation framework designed to directly assess the ability of transformer-based language models to translate natural language sentences into first-order logic expressions, a task that requires both nuanced linguistic understanding and compositional generalization. To demonstrate our framework, we fine-tune two different sizes of the T5 language model using our dataset, evaluating their performance through three experiments that employ four task-specific evaluation metrics. Our findings reveal that while these models achieve high scores on test data sharing the logical and structural complexity of the training set, their performance drops markedly as sentence length, the number of truth-functional connectives and predicates, and the depth of hierarchical composition increase. More strikingly, the models fail to generalize even when complexity increases solely through repeated applications of a single truth-functional connective.