Abstract:Despite remarkable progress in multi-modal AI research, there is a salient domain in which modern AI continues to lag considerably behind even human children: the reliable deployment of logical operators. Here, we examine three forms of logical operators: relations, negations, and discrete numbers. We asked human respondents (N=178 in total) to evaluate images generated by a state-of-the-art image-generating AI (DALL-E 3) prompted with these `logical probes', and find that none reliably produce human agreement scores greater than 50\%. The negation probes and numbers (beyond 3) fail most frequently. In a 4th experiment, we assess a `grounded diffusion' pipeline that leverages targeted prompt engineering and structured intermediate representations for greater compositional control, but find its performance is judged even worse than that of DALL-E 3 across prompts. To provide further clarity on potential sources of success and failure in these text-to-image systems, we supplement our 4 core experiments with multiple auxiliary analyses and schematic diagrams, directly quantifying, for example, the relationship between the N-gram frequency of relational prompts and the average match to generated images; the success rates for 3 different prompt modification strategies in the rendering of negation prompts; and the scalar variability / ratio dependence (`approximate numeracy') of prompts involving integers. We conclude by discussing the limitations inherent to `grounded' multimodal learning systems whose grounding relies heavily on vector-based semantics (e.g. DALL-E 3), or under-specified syntactical constraints (e.g. `grounded diffusion'), and propose minimal modifications (inspired by development, based in imagery) that could help to bridge the lingering compositional gap between scale and structure. All data and code is available at <a class="link-external link-https" href="https://github.com/ColinConwell/T2I-Probology" rel="external noopener nofollow">this https URL</a>

Illustrating a neural model of logic computations: The case of Sherlock Holmes' old maxim

Transfinite Modal Logic: a Semi-quantitative Explanation for Bayesian Reasoning

Models of Possibilities Instead of Logic as the Basis of Human Reasoning

A Simple Generative Model of Logical Reasoning and Statistical Learning

A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

Towards Unifying Perceptual Reasoning and Logical Reasoning

The Holmesian logician: Sherlock Holmes’ “Science of Deduction and Analysis” and the logic of discovery

Interpretable Multimodal Misinformation Detection with Logic Reasoning

A Logical Approach to Criminal Case Investigation

Relations, Negations, and Numbers: Looking for Logic in Generative Text-to-Image Models

Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models

Reliable Reasoning Beyond Natural Language

Inference of Abstraction for a Unified Account of Symbolic Reasoning from Data

Proof of Thought : Neurosymbolic Program Synthesis allows Robust and Interpretable Reasoning

Positive Logic with Adjoint Modalities: Proof Theory, Semantics and Reasoning about Information

Conditional and Modal Reasoning in Large Language Models

Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models

Reasoning From Quantified Modal Premises

Neural Natural Logic Inference for Interpretable Question Answering.

Neural networks and logical reasoning systems: a translation table