Abstract:Transformer-based large language models have achieved remarkable performance across various natural language processing tasks. However, they often struggle with seemingly easy tasks like arithmetic despite their vast capabilities. This stark disparity raise human's concerns about their safe and ethical use, hinder their widespread <a class="link-external link-http" href="http://adoption.In" rel="external noopener nofollow">this http URL</a> this paper, we focus on a typical arithmetic task, integer multiplication, to explore and explain the imperfection of transformers in this domain. We provide comprehensive analysis of a vanilla transformer trained to perform n-digit integer multiplication. Our observations indicate that the model decomposes multiplication task into multiple parallel subtasks, sequentially optimizing each subtask for each digit to complete the final multiplication. Based on observation and analysis, we infer the reasons of transformers deficiencies in multiplication tasks lies in their difficulty in calculating successive carryovers and caching intermediate results, and confirmed this inference through experiments. Guided by these findings, we propose improvements to enhance transformers performance on multiplication tasks. These enhancements are validated through rigorous testing and mathematical modeling, not only enhance transformer's interpretability, but also improve its performance, e.g., we achieve over 99.9% accuracy on 5-digit integer multiplication with a tiny transformer, outperform LLMs GPT-4. Our method contributes to the broader fields of model understanding and interpretability, paving the way for analyzing more complex tasks and Transformer models. This work underscores the importance of explainable AI, helping to build trust in large language models and promoting their adoption in critical applications.

Mechanistic Interpretability of Binary and Ternary Transformers

Binary and Ternary Natural Language Generation

Unifying and Verifying Mechanistic Interpretations: A Case Study with Group Operations

Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars

How Transformers Solve Propositional Logic Problems: A Mechanistic Analysis

Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models

Mechanistically Interpreting a Transformer-based 2-SAT Solver: An Axiomatic Approach

A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task

Mechanistic interpretability of large language models with applications to the financial services industry

Transformers with Competitive Ensembles of Independent Mechanisms.

Evaluating Brain-Inspired Modular Training in Automated Circuit Discovery for Mechanistic Interpretability

A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models

Learning Transformer Programs

Representations as Language: An Information-Theoretic Framework for Interpretability

Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks

Modular addition without black-boxes: Compressing explanations of MLPs that compute numerical integration

Dissecting Multiplication in Transformers: Insights into LLMs

Transcoders Find Interpretable LLM Feature Circuits

Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability

Investigating the Limitations of Transformers with Simple Arithmetic Tasks

Extracting Finite State Machines from Transformers