Abstract:Tensor networks are efficient for extremely high-dimensional representation, but their model selection, known as tensor network structure search (TN-SS), is a challenging problem. Although several works have targeted TN-SS, most existing algorithms are manually crafted heuristics with poor performance, suffering from the curse of dimensionality and local convergence. In this work, we jump out of the box, studying how to harness large language models (LLMs) to automatically discover new TN-SS algorithms, replacing the involvement of human experts. By observing how human experts innovate in research, we model their common workflow and propose an automatic algorithm discovery framework called tnGPS. The proposed framework is an elaborate prompting pipeline that instruct LLMs to generate new TN-SS algorithms through iterative refinement and enhancement. The experimental results demonstrate that the algorithms discovered by tnGPS exhibit superior performance in benchmarks compared to the current state-of-the-art methods.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges in Tensor Network Structure Search (TN - SS). Specifically, although Tensor Networks (TNs) are very effective in dealing with extremely high - dimensional representations, choosing the optimal tensor network structure (i.e., TN - SS) is a very difficult problem. Most of the existing TN - SS algorithms are heuristics designed manually, and these methods have the problems of the curse of dimensionality and local convergence, resulting in poor performance. In addition, different downstream tasks require different balance points, and the existing heuristic strategies are difficult to find a suitable balance between exploration and exploitation. To overcome these problems, the paper proposes a new framework - tnGPS, which uses Large Language Models (LLMs) to automatically discover new TN - SS algorithms, thereby reducing the dependence on human experts and improving the performance of the algorithms. ### Main Contributions 1. **Proposing the tnGPS framework**: This is an automated framework based on LLMs, aiming to automatically generate novel and effective TN - SS algorithms for specific downstream tasks. 2. **Experimental result verification**: The experimental results show that the algorithms discovered by tnGPS are not only superior to the existing TN - SS algorithms on the benchmark data, but also perform well on out - of - domain data. ### Method Overview The design of the tnGPS framework is inspired by the research and innovation process of human experts, and mainly includes the following stages: 1. **Idea Pool**: Collect existing TN - SS algorithms and their performance scores. 2. **Knowledge Categorisation**: Use LLMs to cluster the algorithms, and each cluster represents a method for solving TN - SS. 3. **Idea Dropout**: Randomly screen out the most interesting ideas for in - depth study through the roulette selection mechanism. 4. **Knowledge Recombination**: Combine the advantages of existing algorithms to generate new algorithms. 5. **Incremental Innovation**: Make small improvements to existing algorithms. 6. **Diversity Injection**: Generate new algorithms that are methodologically different from existing algorithms to increase the diversity of the algorithm pool. ### Experimental Verification The paper verifies the effectiveness of tnGPS through several benchmark tests. One of the experiments is natural image compression. In this experiment, tnGPS can discover new TN - SS algorithms, which show better performance than existing methods when compressing natural images. ### Conclusion The tnGPS framework successfully automates the discovery of new TN - SS algorithms by leveraging the full - domain knowledge and reasoning ability of LLMs, significantly improves the performance of the algorithms, and reduces the dependence on human experts. This achievement provides new directions and tools for future TN - SS research.

tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs)

Alternating Local Enumeration (TnALE): Solving Tensor Network Structure Search with Fewer Evaluations

Bayesian tensor network structure search and its application to tensor completion

Adaptive Learning of Tensor Network Structures

SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective

Large Language Models for the Automated Analysis of Optimization Algorithms

TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents

Enabling Tensor Language Model to Assist in Generating High-Performance Tensor Programs for Deep Learning.

Design Principle Transfer in Neural Architecture Search via Large Language Models

TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage

ControlLLM: Augment Language Models with Tools by Searching on Graphs

Large Language Models on Graphs: A Comprehensive Survey

A Momentum-Incorporated Non-Negative Latent Factorization of Tensors Model for Dynamic Network Representation

Symmetric Tensor Networks for Generative Modeling and Constrained Combinatorial Optimization

Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representation

ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph

LLaGA: Large Language and Graph Assistant

TnT-LLM: Text Mining at Scale with Large Language Models

Test-Time Training on Graphs with Large Language Models (LLMs)

Search for Efficient Large Language Models

Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training