Improving Discrete Diffusion Models via Structured Preferential Generation

Severi Rissanen,Markus Heinonen,Arno Solin

2024-05-28

Abstract:In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset. This work paves the way for more advances in discrete diffusion models with potentially significant enhancements in performance.

Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the issue of improving the performance of discrete diffusion models on discrete data types such as text. Specifically, compared to autoregressive generative models, existing discrete diffusion models perform poorly when handling discrete data like text. The authors propose a structured forward process that leverages the inherent hierarchical information within discrete categories to enhance generation performance. Through this method, certain categories are generated before others during the generation process, significantly improving the log-likelihood score on the text8 dataset. This work paves the way for the development of discrete diffusion models and could lead to significant performance improvements.