Abstract:Transformer-based encoder-decoder models have demonstrated impressive results in chemical reaction prediction tasks. However, these models typically rely on pretraining using tens of millions of unlabelled molecules, which can be time-consuming and GPU-intensive. One of the central questions we aim to answer in this work is: Can FlanT5 and ByT5, the encode-decoder models pretrained solely on language data, be effectively specialised for organic reaction prediction through task-specific fine-tuning? We conduct a systematic empirical study on several key issues of the process, including tokenisation, the impact of (SMILES-oriented) pretraining, fine-tuning sample efficiency, and decoding algorithms at inference. Our key findings indicate that although being pretrained only on language tasks, FlanT5 and ByT5 provide a solid foundation to fine-tune for reaction prediction, and thus become 'chemistry domain compatible' in the process. This suggests that GPU-intensive and expensive pretraining on a large dataset of unlabelled molecules may be useful yet not essential to leverage the power of language models for chemistry. All our models achieve comparable Top-1 and Top-5 accuracy although some variation across different models does exist. Notably, tokenisation and vocabulary trimming slightly affect final performance but can speed up training and inference; The most efficient greedy decoding strategy is very competitive while only marginal gains can be achieved from more sophisticated decoding algorithms. In summary, we evaluate FlanT5 and ByT5 across several dimensions and benchmark their impact on organic reaction prediction, which may guide more effective use of these state-of-the-art language models for chemistry-related tasks in the future.

A Chemical Language Model for Molecular Taste Prediction

A Chemical Language Model for Multi-Class Molecular Taste Prediction

FlavorMiner: A Machine Learning Platform for Extracting Molecular Flavor Profiles from Structural Data

Predicting and improving complex beer flavor through machine learning

FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists

Flavor Identification Based on Olfactory-Taste Synesthesia Model and Hybrid Convolutional Neural Network-Random Forest

Predicting human olfactory perception from chemical features of odor molecules

Computational Screening of Umami Tastants Using Deep Learning

Predicting Multiple Taste Sensations with a Multiobjective Machine Learning Method

A Reinforcement Learning Framework to Discover Natural Flavor Molecules

A Proposed Artificial Neural Network based Approach for Molecules Bitter Prediction

Classification-based machine learning approaches to predict the taste of molecules: A review

Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction

Design of virtual libraries of umami-tasting molecules

A Foundation Model for Chemical Design and Property Prediction

Analyzing Learned Molecular Representations for Property Prediction

Taste Bud-Inspired Single-Drop Multitaste Sensing for Comprehensive Flavor Analysis with Deep Learning Algorithms

A Large Encoder-Decoder Family of Foundation Models For Chemical Language

FlavorDB: a database of flavor molecules