Abstract:The advancement of Large Language Models (LLMs) for domain applications in fields such as materials science and engineering depends on the development of fine-tuning strategies that adapt models for specialized, technical capabilities. In this work, we explore the effects of Continued Pretraining (CPT), Supervised Fine-Tuning (SFT), and various preference-based optimization approaches, including Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO), on fine-tuned LLM performance. Our analysis shows how these strategies influence model outcomes and reveals that the merging of multiple fine-tuned models can lead to the emergence of capabilities that surpass the individual contributions of the parent models. We find that model merging leads to new functionalities that neither parent model could achieve alone, leading to improved performance in domain-specific assessments. Experiments with different model architectures are presented, including Llama 3.1 8B and Mistral 7B models, where similar behaviors are observed. Exploring whether the results hold also for much smaller models, we use a tiny LLM with 1.7 billion parameters and show that very small LLMs do not necessarily feature emergent capabilities under model merging, suggesting that model scaling may be a key component. In open-ended yet consistent chat conversations between a human and AI models, our assessment reveals detailed insights into how different model variants perform and show that the smallest model achieves a high intelligence score across key criteria including reasoning depth, creativity, clarity, and quantitative precision. Other experiments include the development of image generation prompts based on disparate biological material design concepts, to create new microstructures, architectural concepts, and urban design based on biological materials-inspired construction principles.

Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning

Benchmarking Linguistic Diversity of Large Language Models

How Far Can We Extract Diverse Perspectives from Large Language Models?

DiverseDialogue: A Methodology for Designing Chatbots with Human-Like Diversity

Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment

Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning

Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities

Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions

PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization

Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Unlocking Large Language Model's Planning Capabilities with Maximum Diversity Fine-tuning

Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning

Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias

Enhancing LLM-Based Human-Robot Interaction with Nuances for Diversity Awareness

PRIMAL: Prompting Multiple Language Models for Low-resource Diverse Response Generation

Capturing Bias Diversity in LLMs

Evaluating and Mitigating Discrimination in Language Model Decisions

Screening Through a Broad Pool: Towards Better Diversity for Lexically Constrained Text Generation

Exploring Accuracy-Fairness Trade-off in Large Language Models

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models