Augmented Large Language Models with Parametric Knowledge Guiding

Ziyang Luo,Can Xu,Pu Zhao,Xiubo Geng,Chongyang Tao,Jing Ma,Qingwei Lin,Daxin Jiang

DOI: https://doi.org/10.48550/arXiv.2305.04757

2023-05-18

Abstract:Large Language Models (LLMs) have significantly advanced natural language processing (NLP) with their impressive language understanding and generation capabilities. However, their performance may be suboptimal for domain-specific tasks that require specialized knowledge due to limited exposure to the related data. Additionally, the lack of transparency of most state-of-the-art (SOTA) LLMs, which can only be accessed via APIs, impedes further fine-tuning with domain custom data. Moreover, providing private data to the LLMs' owner leads to data privacy problems. To address these challenges, we propose the novel Parametric Knowledge Guiding (PKG) framework, which equips LLMs with a knowledge-guiding module to access relevant knowledge without altering the LLMs' parameters. Our PKG is based on open-source "white-box" language models, allowing offline memory of any knowledge that LLMs require. We demonstrate that our PKG framework can enhance the performance of "black-box" LLMs on a range of domain knowledge-intensive tasks that require factual (+7.9%), tabular (+11.9%), medical (+3.0%), and multimodal (+8.1%) knowledge.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on the poor performance of large - language models (LLMs) in domain - specific tasks. Specifically, although large - language models perform well in general natural - language - processing tasks, they may perform poorly in tasks requiring specialized knowledge for the following reasons:

1. **Limited exposure to specialized knowledge**: Due to the limitations of training data, large - language models may not have sufficient relevant - domain knowledge or vocabulary, which leads to their unsatisfactory performance in domain - specific tasks.
2. **Transparency issues of black - box models**: Many state - of - the - art large - language models can only be accessed via APIs. This lack of transparency hinders the possibility of further fine - tuning with domain - specific data.
3. **Data privacy issues**: Providing private data to the owners of large - language models may lead to data - privacy risks, such as security threats like data misuse and leakage.

To solve these problems, the author proposes a new framework - Parameter - Knowledge - Guided (PKG). This framework enables large - language models to access relevant knowledge without changing their parameters by adding a knowledge - guided module. This method not only addresses the above challenges but also significantly improves the performance of large - language models in multiple tasks requiring domain knowledge, including fact - knowledge, table - knowledge, medical - knowledge, and multi - modal - knowledge tasks.