Abstract:Efficient knowledge management plays a pivotal role in augmenting both the operational efficiency and the innovative capacity of businesses and organizations. By indexing knowledge through vectorization, a variety of knowledge retrieval methods have emerged, significantly enhancing the efficacy of knowledge management systems. Recently, the rapid advancements in generative natural language processing technologies paved the way for generating precise and coherent answers after retrieving relevant documents tailored to user queries. However, for enterprise knowledge bases, assembling extensive training data from scratch for knowledge retrieval and generation is a formidable challenge due to the privacy and security policies of private data, frequently entailing substantial costs. To address the challenge above, in this paper, we propose EKRG, a novel Retrieval-Generation framework based on large language models (LLMs), expertly designed to enable question-answering for Enterprise Knowledge bases with limited annotation costs. Specifically, for the retrieval process, we first introduce an instruction-tuning method using an LLM to generate sufficient document-question pairs for training a knowledge retriever. This method, through carefully designed instructions, efficiently generates diverse questions for enterprise knowledge bases, encompassing both fact-oriented and solution-oriented knowledge. Additionally, we develop a relevance-aware teacher-student learning strategy to further enhance the efficiency of the training process. For the generation process, we propose a novel chain of thought (CoT) based fine-tuning method to empower the LLM-based generator to adeptly respond to user questions using retrieved documents. Finally, extensive experiments on real-world datasets have demonstrated the effectiveness of our proposed framework.

Return of EM: Entity-driven Answer Set Expansion for QA Evaluation

Entity-Relation Extraction As Multi-Turn Question Answering

Leveraging Large Language Models for Multiple Choice Question Answering

EfficientEQA: An Efficient Approach for Open Vocabulary Embodied Question Answering

Evaluating Consistencies in LLM responses through a Semantic Clustering of Question Answering

Unlocking the Power of Large Language Models for Entity Alignment

Leveraging Large Language Models for Entity Matching

Central Answer Modeling for an Embodied Multi-LLM System

Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systems

Exploring Language Model Generalization in Low-Resource Extractive QA

SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM

IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering

BoostER: Leveraging Large Language Models for Enhancing Entity Resolution

Entity Alignment with Noisy Annotations from Large Language Models

Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching

Towards leveraging LLMs for Conditional QA

Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models

Predicting Question-Answering Performance of Large Language Models through Semantic Consistency

EffiQA: Efficient Question-Answering with Strategic Multi-Model Collaboration on Knowledge Graphs

Rethinking Generative Large Language Model Evaluation for Semantic Comprehension

Enhancing Software-Related Information Extraction via Single-Choice Question Answering with Large Language Models