Abstract:This paper constructs question answering system for bridge design specification based on large language model. Three implementation schemes are tried: full fine-tuning of the Bert pretrained model, parameter-efficient fine-tuning of the Bert pretrained model, and self-built language model from scratch. Through the self-built question and answer task dataset, based on the tensorflow and keras deep learning platform framework, the model is constructed and trained to predict the start position and end position of the answer in the bridge design specification given by the user. The experimental results show that full fine-tuning of the Bert pretrained model achieves 100% accuracy in the training-dataset, validation-dataset and test-dataset, and the system can extract the answers from the bridge design specification given by the user to answer various questions of the user; While parameter-efficient fine-tuning of the Bert pretrained model and self-built language model from scratch perform well in the training-dataset, their generalization ability in the test-dataset needs to be improved. The research of this paper provides a useful reference for the development of question answering system in professional field.

What problem does this paper attempt to address?

The paper aims to construct a question-answering system for bridge design specifications based on large-scale language models and explores three different implementation schemes: fully fine-tuning a pre-trained BERT model, parameter-efficient fine-tuning of the BERT pre-trained model, and building a language model from scratch. Specifically: 1. **Research Background**: In professional work, people often need to query information from literature. Traditional keyword search techniques have issues such as information loss, excessive returned information, and irrelevant information. Traditional question-answering systems based on structured data or frequently asked questions (FAQ) rely on rules, templates, or a limited set of question-answer pairs, making it difficult to meet complex and variable user needs, and they have high development and maintenance costs. In recent years, the development of large-scale language models has brought new vitality to question-answering systems based on natural language processing (NLP), promising to become the next generation of search engines. 2. **Research Content**: - Constructed a question-answering task dataset for bridge design specifications. - Proposed three implementation schemes: - Fully fine-tuning a pre-trained BERT model. - Parameter-efficient fine-tuning of the BERT pre-trained model. - Building a language model from scratch. 3. **Experimental Results**: - The fully fine-tuned pre-trained BERT model achieved 100% accuracy on the training set, validation set, and test set, showing excellent performance. - The parameter-efficient fine-tuned BERT pre-trained model and the language model built from scratch performed well on the training set but need improvement in generalization ability on the test set. 4. **Conclusion**: This study not only provides an efficient question-answering tool for the field of bridge design but also offers valuable references and methodologies for building question-answering systems in other professional fields. Although the parameter-efficient fine-tuning and the language model built from scratch face challenges in generalization performance on the test set, expanding the dataset size and optimizing the model structure can improve the model's generalization ability.

Question answering system of bridge design specification based on large language model

Bridge Relation Extraction: New Chinese Dataset and Model

BERT and hierarchical cross attention-based question answering over bridge inspection knowledge graph

Complex Knowledge Base Question Answering for Intelligent Bridge Management Based on Multi-Task Learning and Cross-Task Constraints

A Chinese Intelligent Question Answering System Based on Domain Ontology and Sentence Templates

Building a Question Answering System for the Manufacturing Domain

Generalizing Question Answering System with Pre-trained Language Model Fine-tuning

A medical question answering system using large language models and knowledge graphs

LB-KBQA: Large-language-model and BERT based Knowledge-Based Question and Answering System

Research on Intelligent Question-Answering Systems Based on Large Language Models and Knowledge Graphs

Leveraging Large Language Models in Code Question Answering: Baselines and Issues

Multitask Fine Tuning on Pretrained Language Model for Retrieval-Based Question Answering in Automotive Domain

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

GPB and BAC: two novel models towards building an intelligent motor fault maintenance question answering system

Bridge inspection named entity recognition via BERT and lexicon augmented machine reading comprehension neural model

Bridging Anaphora Resolution as Question Answering

Intelligent question and answer system for building information modeling and artificial intelligence of things based on the bidirectional encoder representations from transformers model

Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models

BB-KBQA: BERT-Based Knowledge Base Question Answering

LSTM Based Question Answering for Large Scale Knowledge Base

Discriminate and Reconstruct: Learning from Language Model to Answer Keyword Questions