Abstract:As an essential task for the architecture, engineering, and construction (AEC) industry, information processing and acquiring from unstructured textual data based on natural language processing (NLP) are gaining increasing attention. Although deep learning (DL) models for NLP tasks have been investigated for years, domain-specific pretrained DL models and their advantages are seldomly investigated in the AEC domain. Therefore, this work developed a large scale domain corpora and pretrained domain-specific language models for the AEC domain, and then systematically explores various transfer learning and fine-tuning techniques to explore the performance of pretrained DL models for various NLP tasks. First, both in-domain and close-domain Chinese corpora are developed. Then, two types of pretrained models, including static word embedding models and contextual word embedding models, are pretrained based on various domain corpora. Finally, several widely used DL models for NLP tasks are further trained and tested based on various pretrained models. The result shows that domain corpora can further improve the performance of static word embedding-based DL models and contextual word embedding-based DL models in text classification (TC) and named entity recognition (NER) tasks. Meanwhile, contextual word embedding-based DL models significantly outperform the static word embedding-based DL methods in TC and NER tasks, with maximum improvements of 8.1% and 3.8% in the F1 score, respectively. This research contributes to the body of knowledge in two ways: (1) demonstrating the advantages of domain corpora and pretrained DL models, and (2) opening the first domain-specific dataset and pretrained language models named ARCBERT for the AEC domain. Thus, this work sheds light on the adoption and application of pretrained models in the AEC domain.

Domain-specific language models pre-trained on construction management systems corpora

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains.

Pretrained domain-specific language model for natural language processing tasks in the AEC domain

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Adapting Large Language Models to Domains via Reading Comprehension

A Joint Domain-Specific Pre-Training Method Based on Data Enhancement

TCM-GPT: Efficient Pre-training of Large Language Models for Domain Adaptation in Traditional Chinese Medicine

Domain-oriented Language Pre-training with Adaptive Hybrid Masking and Optimal Transport Alignment

Investigating Pre-trained Language Models on Cross-Domain Datasets, a Step Closer to General AI

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

The Utility of General Domain Transfer Learning for Medical Language Tasks

Structure-aware Domain Knowledge Injection for Large Language Models

Domain-specific Chinese Transformer-XL Language Model with Part-of-speech Information

Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field

Domain Adaptive Remaining Useful Life Prediction with Transformer.

Cross-Domain Sentiment Classification With Bidirectional Contextualized Transformer Language Models

DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis

MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model

From pre-training to fine-tuning: An in-depth analysis of Large Language Models in the biomedical domain

Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain