VSLLaVA: a pipeline of large multimodal foundation model for industrial vibration signal analysis

Qi Li,Jinfeng Huang,Hongliang He,Xinran Zhang,Feibin Zhang,Zhaoye Qin,Fulei Chu

2024-09-03

Abstract:Large multimodal foundation models have been extensively utilized for image recognition tasks guided by instructions, yet there remains a scarcity of domain expertise in industrial vibration signal analysis. This paper presents a pipeline named VSLLaVA that leverages a large language model to integrate expert knowledge for identification of signal parameters and diagnosis of faults. Within this pipeline, we first introduce an expert rule-assisted signal generator. The generator merges signal provided by vibration analysis experts with domain-specific parameter identification and fault diagnosis question-answer pairs to build signal-question-answer triplets. Then we use these triplets to apply low-rank adaptation methods for fine-tuning the linear layers of the Contrastive Language-Image Pretraining (CLIP) and large language model, injecting multimodal signal processing knowledge. Finally, the fine-tuned model is assessed through the combined efforts of large language model and expert rules to evaluate answer accuracy and relevance, which showcases enhanced performance in identifying, analyzing various signal parameters, and diagnosing faults. These enhancements indicate the potential of this pipeline to build a foundational model for future industrial signal analysis and monitoring.

Signal Processing,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper aims to address the lack of domain expertise in large multi-modal foundational models (LMM) for industrial vibration signal analysis. Specifically:

1. **Limitations of LMM in Specific Domains**:
- Although current LMMs perform well in general image recognition tasks, their performance is limited in specialized fields like industrial vibration signal analysis due to the lack of specific signal processing knowledge.
- In the field of industrial Prognostics and Health Management (PHM), signal processing and fault diagnosis are critical tasks, but existing LMMs cannot effectively handle these tasks.

2. **Shortcomings of Traditional Methods**:
- Traditional signal analysis methods such as signal modulation and feature extraction, while effective, require advanced signal processing and machine learning techniques, and each task usually needs a separately designed model.
- There is a lack of a universal interface to handle different signal analysis tasks, leading to complex model design and difficulty in interacting with signal analysis experts.

3. **Bridge Between Language and Signal Semantics**:
- Current methods face challenges in connecting signals with language semantics. Due to the heterogeneity of signal and language representations, establishing this connection directly is very difficult.

To address these issues, the paper proposes the VSLLaVA pipeline, which combines large language models (LLM) with domain expertise to generate signal-question-answer (SQA) triplets. It uses Low-Rank Adaptation (Lora) technology to fine-tune the model, thereby enhancing its performance in signal parameter identification and fault diagnosis.