Abstract:Cyber-physical systems (CPSs) integrate cyber and physical components and enable them to interact with each other to meet user needs. The needs for CPSs span rich application domains such as healthcare and medicine, smart home, smart building, etc. This indicates that CPSs are all about solving real-world problems. With the increasing abundance of sensing devices and effectors, the problems wanted to solve with CPSs are becoming more and more complex. It is also becoming increasingly difficult to extract and express CPS requirements accurately. Problem frame approach aims to shape real-world problems by capturing the characteristics and interconnections of components, where the problem diagram is central to expressing the requirements. CPSs requirements are generally presented in domain-specific documents that are normally expressed in natural language. There is currently no effective way to extract problem diagrams from natural language documents. CPSs requirements extraction and modeling are generally done manually, which is time-consuming, labor-intensive, and error-prone. Large language models (LLMs) have shown excellent performance in natural language understanding. It can be interesting to explore the abilities of LLMs to understand domain-specific documents and identify modeling elements, which this paper is working on. To achieve this goal, we first formulate two tasks (i.e., entity recognition and interaction extraction) and propose a benchmark called CPSBench. Based on this benchmark, extensive experiments are conducted to evaluate the abilities and limitations of seven advanced LLMs. We find some interesting insights. Finally, we establish a taxonomy of LLMs hallucinations in CPSs requirements modeling using problem diagrams. These results will inspire research on the use of LLMs for automated CPSs requirements modeling.

An Evaluation of Requirements Modeling for Cyber-Physical Systems via LLMs

Analyzing Early Requirements of Cyber-physical Systems Through Structure and Goal Modeling

The Ten Lockheed Martin Cyber-Physical Challenges: Formalized, Analyzed, and Explained

Model-based Construction and Verification of Cyber-Physical Systems.

An Extensible Use Case Modeling Approach for Cyber-Physical Systems (Cpss).

Tackling visual and conceptual complexity of problem-oriented modeling of requirements

Building Device Models Using LLMs for Requirements Simulation in Smart Homes

Integration of LLMs and the Physical World: Research and Application

Analyzing Software Requirements for Cyber Physical Systems

CPS-LLM: Large Language Model based Safe Usage Plan Generator for Human-in-the-Loop Human-in-the-Plant Cyber-Physical System

RE4CPS: Requirements Engineering for Cyber-Physical Systems.

Integrating Goal Models and Problem Frames for Requirements Analysis of Self-adaptive CPS

CPSDBench: A Large Language Model Evaluation Benchmark and Baseline for Chinese Public Security Domain

CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity

Transforming Timing Requirements into CCSL Constraints to Verify Cyber-Physical Systems.

Ollabench: Evaluating LLMs' Reasoning for Human-centric Interdependent Cybersecurity

PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation

Exploring Advanced Methodologies in Security Evaluation for LLMs

Testing learning-enabled cyber-physical systems with Large-Language Models: A Formal Approach

Penetrative AI: Making LLMs Comprehend the Physical World

Large Language Model Integrated Healthcare Cyber-Physical Systems Architecture