Abstract:Construction remains one of the most hazardous sectors. Recent advancements in AI, particularly Large Language Models (LLMs), offer promising opportunities for enhancing workplace safety. However, responsible integration of LLMs requires systematic evaluation, as deploying them without understanding their capabilities and limitations risks generating inaccurate information, fostering misplaced confidence, and compromising worker safety. This study evaluates the performance of two widely used LLMs, GPT-3.5 and GPT-4o, across three standardized exams administered by the Board of Certified Safety Professionals (BCSP). Using 385 questions spanning seven safety knowledge areas, the study analyzes the models' accuracy, consistency, and reliability. Results show that both models consistently exceed the BCSP benchmark, with GPT-4o achieving an accuracy rate of 84.6% and GPT-3.5 reaching 73.8%. Both models demonstrate strengths in safety management systems and hazard identification and control, but exhibit weaknesses in science, mathematics, emergency response, and fire prevention. An error analysis identifies four primary limitations affecting LLM performance: lack of knowledge, reasoning flaws, memory issues, and calculation errors. Our study also highlights the impact of prompt engineering strategies, with variations in accuracy reaching 13.5% for GPT-3.5 and 7.9% for GPT-4o. However, no single prompt configuration proves universally effective. This research advances knowledge in three ways: by identifying areas where LLMs can support safety practices and where human oversight remains essential, by offering practical insights into improving LLM implementation through prompt engineering, and by providing evidence-based direction for future research and development. These contributions support the responsible integration of AI in construction safety management toward achieving zero injuries.

Hazard analysis in the era of AI: Assessing the usefulness of ChatGPT4 in STPA hazard analysis

Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT

From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

Can Large Language Models assist in Hazard Analysis?

Exploring Opportunities in Usable Hazard Analysis Processes for AI Engineering

Hazard analysis: Application of STPA to ship-to-ship transfer of LNG

Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis

AI‐PSM: Where are we now?

STPA for Learning-Enabled Systems: A Survey and A New Practice

Using STPA in Compliance with ISO 26262 for Developing a Safe Architecture for Fully Automated Vehicles

Can ChatGPT exceed humans in construction project risk management?

How good are large language models at product risk assessment?

Human-in-the-Loop AI Reviewing: Feasibility, Opportunities, and Risks

Anticipating Accidents through Reasoned Simulation

Hazard Analysis and Risk Assessment for an Automated Unmanned Protective Vehicle

GPT-4 and Safety Case Generation: An Exploratory Analysis

Welcome Your New AI Teammate: On Safety Analysis by Leashing Large Language Models

Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering

The Digital Divide in Process Safety: Quantitative Risk Analysis of Human-AI Collaboration

Using systems thinking-based risk assessment methods to assess hazardous manual tasks: a comparison of Net-HARMS, EAST-BL, FRAM and STPA

Enhancing Autonomous Driving Safety Analysis with Generative AI: A Comparative Study on Automated Hazard and Risk Assessment