Abstract:Large Language Models for Code (code LLMs) have demonstrated remarkable performance across various software engineering (SE) tasks, increasing the application of code LLMs in software development. Despite the success of code LLMs, there remain significant concerns about the actual capabilities and reliability of these models, "whether these models really learn the semantics of code from the training data and leverage the learned knowledge to perform the SE tasks". In this paper, we introduce EMPICA, a comprehensive framework designed to systematically and empirically evaluate the capabilities of code LLMs in understanding code semantics. Specifically, EMPICA systematically introduces controlled modifications/transformations into the input code and examines the models' responses. Generally, code LLMs must be robust to semantically equivalent code inputs and be sensitive to non-equivalent ones for all SE tasks. Specifically, for every SE task, given an input code snippet c and its semantic equivalent variants, code LLMs must robustly produce consistent/equivalent outputs while they are expected to generate different outputs for c and its semantic non-equivalent variants. Our experimental results on three representative code understanding tasks, including code summarization, method name prediction, and output prediction, reveal that the robustness and sensitivity of the state-of-the-art code LLMs to code transformations vary significantly across tasks and transformation operators. In addition, the code LLMs exhibit better robustness to the semantic preserving transformations than their sensitivity to the semantic non-preserving transformations. These results highlight a need to enhance the model's capabilities of understanding code semantics, especially the sensitivity property.

Toward a Method to Generate Capability Ontologies from Natural Language Descriptions

On the Use of Large Language Models to Generate Capability Ontologies

Towards Controllable Generative Design: A Conceptual Design Generation Approach Leveraging the FBS Ontology and Large Language Models

A RAG Approach for Generating Competency Questions in Ontology Engineering

Improving Natural Language Capability of Code Large Language Model

Large language models as oracles for instantiating ontologies with domain-specific knowledge

Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

Towards Goal-oriented Prompt Engineering for Large Language Models: A Survey

Automating Knowledge Discovery from Scientific Literature via LLMs: A Dual-Agent Approach with Progressive Ontology Prompting

Law of the Weakest Link: Cross Capabilities of Large Language Models

Low-code LLM: Graphical User Interface over Large Language Models

Prompt-Time Ontology-Driven Symbolic Knowledge Capture with Large Language Models

Learning from Failures: Translation of Natural Language Requirements into Linear Temporal Logic with Large Language Models

A Layered Architecture for Developing and Enhancing Capabilities in Large Language Model-based Software Systems

A Universal Prompting Strategy for Extracting Process Model Information from Natural Language Text using Large Language Models

Leveraging Large Language Models for Automating Inductive Qualitative Coding: A Comparative Study of Prompt Engineering Techniques

Chatbot-Based Ontology Interaction Using Large Language Models and Domain-Specific Standards

PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models

Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketches

Empowering Large Language Models on Robotic Manipulation with Affordance Prompting

An Empirical Study on Capability of Large Language Models in Understanding Code Semantics