ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages

Mehant Kammakomati,Sameer Pimparkhede,Srikanth Tamilselvam,Prince Kumar,Pushpak Bhattacharyya
2024-08-30
Abstract:Recent work shows Large Language Models (LLMs) struggle to understand natural language constraints for various text generation tasks in zero- and few-shot settings. While, in the code domain, there is wide usage of constraints in code format to maintain the integrity of code written in Domain-Specific Languages (DSLs) like JSON and YAML which are widely used for system-level programming tasks in enterprises. Given that LLMs are increasingly used for system-level code tasks, evaluating if they can comprehend these code constraints is crucial. However, no work has been done to evaluate their controllability over code constraints. Hence, we introduce ConCodeEval, a first-of-its-kind benchmark having two novel tasks for code constraints across five representations. Our findings suggest that language models struggle with code constraints. Code languages that perform excellently for normal code tasks do not perform well when the same languages represent fine-grained constraints.
Software Engineering,Artificial Intelligence,Computation and Language
What problem does this paper attempt to address?
This paper aims to address the ability of large - language models (LLMs) when dealing with code constraints in domain - specific languages (DSLs). Specifically, although LLMs perform excellently in natural - language - processing tasks, they have difficulties in understanding and following the fine - grained constraints in code formats. These constraints are crucial for maintaining code integrity in enterprise - level system - programming tasks, especially when writing code using DSLs such as JSON and YAML. Therefore, it becomes particularly important to evaluate whether LLMs can understand and apply these code constraints. However, there is currently no dedicated work to evaluate the control ability of LLMs in this regard. For this reason, the author introduced ConCodeEval, which is a brand - new benchmark test that includes two novel tasks for evaluating the code - constraint - handling ability of LLMs in five different representations. Through these tasks, the research found that language models do indeed have difficulties when dealing with code constraints, and even in code languages that usually perform well, their performance will decline significantly when it comes to fine - grained constraints.