Abstract:Large Language Models (LLMs) show promising potential in Software Engineering, especially for code-related tasks like code completion and code generation. LLMs' evaluation is generally centred around general metrics computed over benchmarks. While painting a macroscopic view of the benchmarks and of the LLMs' capacity, it is unclear how each programming task in these benchmarks assesses the capabilities of the LLMs. In particular, the difficulty level of the tasks in the benchmarks is not reflected in the score used to report the performance of the model. Yet, a model achieving a 90% score on a benchmark of predominantly easy tasks is likely less capable than a model achieving a 90% score on a benchmark containing predominantly difficult tasks. This paper devises a framework, HardEval, for assessing task difficulty for LLMs and crafting new tasks based on identified hard tasks. The framework uses a diverse array of prompts for a single task across multiple LLMs to obtain a difficulty score for each task of a benchmark. Using two code generation benchmarks, HumanEval+ and ClassEval, we show that HardEval can reliably identify the hard tasks within those benchmarks, highlighting that only 21% of HumanEval+ and 27% of ClassEval tasks are hard for LLMs. Through our analysis of task difficulty, we also characterize 6 practical hard task topics which we used to generate new hard tasks. Orthogonal to current benchmarking evaluation efforts, HardEval can assist researchers and practitioners in fostering better assessments of LLMs. The difficulty score can be used to identify hard tasks within existing benchmarks. This, in turn, can be leveraged to generate more hard tasks centred around specific topics either for evaluation or improvement of LLMs. HardEval generalistic approach can be applied to other domains such as code completion or Q/A.

Evaluation of Topic Difficulty

Part-of-Speech Sense Matrix Model Experiments in the TREC 2004 Robust Track at ICL, PKU.

Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing

Multi-Faceted Question Complexity Estimation Targeting Topic Domain-Specificity

Text Difficulty Study: Do Machines Behave the Same as Humans Regarding Text Difficulty?

Automatic Classification and Comparison of Words by Difficulty.

Question Difficulty Prediction for READING Problems in Standard Tests.

Predicting Spatial Visualization Problems' Difficulty Level from Eye-Tracking Data

A Comparative Study of the Relationship between the Subjective Difficulty, Objective Difficulty of Search Tasks and Search Behaviors

Predicting Search Task Difficulty at Different Search Stages.

Exploring and predicting search task difficulty.

Linguistic features for sentence difficulty prediction in ABSA

Statistical Word Sense Aware Topic Models

Predicting task difficulty for different task types

Question Difficulty Ranking for Multiple-Choice Reading Comprehension

Difficulty-level modeling of ontology-based factual questions

Word Sense Disambiguation Method with Topic Feature

Difficulty-Aware Machine Translation Evaluation

Exploiting Syntactic and Semantic Relationships Between Terms for Opinion Retrieval

Topic Model With Constrainted Word Burstiness Intensities

Assessing Programming Task Difficulty for Efficient Evaluation of Large Language Models