The Role of Code Proficiency in the Era of Generative AI

Gregorio Robles,Christoph Treude,Jesus M. Gonzalez-Barahona,Raula Gaikovina Kula

2024-04-08

Abstract:At the current pace of technological advancements, Generative AI models, including both Large Language Models and Large Multi-modal Models, are becoming integral to the developer workspace. However, challenges emerge due to the 'black box' nature of many of these models, where the processes behind their outputs are not transparent. This position paper advocates for a 'white box' approach to these generative models, emphasizing the necessity of transparency and understanding in AI-generated code to match the proficiency levels of human developers and better enable software maintenance and evolution. We outline a research agenda aimed at investigating the alignment between AI-generated code and developer skills, highlighting the importance of responsibility, security, legal compliance, creativity, and social value in software development. The proposed research questions explore the potential of white-box methodologies to ensure that software remains an inspectable, adaptable, and trustworthy asset in the face of rapid AI integration, setting a course for research that could shape the role of code proficiency into 2030 and beyond.

Software Engineering

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is, in the era of Generative AI, how to ensure that the code generated by AI has transparency and understandability, so as to match the skill levels of human developers and better support the maintenance and evolution of software. Specifically, the paper explores the applications of the "black - box" and "white - box" approaches in AI - generated code, emphasizing the importance of transparency and understanding.

### Main Issues
1. **Black - box vs White - box**:
- **Black - box Approach**: Developers interact with AI through natural language or other high - level abstractions, and the AI directly generates executable code or compiled binary code, and developers do not need to understand the internal details.
- **White - box Approach**: Developers can inspect and understand the source code generated by AI to ensure that it meets the expected behavior and quality standards.

2. **Code Transparency and Understandability**:
- Due to the "black - box" nature of generative AI models, in many cases, developers cannot fully understand the process and logic of AI - generated code, which may lead to code that is difficult to maintain and debug.
- The paper advocates the "white - box" approach, enabling developers to inspect, understand, and modify the code generated by AI, thereby ensuring the quality and reliability of the code.

3. **Code Proficiency**:
- In the white - box approach, the code generated by AI should be adjusted according to the skill level of the developer, making it easy to understand and maintain.
- The paper proposes the need to create a proficiency assessment framework similar to natural languages (such as CEFR) for programming languages, so that AI can generate appropriate code according to the developer's skill level.

### Research Agenda
The paper also proposes a research agenda, aiming to explore the application of the white - box approach in the era of generative AI and its impact on code proficiency. Specific research questions include:

- **The Relationship between Code Quality and Developer Skills**:
- Does the quality of AI - generated code reflect the skill level of the developer?
- Is the number of iterations required to achieve high - quality code related to the developer's skill level?

- **Responsibility, Safety, and Legal Compliance**:
- How does the white - box approach help clarify the attribution of responsibility for code results?
- How to ensure that the code generated by AI remains safe and meets legal requirements at different skill levels?

- **Creativity and Social Value**:
- Is high - quality code more creative?
- Will the code generated by AI reduce diverse problem - solving methods?
- Do developers have sufficient skills to review automatically generated code?

### Conclusion
The paper believes that with the wide application of generative AI in software development, adopting the white - box approach is the key to ensuring code transparency and understandability. This not only helps to improve code quality and maintainability but also enhances developers' trust in AI - generated code, ensuring responsibility, safety, and legal compliance in the software development process.