Abstract:Background: Video programming tutorials are becoming a popular resource for developers looking for quick answers to a specific programming problem or trying to learn a programming topic in more depth. Since the most important source of information for developers in many such videos is source code, it is important to be able to accurately extract this code from the screen, such that developers can easily integrate it into their programs. Aims: Our main goal is to facilitate the accurate and noise-free extraction of code appearing in programming video tutorials. In particular, in this paper we aim to accurately predict the location of source code in video frames. This will allow for the dramatic reduction of noise when using extraction techniques such as Optical Character Recognition, which could otherwise extract a large amount of irrelevant text (e.g., text found in menu items, package hierarchy, etc.). Method: We propose an approach using a deep Convolutional Neural Network (CNN) to predict the bounding box of fully-visible code sections in video frames. To evaluate our approach, we collected a set of 150 Java programming tutorials, having more than 82K frames in total. A sample of 4,000 frames from these videos were then manually annotated with the code bounding box location and used as the ground truth in an experiment evaluating our approach. Results: The results of the evaluation show that our approach is able to successfully predict the code bounding box in a given frame with 92% accuracy. Conclusions: Our CNN-based approach is able to accurately predict the location of source code within the frames of programming video tutorials.

Enhancing Developer Interactions with Programming Screencasts Through Accurate Code Extraction

psc2code: Denoising Code Extraction from Programming Screencasts

Efficient Search of Live-Coding Screencasts from Online Videos

Scvripper: Video Scraping Tool for Modeling Developers' Behavior Using Interaction Data

Extracting and Analyzing Time-Series HCI Data from Screen-Captured Task Videos

Reverse Engineering Time-Series Interaction Data from Screen-Captured Videos.

CodeSCAN: ScreenCast ANalysis for Video Programming Tutorials

SeeHow: Workflow Extraction from Programming Screencasts through Action-Aware Video Analytics

BPMiner: mining developers' behavior patterns from screen-captured task videos.

Optimizing OCR Performance for Programming Videos: The Role of Image Super-Resolution and Large Language Models

VT-Revolution: Interactive Programming Video Tutorial Authoring and Watching System

ActionNet: Vision-Based Workflow Action Recognition from Programming Screencasts

An Adobe Flash-based Screencast Solution with Customized Screen Codec

PinNet: Pinpoint Instructive Information for Retrieval Augmented Code-to-Text Generation

Accurately Predicting the Location of Code Fragments in Programming Video Tutorials Using Deep Learning

CPDScorer: Modeling and Evaluating Developer Programming Ability Across Software Communities.

Incorporating Code Structure and Quality in Deep Code Search

Automatically Modeling Developer Programming Ability and Interest Across Software Communities

Codehow: Effective Code Search Based On Api Understanding And Extended Boolean Model

Empowering Computer Vision in Higher Education: A Novel Tool for Enhancing Video Coding Comprehension

Revisiting Code Search in a Two-Stage Paradigm