Abstract:Context: When software is released publicly, it is common to include with it either the full text of the license or licenses under which it is published, or a detailed reference to them. Therefore public licenses, including FOSS (free, open source software) licenses, are usually publicly available in source code repositories.Objective: To compile a dataset containing as many documents as possible that contain the text of software licenses, or references to the license terms. Once compiled, characterize the dataset so that it can be used for further research, or practical purposes related to license analysis.Method: Retrieve from Software Heritage-the largest publicly available archive of FOSS source code-all versions of all files whose names are commonly used to convey licensing terms. All retrieved documents will be characterized in various ways, using automated and manual analyses.Results: The dataset consists of 6.9 million unique license files. Additional metadata about shipped license files is also provided, making the dataset ready to use in various contexts, including: file length measures, MIME type, SPDX license (detected using ScanCode), and oldest appearance. The results of a manual analysis of 8102 documents is also included, providing a ground truth for further analysis. The dataset is released as open data as an archive file containing all deduplicated license files, plus several portable CSV files with metadata, referencing files via cryptographic checksums.Conclusions: Thanks to the extensive coverage of Software Heritage, the dataset presented in this paper covers a very large fraction of all software licenses for public code. We have assembled a large body of software licenses, characterized it quantitatively and qualitatively, and validated that it is mostly composed of licensing information and includes almost all known license texts. The dataset can be used to conduct empirical studies on open source licensing, training of automated license classifiers, natural language processing (NLP) analyses of legal texts, as well as historical and phylogenetic studies on FOSS licensing. It can also be used in practice to improve tools detecting licenses in source code.

A Large-scale Dataset of (Open Source) License Text Variants

The Software Heritage License Dataset (2022 Edition)

A Large-Scale Empirical Study of Open Source License Usage: Practices and Challenges

OSS License Identification at Scale: A Comprehensive Dataset Using World of Code

The Software Heritage Graph Dataset: Large-scale Analysis of Public Software Development History

GitHub OSS Governance File Dataset

Catch the Butterfly: Peeking into the Terms and Conflicts among SPDX Licenses

The Stack: 3 TB of permissively licensed source code

A large-scale audit of dataset licensing and attribution in AI

Analyzing FOSS license usage in publicly available software at scale via the SWH-analytics framework

The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI

Seeing eye to eye? An exploratory study of free open source software users' perceptions

An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets

A Novel Model for Automatic Identification of Open Source Software License Terms

The Software Heritage Open Science Ecosystem

Public Git Archive: a Big Code dataset for all

A Dataset of Enterprise-Driven Open Source Software

Dataset: Copy-based Reuse in Open Source Software

LiSum: Open Source Software License Summarization with Multi-Task Learning

LiScopeLens: An Open-Source License Incompatibility Analysis Tool Based on Scope Representation of License Terms

LiDetector: License Incompatibility Detection for Open Source Software.