The Foundation Model Transparency Index v1.1: May 2024

Rishi Bommasani,Kevin Klyman,Sayash Kapoor,Shayne Longpre,Betty Xiong,Nestor Maslej,Percy Liang
2024-07-18
Abstract:Foundation models are increasingly consequential yet extremely opaque. To characterize the status quo, the Foundation Model Transparency Index was launched in October 2023 to measure the transparency of leading foundation model developers. The October 2023 Index (v1.0) assessed 10 major foundation model developers (e.g. OpenAI, Google) on 100 transparency indicators (e.g. does the developer disclose the wages it pays for data labor?). At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study (v1.1) after 6 months: we score 14 developers against the same 100 indicators. While in v1.0 we searched for publicly available information, in v1.1 developers submit reports on the 100 transparency indicators, potentially including information that was not previously public. We find that developers now score 58 out of 100 on average, a 21 point improvement over v1.0. Much of this increase is driven by developers disclosing information during the v1.1 process: on average, developers disclosed information related to 16.6 indicators that was not previously public. We observe regions of sustained (i.e. across v1.0 and v1.1) and systemic (i.e. across most or all developers) opacity such as on copyright status, data access, data labor, and downstream impact. We publish transparency reports for each developer that consolidate information disclosures: these reports are based on the information disclosed to us via developers. Our findings demonstrate that transparency can be improved in this nascent ecosystem, the Foundation Model Transparency Index likely contributes to these improvements, and policymakers should consider interventions in areas where transparency has not improved.
Machine Learning,Artificial Intelligence,Computers and Society
What problem does this paper attempt to address?
The research paper "Transparency Index of Basic Models v1.1" focuses on the transparency issue of basic models in the field of artificial intelligence. The authors evaluated 14 major developers of basic models and used the same set of 100 transparency indicators as before to measure their transparency in terms of data, labor, computing power input, model characteristics, downstream applications, and impact. In the initial evaluation (v1.0), the average score was only 37, but in the follow-up study six months later (v1.1), the average score improved to 58, showing significant improvement. Developers proactively provided more undisclosed information in the v1.1 stage, with an average of 16.6 new indicator-related information disclosed per developer. However, the study also identified some persistent areas of opacity, such as copyright status, data access, data labor, and downstream impact. The report suggests that policymakers consider intervention measures in these areas where transparency has not improved. The paper also discusses the impact of the transparency index on the ecosystem, suggesting that it may facilitate transparency improvement and pointing out that developers can learn from transparency practices of other companies to further enhance transparency. In addition, although developers have improved their scores in certain areas, there are still significant gaps, particularly in upstream resources, namely data, labor, and computing resources required for model construction. Overall, the paper aims to track and promote transparency improvement in the ecosystem of basic models through the transparency index, calling for more information disclosure and policy interventions to foster accountability, competition, and collective understanding.