Free to play: UN Trade and Development's experience with developing its own open-source Retrieval Augmented Generation Large Language Model application

Daniel Hopp

2024-06-18

Abstract:Generative artificial intelligence (AI), and in particular Large Language Models (LLMs), have exploded in popularity and attention since the release to the public of ChatGPT's Generative Pre-trained Transformer (GPT)-3.5 model in November of 2022. Due to the power of these general purpose models and their ability to communicate in natural language, they can be useful in a range of domains, including the work of official statistics and international organizations. However, with such a novel and seemingly complex technology, it can feel as if generative AI is something that happens to an organization, something that can be talked about but not understood, that can be commented on but not contributed to. Additionally, the costs of adoption and operation of proprietary solutions can be both uncertain and high, a barrier for often cost-constrained international organizations. In the face of these challenges, United Nations Trade and Development (UNCTAD), through its Global Crisis Response Group (GCRG), has explored and developed its own open-source Retrieval Augmented Generation (RAG) LLM application. RAG makes LLMs aware of and more useful for the organization's domain and work. Developing in-house solutions comes with pros and cons, with pros including cost, flexibility, and fostering institutional knowledge. Cons include time and skill investments and gaps and application polish and power. The three libraries developed to produce the app, nlp_pipeline for document processing and statistical analysis, local_rag_llm for running a local RAG LLM, and streamlit_rag for the user interface, are publicly available on PyPI and GitHub with Dockerfiles. A fourth library, local_llm_finetune, is also available for fine-tuning existing LLMs which can then be used in the application.

Computers and Society,Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

The paper aims to explore how the United Nations Conference on Trade and Development (UNCTAD) can develop its own open-source Retrieval-Augmented Generation (RAG) LLM application in the context of the rapid development of generative artificial intelligence, especially large language models (LLMs). Specifically, the paper focuses on how to overcome challenges related to cost, flexibility, and data privacy encountered when adopting proprietary solutions by building internal solutions. Additionally, the paper discusses the pros and cons of developing such applications, including advantages like cost savings, increased flexibility, facilitation of institutional knowledge accumulation, and data privacy protection, as well as challenges such as resource availability, scalability, product refinement, and access to cutting-edge models. By detailing UNCTAD's specific work and technical details, the paper demonstrates how to build a fully functional RAG LLM application using existing open-source tools and frameworks (such as LlamaIndex, pgvector, Streamlit, etc.). This not only provides a valuable reference case for other international organizations and national statistical offices but also emphasizes the importance of giving data science projects sufficient institutional space and support in a rapidly evolving technological field. In summary, the problem the paper attempts to address is how to effectively utilize generative AI technology, particularly RAG LLM applications tailored to specific domains and current information, to support the core work and services of international organizations while ensuring cost-effectiveness, data security, and flexibility through the use of open-source technology and internal development capabilities.

Free to play: UN Trade and Development's experience with developing its own open-source Retrieval Augmented Generation Large Language Model application

Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report

RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation

RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture

Deploying Large Language Models With Retrieval Augmented Generation

Faculty Perspectives on the Potential of RAG in Computer Science Higher Education

Retrieval-Augmented Test Generation: How Far Are We?

RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation

Investigating the performance of Retrieval-Augmented Generation and fine-tuning for the development of AI-driven knowledge-based systems

Web Application for Retrieval-Augmented Generation: Implementation and Testing

RAG based Chatbot using LLMs

Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems

Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

T-RAG: Lessons from the LLM Trenches

Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely

NinjaLLM: Fast, Scalable and Cost-effective RAG using Amazon SageMaker and AWS Trainium and Inferentia2

An advanced retrieval-augmented generation system for manufacturing quality control

Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models

DRAGIN: Dynamic Retrieval Augmented Generation Based on the Real-time Information Needs of Large Language Models.

A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability