Abstract:Enterprise searches require users to have complex knowledge of queries, configurations, and metadata, rendering it difficult for them to access information as needed. Most go-to-market (GTM) platforms utilize advanced search, an interface that enables users to filter queries by various fields using categories or keywords, which, historically, however, has proven to be exceedingly cumbersome, as users are faced with seemingly hundreds of options, fields, and buttons. Consequently, querying with natural language has long been ideal, a notion further empowered by Large Language Models (LLMs). In this paper, we implement and evaluate a solution for the Zoominfo product for sellers, which prompts the LLM with natural language, producing search fields through entity extraction that are then converted into a search query. The intermediary search fields offer numerous advantages for each query, including the elimination of syntax errors, simpler ground truths, and an intuitive format for the LLM to interpret. We paired this pipeline with many advanced prompt engineering strategies, featuring an intricate system message, few-shot prompting, chain-of-thought (CoT) reasoning, and execution refinement. Furthermore, we manually created the ground truth for 500+ natural language queries, enabling the supervised fine-tuning of Llama-3-8B-Instruct and the introduction of sophisticated numerical metrics. Comprehensive experiments with closed, open source, and fine-tuned LLM models were conducted through exact, Jaccard, cosine, and semantic similarity on individual search entities to demonstrate the efficacy of our approach. Overall, the most accurate closed model had an average accuracy of 97% per query, with only one field performing under 90%, with comparable results observed from the fine-tuned models.

Using off-the-shelf LLMs to query enterprise data by progressively revealing ontologies

Large language models as oracles for instantiating ontologies with domain-specific knowledge

Chatbot-Based Ontology Interaction Using Large Language Models and Domain-Specific Standards

Fine-tuning Large Enterprise Language Models via Ontological Reasoning

Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!

LLMs4OL: Large Language Models for Ontology Learning

LLMs4OM: Matching Ontologies with Large Language Models

Enriching Ontologies with Disjointness Axioms using Large Language Models

Boosting LLMS with Ontology-Aware Prompt for Ner Data Augmentation

Querying Large Language Models with SQL

Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

Making LLMs Work for Enterprise Data Tasks

Can Large Language Models Understand DL-Lite Ontologies? An Empirical Study

Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms

End-to-End Ontology Learning with Large Language Models

Lucy: Think and Reason to Solve Text-to-SQL

Do LLMs Really Adapt to Domains? An Ontology Learning Perspective

OLaLa: Ontology Matching with Large Language Models

From human experts to machines: An LLM supported approach to ontology and knowledge graph construction

Generating SPARQL Queries over CIDOC-CRM using a Two-Stage Ontology Path Patterns Method in LLM Prompts

Combining large language models with enterprise knowledge graphs: a perspective on enhanced natural language understanding