Abstract:Recent approaches in domain-specific named entity recognition (NER), such as biomedical NER, have shown remarkable advances. However, they still lack of faithfulness, producing erroneous predictions. We assume that knowledge of entities can be useful in verifying the correctness of the predictions. Despite the usefulness of knowledge, resolving such errors with knowledge is nontrivial, since the knowledge itself does not directly indicate the ground-truth label. To this end, we propose VerifiNER, a post-hoc verification framework that identifies errors from existing NER methods using knowledge and revises them into more faithful predictions. Our framework leverages the reasoning abilities of large language models to adequately ground on knowledge and the contextual information in the verification process. We validate effectiveness of VerifiNER through extensive experiments on biomedical datasets. The results suggest that VerifiNER can successfully verify errors from existing models as a model-agnostic approach. Further analyses on out-of-domain and low-resource settings show the usefulness of VerifiNER on real-world applications.
What problem does this paper attempt to address?
The problems that this paper attempts to solve are as follows: Although existing named entity recognition (NER) models in specific domains, such as NER models in the biomedical field, have made remarkable progress, they still have deficiencies in prediction accuracy. In particular, they are prone to making incorrect predictions in areas requiring professional knowledge. These problems are especially prominent in application scenarios with high - precision requirements.
Specifically, the paper points out the following problems:
1. **Lack of accuracy**: Although existing methods can generate seemingly reasonable prediction results, these results are not always accurate.
2. **Error types**: Common errors include mislabeling of entity types, misjudgment of entity scopes, and completely wrong entity predictions.
3. **Insufficient use of knowledge**: Although external knowledge is helpful for verifying and correcting errors, since the knowledge itself does not directly indicate the true labels, how to effectively use this knowledge is a challenge.
4. **Insufficient generalization ability of the model**: When dealing with low - resource or out - of - distribution data, the performance of existing models is not ideal.
To solve these problems, the author proposes a framework named VERIFI NER, which aims to identify and correct incorrect predictions in existing NER models by introducing the reasoning ability of external knowledge bases (KB) and large - language models (LLM). The main contributions of VERIFI NER are as follows:
- Proposing a knowledge - based reasoning framework for identifying and correcting errors in NER models.
- Ensuring the accuracy and consistency of predictions through the factual and context - relevance verification modules.
- Conducting extensive experiments on multiple biomedical datasets to verify the effectiveness and generalization ability of VERIFI NER.
### Formula Summary
To express the working process of VERIFI NER more clearly, the following are the key steps involved and their formula representations:
1. **Candidate Span Extraction**:
- Given an input sequence \( X=\{x_1, x_2,\ldots,x_n\} \) and a predefined type set \( T \), the goal is to generate a revised entity prediction \( \bar{e} \) from the entities \( e = (s, t) \) predicted by the existing NER model.
- Expand the left and right offsets of the span \( s \) of the predicted entity to form a set of candidate spans \( \tilde{S} \).
2. **Factual Verification**:
- For each candidate span \( \tilde{s}\in\tilde{S} \), query the knowledge base (KB) to verify its factuality and collect associated knowledge \( k \).
- If the candidate span is defined in the KB, it is considered that its factuality has been verified; otherwise, it is regarded as noise and excluded.
3. **Type Factual Verification**:
- According to the retrieved knowledge \( k \), re - assign the type \( \tilde{t} \) of the candidate span \( \tilde{s} \).
- Use the LLM to generate knowledge - based evidence \( k' \) and re - assign the type \( \tilde{t} \) according to the evidence.
4. **Context - Relevance Verification**:
- Combine the input context \( X \) and the generated evidence \( k' \), and use the reasoning ability of the LLM to select the candidate entity \( \tilde{e} \) that best conforms to the context.
- Select the final revised entity \( \bar{e}=(\bar{s},\bar{t}) \) through a unanimous voting mechanism.
In this way, VERIFI NER can significantly improve the accuracy of NER predictions without retraining the model, especially in areas requiring professional knowledge.