Transformer-Based Siamese Textual Encoder for Knowledge Base Completion

Mengyao Li,Xiuhua Ren,Guanhai Li,Bo Wang,Tao Shen,Zonghan Wu,Yang Li,Hao Huang,Peng Zhang

2021-01-01

Abstract:Transformer-Based Siamese Textual Encoder for Knowledge Base Completion

What problem does this paper attempt to address?

Improving Knowledge Base Completion by Incorporating Implicit Information.

Wenqiang He,Yansong Feng,Dongyan Zhao

DOI: https://doi.org/10.1007/978-3-319-31676-5_10

2015-01-01

Abstract:Over the past few years, many large Knowledge Bases (KBs) have been constructed through relation extraction technology but they are still often incomplete. As a supplement to training a more powerful extractor, Knowledge Base Completion which aims at learning new facts based on existing ones has recently attracted much attention. Most of the existing methods, however, are only utilizing the explicit facts in a single KB. By analyzing the data, we find that some implicit information should also been captured for a more comprehensive consideration during completion process. These information include the intrinsic properties of KBs (e.g. relational constraints) and potential synergies between various KBs (i.e. semantic similarity). For the former, we distinguish the missing data by using relational constraints to reduce the data sparsity. For the later, we incorporate two semantical regularizations into the learning model to encode the semantic similarity. Experimental results show that our approach is better than the methods that consider only explicit facts or only a single knowledge base, and achieves significant accuracy improvements in binary relation prediction.
Generative Transformer with Knowledge-Guided Decoding for Academic Knowledge Graph Completion

Xiangwen Liu,Shengyu Mao,Xiaohan Wang,Jiajun Bu

DOI: https://doi.org/10.3390/math11051073

IF: 2.4

2023-01-01

Mathematics

Abstract:Academic knowledge graphs are essential resources and can be beneficial in widespread real-world applications. Most of the existing academic knowledge graphs are far from completion; thus, knowledge graph completion—the task of extending a knowledge graph with missing entities and relations—attracts many researchers. Most existing methods utilize low-dimensional embeddings to represent entities and relations and follow the discrimination paradigm for link prediction. However, discrimination approaches may suffer from the scaling issue during inference with large-scale academic knowledge graphs. In this paper, we propose a novel approach of a generative transformer with knowledge-guided decoding for academic knowledge graph completion. Specifically, we introduce generative academic knowledge graph pre-training with a transformer. Then, we propose knowledge-guided decoding, which leverages relevant knowledge in the training corpus as guidance for help. We conducted experiments on benchmark datasets for knowledge graph completion. The experimental results show that the proposed approach can achieve performance gains of 30 units of the MRR score over the baselines on the academic knowledge graph AIDA.
kTrans: Knowledge-Aware Transformer for Binary Code Embedding

Wenyu Zhu,Hao Wang,Yuchen Zhou,Jiaming Wang,Zihan Sha,Zeyu Gao,Chao Zhang

2023-08-24

Abstract:Binary Code Embedding (BCE) has important applications in various reverse engineering tasks such as binary code similarity detection, type recovery, control-flow recovery and data-flow analysis. Recent studies have shown that the Transformer model can comprehend the semantics of binary code to support downstream tasks. However, existing models overlooked the prior knowledge of assembly language. In this paper, we propose a novel Transformer-based approach, namely kTrans, to generate knowledge-aware binary code embedding. By feeding explicit knowledge as additional inputs to the Transformer, and fusing implicit knowledge with a novel pre-training task, kTrans provides a new perspective to incorporating domain knowledge into a Transformer framework. We inspect the generated embeddings with outlier detection and visualization, and also apply kTrans to 3 downstream tasks: Binary Code Similarity Detection (BCSD), Function Type Recovery (FTR) and Indirect Call Recognition (ICR). Evaluation results show that kTrans can generate high-quality binary code embeddings, and outperforms state-of-the-art (SOTA) approaches on downstream tasks by 5.2%, 6.8%, and 12.6% respectively. kTrans is publicly available at: <a class="link-external link-https" href="https://github.com/Learner0x5a/kTrans-release" rel="external noopener nofollow">this https URL</a>

Software Engineering,Artificial Intelligence,Cryptography and Security
TE-TFN: A Text-enhanced Transformer Fusion Network for Multimodal Knowledge Graph Completion

Jingchao Wang,Xiao Liu,Weimin Li,Fangfang Liu,Xing Wu,Qun Jin

DOI: https://doi.org/10.1109/mis.2024.3378921

IF: 6.744

2024-01-01

IEEE Intelligent Systems

Abstract:Multimodal knowledge graphs (MKGs) organize multimodal facts in the form of entities and relations, and have been successfully applied to several downstream tasks. Since most MKGs are incomplete, the MKG completion (MKGC) task has been proposed to address this problem, which aims to complete missing entities in MKGs. Previous most works obtain reasoning ability by capturing the correlation between target triplets and related images, but they ignore contextual semantic information and the reasoning process is not easily explainable. To address these issues, we propose a novel text-enhanced transformer fusion network called TE-TFN, which converts the context path between head and tail entities into natural language text and fuses multimodal features from both coarse and fine granularities through a multi-granularity fuser. It not only effectively enhances text semantic information, but also improves the interpretability of the model by introducing paths. Experimental results on benchmark datasets demonstrate the effectiveness of our model.

computer science, artificial intelligence,engineering, electrical & electronic
Jointly Modeling Fact Triples and Text Information for Knowledge Base Completion.

Xiuxing Li,Zhenyu Li,Zhichao Duan,Jiacheng Xu,Ning Liu,Jianyong Wang

DOI: https://doi.org/10.1109/ickg52313.2021.00037

2021-01-01

Abstract:Knowledge bases become essential resources for many data mining and information retrieval tasks, but they remain far from complete. Knowledge base completion has attracted extensive research efforts from researchers and prac-titioners in diverse areas, which aims to infer missing facts from existing ones in a knowledge base. Quantities of knowledge base completion methods have been developed by regarding each relation as a translation from head entity to tail entity. However, existing methods merely concentrate on fact triples in the knowledge base or co-occurrence of words in the text, while supplementary semantic information expressed via related entities in the text has not been fully exploited. Meanwhile, the representation ability of current methods encounters bottlenecks due to the structure sparseness of knowledge base. In this paper, we propose a novel knowledge base representation learning method by taking advantage of the rich semantic information expressed via related entities in the textual corpus to expand the semantic structure of knowledge base. In this way, our model can break through the limitation of structure sparseness and promote the performance of knowledge base completion. Extensive experiments on two real-world datasets show that the proposed method successfully addresses the above issues and significantly outperforms the state-of-the-art methods on the benchmark task of link prediction.
Bidirectional transformer with knowledge graph for video captioning

Maosheng Zhong,Youde Chen,Hao Zhang,Hao Xiong,Zhixiang Wang,Chen, Youde

DOI: https://doi.org/10.1007/s11042-023-17822-4

IF: 2.577

2023-12-22

Multimedia Tools and Applications

Abstract:Models based on transformer architecture have risen to prominence for video captioning. However, most models are only to improve either the encoder or the decoder, because when we improve the encoder and decoder simultaneously, the shortcomings of either side may be amplified. Based on the transformer architecture, we connect a bidirectional decoder and an encoder that integrates fine-grained spatio-temporal features, objects, and relationships between the objects in the video. Experiments show that improvements in the encoder amplify the information leakage of the bidirectional decoder and further produce a worse result. To tackle this problem, we generate pseudo reverse captions and propose a Bidirectional Transformer with Knowledge Graph (BTKG), which integrates the outputs of two encoders into the forward and backward decoders of the bidirectional decoder, respectively. In addition, we make fine-grained improvements on the interior of the different encoders according to four modal features of the video. Experiments on two mainstream benchmark datasets, i.e., MSVD and MSR-VTT, demonstrate the effectiveness of BTKG, which achieves state-of-the-art performance in significant metrics. Moreover, the sentences generated by BTKG contain scene words and modifiers, that are more in line with human language habits. Codes are available on https://github.com/nickchen121/BTKG.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching

Liu Yang,Mingyang Zhang,Cheng Li,Michael Bendersky,Marc Najork

DOI: https://doi.org/10.1145/3340531.3411908

2020-10-13

Abstract:Many natural language processing and information retrieval problems can be formalized as the task of semantic matching. Existing work in this area has been largely focused on matching between short texts (e.g., question answering), or between a short and a long text (e.g., ad-hoc retrieval). Semantic matching between long-form documents, which has many important applications like news recommendation, related article recommendation and document clustering, is relatively less explored and needs more research effort. In recent years, self-attention based models like Transformers and BERT have achieved state-of-the-art performance in the task of text matching. These models, however, are still limited to short text like a few sentences or one paragraph due to the quadratic computational complexity of self-attention with respect to input text length. In this paper, we address the issue by proposing the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for long-form document matching. Our model contains several innovations to adapt self-attention models for longer text input. In order to better capture sentence level semantic relations within a document, we pre-train the model with a novel masked sentence block language modeling task in addition to the masked word language modeling task used by BERT. Our experimental results on several benchmark datasets for long-form document matching show that our proposed SMITH model outperforms the previous state-of-the-art models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT. Comparing to BERT based baselines, our model is able to increase maximum input text length from 512 to 2048. We will open source a Wikipedia based benchmark dataset, code and a pre-trained checkpoint to accelerate future research on long-form document matching.

Information Retrieval,Computation and Language
DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection

Nic Jedema,Thuy Vu,Manish Gupta,Alessandro Moschitti

DOI: https://doi.org/10.48550/arXiv.2203.09598

2022-03-17

Computation and Language

Abstract:While transformers demonstrate impressive performance on many knowledge intensive (KI) tasks, their ability to serve as implicit knowledge bases (KBs) remains limited, as shown on several slot-filling, question-answering (QA), fact verification, and entity-linking tasks. In this paper, we implement an efficient, data-programming technique that enriches training data with KB-derived context and improves transformer utilization of encoded knowledge when fine-tuning for a particular QA task, namely answer sentence selection (AS2). Our method outperforms state of the art transformer approach on WikiQA and TrecQA, two widely studied AS2 benchmarks, increasing by 2.0% p@1, 1.3% MAP, 1.1% MRR, and 4.4% p@1, 0.9% MAP, 2.4% MRR, respectively. To demonstrate our improvements in an industry setting, we additionally evaluate our approach on a proprietary dataset of Alexa QA pairs, and show increase of 2.3% F1 and 2.0% MAP. We additionally find that these improvements remain even when KB context is omitted at inference time, allowing for the use of our models within existing transformer workflows without additional latency or deployment costs.
Convolutional Network Embedding of Text-Enhanced Representation for Knowledge Graph Completion

Feng Zhao,Tao Xu,Langjunqing Jin,Hai Jin

DOI: https://doi.org/10.1109/jiot.2020.3039750

IF: 10.6

2021-12-01

IEEE Internet of Things Journal

Abstract:In recent years, knowledge graph representation learning has prompted extensive research. Machine learning models are used to map entity and relational data in knowledge graphs to vector representations in low-dimensional spaces to predict and analyze potential relationships. Current works mainly focus on the knowledge representation of the triple structure and relationship path in knowledge graphs without fully utilizing external textual information to semantically supplement knowledge representation. However, the existing knowledge inventory, such as that for smart health and emotion care systems, is relatively meager, and structural knowledge is incomplete; therefore, knowledge graph completion is essential. In this article, we propose a novel joint representation learning model that introduces text description information and extracts reliable feature information from text data by using a convolutional neural network (CNN) model. Furthermore, being based on the attention mechanism, the proposed model distinguishes the characteristic credibility of different relationships, enhances the representation of the entity relationship structure vector in the existing knowledge graph, and obtains rich semantic information. Finally, the 2-D convolution operation is used to process the joint representation vectors of entities and relationships to obtain nonlinear features, and the knowledge graph is completed by completing the calculation of the score function of the joint representation vector of the entity and the relationship. Experiments performing tasks, such as link prediction and triple classification, on the FreeBase (FB15k), WordNet (WN18) and Yet Another Great Ontology (YAGO3-10) data sets reveal that our model performs better than the benchmark model and has some degree of scalability.

computer science, information systems,telecommunications,engineering, electrical & electronic
Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

Xiang Chen,Ningyu Zhang,Lei Li,Shumin Deng,Chuanqi Tan,Changliang Xu,Fei Huang,Luo Si,Huajun Chen

DOI: https://doi.org/10.1145/3477495.3531992

2023-09-19

Abstract:Multimodal Knowledge Graphs (MKGs), which organize visual-text factual knowledge, have recently been successfully applied to tasks such as information retrieval, question answering, and recommendation system. Since most MKGs are far from complete, extensive knowledge graph completion studies have been proposed focusing on the multimodal entity, relation extraction and link prediction. However, different tasks and modalities require changes to the model architecture, and not all images/objects are relevant to text input, which hinders the applicability to diverse real-world scenarios. In this paper, we propose a hybrid transformer with multi-level fusion to address those issues. Specifically, we leverage a hybrid transformer architecture with unified input-output for diverse multimodal knowledge graph completion tasks. Moreover, we propose multi-level fusion, which integrates visual and text representation via coarse-grained prefix-guided interaction and fine-grained correlation-aware fusion modules. We conduct extensive experiments to validate that our MKGformer can obtain SOTA performance on four datasets of multimodal link prediction, multimodal RE, and multimodal NER. Code is available in <a class="link-external link-https" href="https://github.com/zjunlp/MKGformer" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning,Multimedia
An Interpretable Knowledge Transfer Model for Knowledge Base Completion

Qizhe Xie,Xuezhe Ma,Zihang Dai,Eduard Hovy

DOI: https://doi.org/10.48550/arXiv.1704.05908

2017-04-19

Computation and Language

Abstract:Knowledge bases are important resources for a variety of natural language processing tasks but suffer from incompleteness. We propose a novel embedding model, \emph{ITransF}, to perform knowledge base completion. Equipped with a sparse attention mechanism, ITransF discovers hidden concepts of relations and transfer statistical strength through the sharing of concepts. Moreover, the learned associations between relations and concepts, which are represented by sparse attention vectors, can be interpreted easily. We evaluate ITransF on two benchmark datasets---WN18 and FB15k for knowledge base completion and obtains improvements on both the mean rank and Hits@10 metrics, over all baselines that do not use additional information.
Empirical Study of Transformers for Source Code

Nadezhda Chirkova,Sergey Troshin

DOI: https://doi.org/10.1145/3468264.3468611

2021-06-24

Abstract:Initially developed for natural language processing (NLP), Transformers are now widely used for source code processing, due to the format similarity between source code and text. In contrast to natural language, source code is strictly structured, i.e., it follows the syntax of the programming language. Several recent works develop Transformer modifications for capturing syntactic information in source code. The drawback of these works is that they do not compare to each other and consider different tasks. In this work, we conduct a thorough empirical study of the capabilities of Transformers to utilize syntactic information in different tasks. We consider three tasks (code completion, function naming and bug fixing) and re-implement different syntax-capturing modifications in a unified framework. We show that Transformers are able to make meaningful predictions based purely on syntactic information and underline the best practices of taking the syntactic information into account for improving the performance of the model.

Machine Learning,Computation and Language,Software Engineering
Knowledge Base Completion Using Matrix Factorization

Wenqiang He,Yansong Feng,Lei Zou,Dongyan Zhao

DOI: https://doi.org/10.1007/978-3-319-25255-1_21

2015-01-01

Abstract:With the development of Semantic Web, the automatic construction of large scale knowledge bases (KBs) has been receiving increasing attention in recent years. Although these KBs are very large, they are still often incomplete. Many existing approaches to KB completion focus on performing inference over a single KB and suffer from the feature sparsity problem. Moreover, traditional KB completion methods ignore complementarity which exists in various KBs implicitly. In this paper, we treat KBs completion as a large matrix completion task and integrate different KBs to infer new facts simultaneously. We present two improvements to the quality of inference over KBs. First, in order to reduce the data sparsity, we utilize the type consistency constraints between relations and entities to initialize negative data in the matrix. Secondly, we incorporate the similarity of relations between different KBs into matrix factorization model to take full advantage of the complementarity of various KBs. Experimental results show that our approach performs better than methods that consider only existing facts or only a single knowledge base, achieving significant accuracy improvements in binary relation prediction.
Relation classification via knowledge graph enhanced transformer encoder

Wenti Huang,Yiyu Mao,Zhan Yang,Lei Zhu,Jun Long

DOI: https://doi.org/10.1016/j.knosys.2020.106321

2020-10-01

Abstract:<p>Relation classification is an important task in natural language processing fields. The goal is to predict predefined relations for the marked nominal pairs in given sentences. State-of-the-art works usually focus on using deep neural networks as classifier to conduct the relation prediction. The rich semantic information of relationships in the triples of existing knowledge graph (KG) can be used as additional supervision for relation classification. However, these relationships were simply used as labels to specify the class of sentences in previous works, and their semantic information was completely ignored. In this paper, a novel approach is proposed for relation classification, which jointly uses information from textual sentences and knowledge graphs. To this end, we introduce a Transformer encoder to measure the semantic similarity between sentences and relation types. Besides, we connect the semantic information of marked nominals in sentences with that of the corresponding entities in knowledge graph to generate the semantic matching information between textual relations and KG relations. The matching information can provide additional supervision for relation classification. Since the words and entities are used interactively with each other in our work, we propose an embedding translating strategy to handle the semantic gap problem between word embeddings and entity embeddings. Experimental results on two widely used datasets, SemEval-2010 Task 8 and TACRED, show that our approach is able to efficiently use the semantic information from the knowledge graph to enhance the performance of the Transformer encoder for relation classification.</p>

computer science, artificial intelligence
Enhancing Transformer with Sememe Knowledge.

Yuhui Zhang,Chenghao Yang,Zhengping Zhou,Zhiyuan Liu

DOI: https://doi.org/10.18653/v1/2020.repl4nlp-1.21

2020-01-01

Abstract:While large-scale pretraining has achieved great success in many NLP tasks, it has not been fully studied whether external linguistic knowledge can improve data-driven models. In this work, we introduce sememe knowledge into Transformer and propose three sememe-enhanced Transformer models. Sememes, by linguistic definition, are the minimum semantic units of language, which can well represent implicit semantic meanings behind words. Our experiments demonstrate that introducing sememe knowledge into Transformer can consistently improve language modeling and downstream tasks. The adversarial test further demonstrates that sememe knowledge can substantially improve model robustness.
Knowledge-Based Semantic Embedding for Machine Translation

Chen Shi,Shujie Liu,Shuo Ren,Shi Feng,Mu Li,Ming Zhou,Xu Sun,Houfeng Wang

DOI: https://doi.org/10.18653/v1/p16-1212

2016-01-01

Abstract:In this paper, with the help of knowledge base, we build and formulate a semantic space to connect the source and target languages, and apply it to the sequence-to-sequence framework to propose a Knowledge-Based Semantic Embedding (KBSE) method. In our KB SE method, the source sentence is firstly mapped into a knowledge based semantic space, and the target sentence is generated using a recurrent neural network with the internal meaning preserved. Experiments are conducted on two translation tasks, the electric business data and movie data, and the results show that our proposed method can achieve outstanding performance, compared with both the traditional SMT methods and the existing encoder-decoder models.
Semantic-Enhanced Knowledge Graph Completion

Xu Yuan,Jiaxi Chen,Yingbo Wang,Anni Chen,Yiou Huang,Wenhong Zhao,Shuo Yu

DOI: https://doi.org/10.3390/math12030450

IF: 2.4

2024-02-01

Mathematics

Abstract:Knowledge graphs (KGs) serve as structured representations of knowledge, comprising entities and relations. KGs are inherently incomplete, sparse, and have a strong need for completion. Although many knowledge graph embedding models have been designed for knowledge graph completion, they predominantly focus on capturing observable correlations between entities. Due to the sparsity of KGs, potential semantic correlations are challenging to capture. To tackle this problem, we propose a model entitled semantic-enhanced knowledge graph completion (SE-KGC). SE-KGC effectively addresses the issue by incorporating predefined semantic patterns, enabling the capture of semantic correlations between entities and enhancing features for representation learning. To implement this approach, we employ a multi-relational graph convolution network encoder, which effectively encodes the KG. Subsequently, we utilize a scoring decoder to evaluate triplets. Experimental results demonstrate that our SE-KGC model outperforms other state-of-the-art methods in link-prediction tasks across three datasets. Specifically, compared to the baselines, SE-KGC achieved improvements of 11.7%, 1.05%, and 2.30% in terms of MRR on these three datasets. Furthermore, we present a comprehensive analysis of the contributions of different semantic patterns, and find that entities with higher connectivity play a pivotal role in effectively capturing and characterizing semantic information.

mathematics
A Multiple-Integration Encoder for Multi-Turn Text-to-SQL Semantic Parsing.

Run-Ze Wang,Zhen-Hua Ling,Jing-Bo Zhou,Yu Hu

DOI: https://doi.org/10.1109/taslp.2021.3070726

2021-01-01

Abstract:This paper studies multi-turn text-to-SQL generation, which is a new but important task in semantic parsing. In order to deal with its two challenges, i.e., multi-turn interaction and cross-domain evaluation, this paper proposes a multiple-integration encoder, which derives the vector representations of user utterances and database schemas using three custom-designed modules for information integration. First, an utterance representation enhancing module is built to integrate the information of history utterances into the representation of each token in current utterance by attentive selection. Second, a schema discrepancy enhancing module is designed to integrate previous predicted SQL query into the representation of schema items. Third, a latent schema linking module is employed to integrate schema information into utterance representations for better dealing with unseen database schemas. These three modules are all implemented based on a lightweight multi-head attention mechanism, which reduces the number of parameters in conventional multi-head attention. Experimental results on the SParC dataset show that our method achieved better accuracy of multi-turn text-to-SQL generation than the most advanced benchmarks. Further ablations studies and analysis also demonstrate the effectiveness of the three modules designed for information integration in the encoder.
Multi-Level Head-Wise Match and Aggregation in Transformer for Textual Sequence Matching.

Shuohang Wang,Yunshi Lan,Yi Tay,Jing Jiang,Jingjing Liu

DOI: https://doi.org/10.1609/aaai.v34i05.6458

2020-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Transformer has been successfully applied to many natural language processing tasks. However, for textual sequence matching, simple matching between the representation of a pair of sequences might bring in unnecessary noise. In this paper, we propose a new approach to sequence pair matching with Transformer, by learning head-wise matching representations on multiple levels. Experiments show that our proposed approach can achieve new state-of-the-art performance on multiple tasks that rely only on pre-computed sequence-vector-representation, such as SNLI, MNLI-match, MNLI-mismatch, QQP, and SQuAD-binary.
Semantic Interaction Matching Network for Few-shot Knowledge Graph Completion

Pengfei Luo,Xi Zhu,Tong Xu,Yi Zheng,Enhong Chen

DOI: https://doi.org/10.1145/3589557

IF: 3.35

2023-03-31

ACM Transactions on the Web

Abstract:The prosperity of knowledge graphs (KG), as well as related downstream applications, have raised the urgent request of knowledge graph completion techniques for fully supporting the KG reasoning tasks, especially under the circumstance of training data scarcity. Though large efforts have been made on solving this challenge via few-shot learning tools, they mainly focus on simply aggregating entity neighbors to represent few-shot references, while the enhancement from latent semantic correlation within neighbors has been largely ignored. To that end, in this paper, we propose a novel few-shot learning solution, named as Semantic Interaction Matching network (SIM), which applies Transformer framework to enhance the entity representation with capturing semantic interaction between entity neighbors. Specifically, we first design entity-relation fusion module to adaptively encode neighbors with incorporating relation representation. Along this line, Transformer layers are integrated to capture latent correlation within neighbors, as well as the semantic diversification of the support set. Finally, a similarity score is attentively estimated with attention mechanism. Extensive experiments on two public benchmark datasets demonstrate that our model outperforms a variety of state-of-the-art methods with a significant margin.

computer science, information systems, software engineering

Transformer-Based Siamese Textual Encoder for Knowledge Base Completion

Improving Knowledge Base Completion by Incorporating Implicit Information.

Generative Transformer with Knowledge-Guided Decoding for Academic Knowledge Graph Completion

kTrans: Knowledge-Aware Transformer for Binary Code Embedding

TE-TFN: A Text-enhanced Transformer Fusion Network for Multimodal Knowledge Graph Completion

Jointly Modeling Fact Triples and Text Information for Knowledge Base Completion.

Bidirectional transformer with knowledge graph for video captioning

Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching

DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection

Convolutional Network Embedding of Text-Enhanced Representation for Knowledge Graph Completion

Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

An Interpretable Knowledge Transfer Model for Knowledge Base Completion

Empirical Study of Transformers for Source Code

Knowledge Base Completion Using Matrix Factorization

Relation classification via knowledge graph enhanced transformer encoder

Enhancing Transformer with Sememe Knowledge.

Knowledge-Based Semantic Embedding for Machine Translation

Semantic-Enhanced Knowledge Graph Completion

A Multiple-Integration Encoder for Multi-Turn Text-to-SQL Semantic Parsing.

Multi-Level Head-Wise Match and Aggregation in Transformer for Textual Sequence Matching.

Semantic Interaction Matching Network for Few-shot Knowledge Graph Completion