Mutually-paced Knowledge Distillation for Cross-lingual Temporal Knowledge Graph Reasoning

Ruijie Wang,Zheng Li,Jingfeng Yang,Tianyu Cao,Chao Zhang,Bing Yin,Tarek Abdelzaher

DOI: https://doi.org/10.48550/arXiv.2303.14898

2023-03-27

Abstract:This paper investigates cross-lingual temporal knowledge graph reasoning problem, which aims to facilitate reasoning on Temporal Knowledge Graphs (TKGs) in low-resource languages by transfering knowledge from TKGs in high-resource ones. The cross-lingual distillation ability across TKGs becomes increasingly crucial, in light of the unsatisfying performance of existing reasoning methods on those severely incomplete TKGs, especially in low-resource languages. However, it poses tremendous challenges in two aspects. First, the cross-lingual alignments, which serve as bridges for knowledge transfer, are usually too scarce to transfer sufficient knowledge between two TKGs. Second, temporal knowledge discrepancy of the aligned entities, especially when alignments are unreliable, can mislead the knowledge distillation process. We correspondingly propose a mutually-paced knowledge distillation model MP-KD, where a teacher network trained on a source TKG can guide the training of a student network on target TKGs with an alignment module. Concretely, to deal with the scarcity issue, MP-KD generates pseudo alignments between TKGs based on the temporal information extracted by our representation module. To maximize the efficacy of knowledge transfer and control the noise caused by the temporal knowledge discrepancy, we enhance MP-KD with a temporal cross-lingual attention mechanism to dynamically estimate the alignment strength. The two procedures are mutually paced along with model training. Extensive experiments on twelve cross-lingual TKG transfer tasks in the EventKG benchmark demonstrate the effectiveness of the proposed MP-KD method.

Machine Learning,Artificial Intelligence,Computation and Language,Social and Information Networks

What problem does this paper attempt to address?

The paper aims to address the problem of Cross-lingual Temporal Knowledge Graph Reasoning. Specifically, the main objective of the research is to enhance reasoning performance in low-resource language environments by transferring knowledge from Temporal Knowledge Graphs (TKGs) in high-resource languages to TKGs in low-resource languages.

The paper points out that existing methods perform poorly when dealing with severely incomplete TKGs, especially in low-resource languages. To this end, the authors propose a method called "Mutually-paced Knowledge Distillation" (MP-KD). MP-KD primarily addresses two key challenges:

1. **Scarcity of Cross-lingual Alignments**: To alleviate this issue, MP-KD extends the upper limit of knowledge transfer by generating pseudo alignments, and this process is synchronized with model training.

2. **Temporal Knowledge Discrepancies**: To mitigate the negative impact of these discrepancies, MP-KD employs a method of dynamically adjusting alignment strength, estimating the temporal alignment strength distribution of the graph structure through a spatiotemporal attention mechanism.

Through the aforementioned methods, MP-KD is able to achieve more efficient knowledge transfer in cross-lingual temporal knowledge graphs, and its effectiveness is validated through experiments.