Abstract:Semantic representation and inference is essential for Natural Language Processing (NLP). The state of the art for semantic representation and inference is deep learning, and particularly Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and transformer Self-Attention models. This thesis investigates the use of deep learning for novel semantic representation and inference, and makes contributions in the following three areas: creating training data, improving semantic representations and extending inference learning. In terms of creating training data, we contribute the largest publicly available dataset of real-life factual claims for the purpose of automatic claim verification (MultiFC), and we present a novel inference model composed of multi-scale CNNs with different kernel sizes that learn from external sources to infer fact checking labels. In terms of improving semantic representations, we contribute a novel model that captures non-compositional semantic indicators. By definition, the meaning of a non-compositional phrase cannot be inferred from the individual meanings of its composing words (e.g., hot dog). Motivated by this, we operationalize the compositionality of a phrase contextually by enriching the phrase representation with external word embeddings and knowledge graphs. Finally, in terms of inference learning, we propose a series of novel deep learning architectures that improve inference by using syntactic dependencies, by ensembling role guided attention heads, incorporating gating layers, and concatenating multiple heads in novel and effective ways. This thesis consists of seven publications (five published and two under review).

Evaluating Distributed Representations for Multi-Level Lexical Semantics: A Research Proposal

A Model of Coherence Based on Distributed Sentence Representation.

Visualizing and Understanding Neural Models in NLP

Semantic Representation and Inference for NLP

Distributed Representation of Semantics in the Human Brain: Evidence from Studies Using Natural Language Processing Techniques

Leveraging Multi-Level Semantic Understanding in a Unified NER Model

More than Syntaxes: Investigating Semantics to Zero-shot Cross-lingual Relation Extraction and Event Argument Role Labelling

Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks

RUM: network Representation learning throUgh Multi-level structural information preservation

Incorporating Linguistic Knowledge for Learning Distributed Word Representations.

Semantic Processing Through Distributed Representation of Chinese Words

Neuron-level Interpretation of Deep NLP Models: A Survey

Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics

Semantic Regularities in Document Representations.

Learning Task Specific Distributed Paragraph Representations Using a 2-Tier Convolutional Neural Network

Sentence Representation Method Based on Multi-Layer Semantic Network

Representations Using a Two-Tier Convolutional Neural Network

Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations

Semantic Hilbert Space for Text Representation Learning

Frame-based Multi-level Semantics Representation for text matching

A 3-Layer Structural Analysis Framework for Chinese Text Understanding