Notes on Deep Learning for NLP

Antoine J. -P. Tixier,Antoine J.-P. Tixier

DOI: https://doi.org/10.48550/arXiv.1808.09772

2018-08-29

Computation and Language

Abstract:My notes on Deep Learning for NLP.

What problem does this paper attempt to address?

Deep Reinforcement Learning for NLP.

William Yang Wang,Jiwei Li,Xiaodong He

DOI: https://doi.org/10.18653/v1/p18-5007

2018-01-01

Abstract:Many Natural Language Processing (NLP) tasks (including generation, language grounding, reasoning, information extraction, coreference resolution, and dialog) can be formulated as deep reinforcement learning (DRL) problems. However, since language is often discrete and the space for all sentences is infinite, there are many challenges for formulating reinforcement learning problems of NLP tasks. In this tutorial, we provide a gentle introduction to the foundation of deep reinforcement learning, as well as some practical DRL solutions in NLP. We describe recent advances in designing deep reinforcement learning for NLP, with a special focus on generation, dialogue, and information extraction. Finally, we discuss why they succeed, and when they may fail, aiming at providing some practical advice about deep reinforcement learning for solving real-world NLP problems.
Deep Adversarial Learning for NLP.

William Yang Wang,Sameer Singh,Jiwei Li

DOI: https://doi.org/10.18653/v1/n19-5001

2019-01-01

Abstract:Adversarial learning is a game-theoretic learning paradigm, which has achieved huge successes in the field of Computer Vision recently. Adversarial learning is also a general framework that enables a variety of learning models, including the popular Generative Adversarial Networks (GANs). Due to the discrete nature of language, designing adversarial learning models is still challenging for NLP problems. In this tutorial, we provide a gentle introduction to the foundation of deep adversarial learning, as well as some practical problem formulations and solutions in NLP. We describe recent advances in deep adversarial learning for NLP, with a special focus on generation, adversarial examples & rules, and dialogue. We provide an overview of the research area, categorize different types of adversarial learning models, and discuss pros and cons, aiming at providing some practical perspectives on the future of adversarial learning for solving real-world NLP problems.
Notes on Deep Learning Theory

Eugene A. Golikov

DOI: https://doi.org/10.48550/arXiv.2012.05760

2020-12-10

Abstract:These are the notes for the lectures that I was giving during Fall 2020 at the Moscow Institute of Physics and Technology (MIPT) and at the Yandex School of Data Analysis (YSDA). The notes cover some aspects of initialization, loss landscape, generalization, and a neural tangent kernel theory. While many other topics (e.g. expressivity, a mean-field theory, a double descent phenomenon) are missing in the current version, we plan to add them in future revisions.

Machine Learning,Artificial Intelligence
Automatic Document Summarization Via Deep Neural Networks

Chengwei Yao,Jianfen Shen,Gencai Chen

DOI: https://doi.org/10.1109/iscid.2015.83

2015-01-01

Abstract:Automatic document summarization aim to extracting sentences which might cover the main content of a document or documents. To achieve this, many algorithms have been tried to rank the sentences by using task-specific features in a shallow architecture. The main challenge of these approaches is to keep balance between information coverage and redundancy because of absence of discovering the intrinsic semantic representation. Inspired by the recent successful achievement of Deep Learning, this paper proposes a new framework of document summarization via Deep Neural Networks (DNNs). Specifically, we feed the sentences as the input to the visible layer of DNNs. After pretraining layer by layer and fine-tuning, the lower dimensional semantic space can be revealed. Based on this space, we design sentences extraction algorithm to construct the summary. Experiments on the DUC2006 and DUC2007 dataset show that our framework works better than state-of-the-art methods.
An introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools

Ivano Lauriola,Alberto Lavelli,Fabio Aiolli

DOI: https://doi.org/10.1016/j.neucom.2021.05.103

IF: 6

2022-01-01

Neurocomputing

Abstract:Natural Language Processing (NLP) is a branch of artificial intelligence that involves the design and implementation of systems and algorithms able to interact through human language. Thanks to the recent advances of deep learning, NLP applications have received an unprecedented boost in performance. In this paper, we present a survey of the application of deep learning techniques in NLP, with a focus on the various tasks where deep learning is demonstrating stronger impact. Additionally, we explore, describe, and revise the main resources in NLP research, including software, hardware, and popular corpora. Finally, we emphasize the main limits of deep learning in NLP and current research directions.

computer science, artificial intelligence
A Joint Introduction to Natural Language Processing and to Deep Learning

Li Deng,Yang Liu

DOI: https://doi.org/10.1007/978-981-10-5209-5_1

2018-01-01

Abstract:In this chapter, we set up the fundamental framework for the book. We first provide an introduction to the basics of natural language processing (NLP) as an integral part of artificial intelligence. We then survey the historical development of NLP, spanning over five decades, in terms of three waves. The first two waves arose as rationalism and empiricism, paving ways to the current deep learning wave. The key pillars underlying the deep learning revolution for NLP consist of (1) distributed representations of linguistic entities via embedding, (2) semantic generalization due to the embedding, (3) long-span deep sequence modeling of natural language, (4) hierarchical networks effective for representing linguistic levels from low to high, and (5) end-to-end deep learning methods to jointly solve many NLP tasks. After the survey, several key limitations of current deep learning technology for NLP are analyzed. This …
Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application

Keyu Chen,Cheng Fei,Ziqian Bi,Junyu Liu,Benji Peng,Sen Zhang,Xuanhe Pan,Jiawei Xu,Jinlang Wang,Caitlyn Heqi Yin,Yichao Zhang,Pohsun Feng,Yizhu Wen,Tianyang Wang,Ming Li,Jintao Ren,Qian Niu,Silin Chen,Weiche Hsieh,Lawrence K.Q. Yan,Chia Xin Liang,Han Xu,Hong-Ming Tseng,Xinyuan Song,Ming Liu

2024-10-30

Abstract:With a focus on natural language processing (NLP) and the role of large language models (LLMs), we explore the intersection of machine learning, deep learning, and artificial intelligence. As artificial intelligence continues to revolutionize fields from healthcare to finance, NLP techniques such as tokenization, text classification, and entity recognition are essential for processing and understanding human language. This paper discusses advanced data preprocessing techniques and the use of frameworks like Hugging Face for implementing transformer-based models. Additionally, it highlights challenges such as handling multilingual data, reducing bias, and ensuring model robustness. By addressing key aspects of data processing and model fine-tuning, this work aims to provide insights into deploying effective and ethically sound AI solutions.

Computation and Language,Human-Computer Interaction
Some Ideas on Natural Language Processing

Liu Haita

DOI: https://doi.org/10.3969/j.issn.1007-2489.2001.01.007

2001-01-01

Abstract:Natural language processing (NLP) is the necessity of human development in information age, it is the natural chain of the human history. Therefor we should explore and research it from many fields, and not only independently from computational technology.
Epilogue: Frontiers of NLP in the Deep Learning Era

Li Deng,Yang Liu

DOI: https://doi.org/10.1007/978-981-10-5209-5_11

2018-01-01

Abstract:In the first part of this epilogue, we summarize the book holistically from two perspectives. The first, task-centric perspective ties together and categories a wide range of NLP techniques discussed in book in terms of general machine learning paradigms. In this way, the majority of sections and chapters of the book can be naturally clustered into four classes: classification, sequence-based prediction, higher-order structured prediction, and sequential decision-making. The second, representation-centric perspective distills insight from holistically analyzed book chapters from cognitive science viewpoints and in terms of two basic types of natural language representations: symbolic and distributed representations. In the second part of the epilogue, we update the most recent progress on deep learning in NLP (mainly during the later part of 2017, not surveyed in earlier chapters). Based on our reviews of these …
Deep Learning in Natural Language Processing: A State-of-the-Art Survey

Junyi Chai,Anming Li

DOI: https://doi.org/10.1109/ICMLC48188.2019.8949185

2019-01-01

Abstract:Deep learning raises interests of research community as their overwhelming successes in information processing such specific tasks as video/speech recognition. In this paper, we provide a state-of-the-art analysis of deep learning with its applications in an important direction: natural language processing. We attempt to provide a clear and critical summarization for researchers and participators who are interested in incorporating the deep learning techniques in their specific domains.
Deep Learning and Its Applications to Natural Language Processing

Haiqin Yang,Linkai Luo,Lap Pong Chueng,David Ling,Francis Chin

DOI: https://doi.org/10.1007/978-3-030-06073-2_4

2019-01-01

Abstract:Natural language processing (NLP), utilizing computer programs to process large amounts of language data, is a key research area in artificial intelligence and computer science. Deep learning technologies have been well developed and applied in this area. However, the literature still lacks a succinct survey, which would allow readers to get a quick understanding of (1) how the deep learning technologies apply to NLP and (2) what the promising applications are. In this survey, we try to investigate the recent developments of NLP, centered around natural language understanding, to answer these two questions. First, we explore the newly developed word embedding or word representation methods. Then, we describe two powerful learning models, Recurrent Neural Networks and Convolutional Neural Networks. Next, we outline five key NLP applications, including (1) part-of-speech tagging and named entity recognition, two fundamental NLP applications; (2) machine translation and automatic English grammatical error correction, two applications with prominent commercial value; and (3) image description, an application requiring technologies of both computer vision and NLP. Moreover, we present a series of benchmark datasets which would be useful for researchers to evaluate the performance of models in the related applications.
Lecture Notes on Neural Information Retrieval

Nicola Tonellotto

DOI: https://doi.org/10.48550/arXiv.2207.13443

2022-07-27

Information Retrieval

Abstract:These lecture notes focus on the recent advancements in neural information retrieval, with particular emphasis on the systems and models exploiting transformer networks. These networks, originally proposed by Google in 2017, have seen a large success in many natural language processing and information retrieval tasks. While there are many fantastic textbook on information retrieval and natural language processing as well as specialised books for a more advanced audience, these lecture notes target people aiming at developing a basic understanding of the main information retrieval techniques and approaches based on deep learning. These notes have been prepared for a IR graduate course of the MSc program in Artificial Intelligence and Data Engineering at the University of Pisa, Italy.
Natural Language Processing (Almost) from Scratch

Michael Karlen,K. Kavukcuoglu,J. Weston,L. Bottou,R. Collobert,Pavel P. Kuksa

DOI: https://doi.org/10.5555/1953048.2078186

2011-02-01

Journal of Machine Learning Research

Abstract:We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

Computer Science,Linguistics
Du TAL au TIL

Michael Zock,Guy Lapalme

DOI: https://doi.org/10.48550/arXiv.1201.4733

2012-01-21

Abstract:Historically two types of NLP have been investigated: fully automated processing of language by machines (NLP) and autonomous processing of natural language by people, i.e. the human brain (psycholinguistics). We believe that there is room and need for another kind, INLP: interactive natural language processing. This intermediate approach starts from peoples' needs, trying to bridge the gap between their actual knowledge and a given goal. Given the fact that peoples' knowledge is variable and often incomplete, the aim is to build bridges linking a given knowledge state to a given goal. We present some examples, trying to show that this goal is worth pursuing, achievable and at a reasonable cost.

Computation and Language,Human-Computer Interaction
Applied Language Technology: NLP for the Humanities

Tuomo Hiippala

DOI: https://doi.org/10.18653/v1/2021.teachingnlp-1.5

2021-05-04

Abstract:This contribution describes a two-course module that seeks to provide humanities majors with a basic understanding of language technology and its applications using Python. The learning materials consist of interactive Jupyter Notebooks and accompanying YouTube videos, which are openly available with a Creative Commons licence.

Computation and Language
Annotated History of Modern AI and Deep Learning

Juergen Schmidhuber

DOI: https://doi.org/10.48550/arXiv.2212.11279

2022-12-21

Neural and Evolutionary Computing

Abstract:Machine learning is the science of credit assignment: finding patterns in observations that predict the consequences of actions and help to improve future performance. Credit assignment is also required for human understanding of how the world works, not only for individuals navigating daily life, but also for academic professionals like historians who interpret the present in light of past events. Here I focus on the history of modern artificial intelligence (AI) which is dominated by artificial neural networks (NNs) and deep learning, both conceptually closer to the old field of cybernetics than to what's been called AI since 1956 (e.g., expert systems and logic programming). A modern history of AI will emphasize breakthroughs outside of the focus of traditional AI text books, in particular, mathematical foundations of today's NNs such as the chain rule (1676), the first NNs (linear regression, circa 1800), and the first working deep learners (1965-). From the perspective of 2022, I provide a timeline of the -- in hindsight -- most important relevant events in the history of NNs, deep learning, AI, computer science, and mathematics in general, crediting those who laid foundations of the field. The text contains numerous hyperlinks to relevant overview sites from my AI Blog. It supplements my previous deep learning survey (2015) which provides hundreds of additional references. Finally, to round it off, I'll put things in a broader historic context spanning the time since the Big Bang until when the universe will be many times older than it is now.
Exploring the frontiers of deep learning and natural language processing: A comprehensive overview of key challenges and emerging trends

Wahab Khan,Ali Daud,Khairullah Khan,Shakoor Muhammad,Rafiul Haq

DOI: https://doi.org/10.1016/j.nlp.2023.100026

2023-09-01

Natural Language Processing Journal

Abstract:In the recent past, more than 5 years or so, Deep Learning (DL) especially the large language models (LLMs) has generated extensive studies out of a distinctly average downturn field of knowledge made up of a traditional society of researchers. As a result, DL is now so pervasive that its use is widespread across the body of research related to machine learning computing. The rapid emergence and apparent dominance of DL architectures over traditional machine learning techniques on a variety of tasks have been truly astonishing to witness. DL models outperformed in a variety of areas, including natural language processing (NLP), image analysis, language understanding, machine translation, computer vision, speech processing, audio recognition, style imitation, and computational biology. In this study, the aim is to explain the rudiments of DL, such as neural networks, convolutional neural networks, deep belief networks, and various variants of DL. The study will explore how these models have been applied to NLP and delve into the underlying mathematics behind them. Additionally, the study will investigate the latest advancements in DL and NLP, while acknowledging the key challenges and emerging trends in the field. Furthermore, it will discuss the core component of DL, namely embeddings, from a taxonomic perspective. Moreover, a literature review will be provided focusing on the application of DL models for six popular pattern recognition tasks: speech recognition, question answering, part of speech tagging, named entity recognition, text classification, and machine translation. Finally, the study will demystify state-of-the-art DL libraries/frameworks and available resources. The outcome and implication of this study reveal that LLMs face challenges in dealing with pragmatic aspects of language due to their reliance on statistical learning techniques and lack of genuine understanding of context, presupposition, implicature, and social norms. Furthermore, this study provides a comprehensive analysis of the current state-of-the-art advancements and highlights significant obstacles and emerging developments. The article has the potential to enhance readers’ understanding of the subject matter.
Deep Learning Models for Automatic Summarization

Pirmin Lemberger

DOI: https://doi.org/10.48550/arXiv.2005.11988

2020-05-25

Abstract:Text summarization is an NLP task which aims to convert a textual document into a shorter one while keeping as much meaning as possible. This pedagogical article reviews a number of recent Deep Learning architectures that have helped to advance research in this field. We will discuss in particular applications of pointer networks, hierarchical Transformers and Reinforcement Learning. We assume basic knowledge of Seq2Seq architecture and Transformer networks within NLP.

Computation and Language,Machine Learning
Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning

Nadav Cohen,Noam Razin

2024-08-25

Abstract:These notes are based on a lecture delivered by NC on March 2021, as part of an advanced course in Princeton University on the mathematical understanding of deep learning. They present a theory (developed by NC, NR and collaborators) of linear neural networks -- a fundamental model in the study of optimization and generalization in deep learning. Practical applications born from the presented theory are also discussed. The theory is based on mathematical tools that are dynamical in nature. It showcases the potential of such tools to push the envelope of our understanding of optimization and generalization in deep learning. The text assumes familiarity with the basics of statistical learning theory. Exercises (without solutions) are included.

Machine Learning,Artificial Intelligence
A Comprehensive Survey of Deep Learning Techniques Natural Language Processing

Jasmin Praful Bharadiya

DOI: https://doi.org/10.47672/ejt.1473

2023-05-23

European Journal of Technology

Abstract:In NLP research, unsupervised or semi-supervised learning techniques are increasingly getting more attention. These learning techniques are capable of learning from data that has not been manually annotated with the necessary answers or by combining non-annotated and annotated data. This essay presents a survey of various natural language processing methods. The discipline of natural language processing, which integrates linguistics, artificial intelligence, and computer science, was established to make it easier for computers and human language to communicate with one another. It is, as we can say, relevant psychopathology for the study of computer-human interaction. The understanding of natural language, which entails enabling machines to naturally interpret human language, is one of the many challenges this area faces. Discourse analysis, morphological separation, machine translation, production and understanding of NLP, part-of-speech tagging, recognition of optical characters, speech recognition, and sentiment analysis are some of the most frequent NLP tasks. As opposed to learning, which is supervised and typically yields few correct results for a given amount of input data, this job is typically quite difficult. However, there is a sizable amount of data available that is unannotated in nature, i.e. the entire contents are available on the internet, and it typically yields less accurate findings.

Notes on Deep Learning for NLP

Deep Reinforcement Learning for NLP.

Deep Adversarial Learning for NLP.

Notes on Deep Learning Theory

Automatic Document Summarization Via Deep Neural Networks

An introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools

A Joint Introduction to Natural Language Processing and to Deep Learning

Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application

Some Ideas on Natural Language Processing

Epilogue: Frontiers of NLP in the Deep Learning Era

Deep Learning in Natural Language Processing: A State-of-the-Art Survey

Deep Learning and Its Applications to Natural Language Processing

Lecture Notes on Neural Information Retrieval

Natural Language Processing (Almost) from Scratch

Du TAL au TIL

Applied Language Technology: NLP for the Humanities

Annotated History of Modern AI and Deep Learning

Exploring the frontiers of deep learning and natural language processing: A comprehensive overview of key challenges and emerging trends

Deep Learning Models for Automatic Summarization

Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning

A Comprehensive Survey of Deep Learning Techniques Natural Language Processing