[Color choice as psychodiagnostic method; use of the color pyramid test in school children].

C. Cazzullo,R. Montanini

Abstract:

What problem does this paper attempt to address?

Advancements in Natural language Processing: An In-depth Review of Language Transformer Models

Ritesh Kumar Singh

DOI: https://doi.org/10.22214/ijraset.2024.63408

2024-06-30

International Journal for Research in Applied Science and Engineering Technology

Abstract:Abstract: This review paper provides a succinct exploration of the history and impact of language transformer models in natural language processing (NLP). Beginning with a literature review, we trace the evolution of key models and elucidate fundamental concepts like self-attention mechanism and positional encoding. A comparative analysis of major transformer models, including BERT, GPT, T5, XLNet, offers insights into their architectures, strengths, and weaknesses. The discussion extends to pretraining objectives, fine-tuning strategies, and evaluation metrics, complemented by real-world examples of successful applications in NLP. We address current challenges, discuss potential future directions, and explore ethical considerations, providing valuable suggestions for researchers and practitioners in the NLP community in our conclusive summary.
Transformers in Natural Language Processing: A Comprehensive Review

Mohd Arsalan

DOI: https://doi.org/10.22214/ijraset.2024.62863

2024-05-31

International Journal for Research in Applied Science and Engineering Technology

Abstract:Abstract: This research paper provides a comprehensive review of transformers, a groundbreaking architecture in Natural Language Processing (NLP), and their impact on the field. We delve into the unique ability of transformers, with their attention mechanisms, to capture complex linguistic dependencies and process sequences effectively. This enables them to grasp contextual nuances, syntactic structures, and semantic intricacies inherent in human language, outperforming traditional models. This review explores the transformer architecture, various pre-trained language models based on it, and their diverse applications in multilingual NLP. We further discuss fine-tuning and transfer learning techniques for adapting transformers to specific tasks, recent developments, challenges, open problems, and future directions. This comprehensive analysis aims to contribute to a deeper understanding of how transformers have revolutionized NLP, offering insights into their potential for shaping the future of language understanding and generation
Overview of the Transformer-based Models for NLP Tasks

Anthony Gillioz,Jacky Casas,Elena Mugellini,Omar Abou Khaled

DOI: https://doi.org/10.15439/2020f20

2020-09-26

Abstract:In 2017, Vaswani et al. proposed a new neural network architecture named Transformer. That modern architecture quickly revolutionized the natural language processing world. Models like GPT and BERT relying on this Transformer architecture have fully outperformed the previous state-of-theart networks. It surpassed the earlier approaches by such a wide margin that all the recent cutting edge models seem to rely on these Transformer-based architectures. In this paper, we provide an overview and explanations of the latest models. We cover the auto-regressive models such as GPT, GPT-2 and XLNET, as well as the auto-encoder architecture such as BERT and a lot of post-BERT models like RoBERTa, ALBERT, ERNIE 1.0/2.0.
Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions

Gokul Yenduri,Ramalingam M,Chemmalar Selvi G,Supriya Y,Gautam Srivastava,Praveen Kumar Reddy Maddikunta,Deepti Raj G,Rutvij H Jhaveri,Prabadevi B,Weizheng Wang,Athanasios V. Vasilakos,Thippa Reddy Gadekallu

2023-05-21

Abstract:The Generative Pre-trained Transformer (GPT) represents a notable breakthrough in the domain of natural language processing, which is propelling us toward the development of machines that can understand and communicate using language in a manner that closely resembles that of humans. GPT is based on the transformer architecture, a deep neural network designed for natural language processing tasks. Due to their impressive performance on natural language processing tasks and ability to effectively converse, GPT have gained significant popularity among researchers and industrial communities, making them one of the most widely used and effective models in natural language processing and related fields, which motivated to conduct this review. This review provides a detailed overview of the GPT, including its architecture, working process, training procedures, enabling technologies, and its impact on various applications. In this review, we also explored the potential challenges and limitations of a GPT. Furthermore, we discuss potential solutions and future directions. Overall, this paper aims to provide a comprehensive understanding of GPT, enabling technologies, their impact on various applications, emerging challenges, and potential solutions.

Computation and Language,Artificial Intelligence
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions

Gokul Yenduri,M. Ramalingam,G. Chemmalar Selvi,Y. Supriya,Gautam Srivastava,Praveen Kumar Reddy Maddikunta,G. Deepti Raj,Rutvij H. Jhaveri,B. Prabadevi,Weizheng Wang,Athanasios V. Vasilakos,Thippa Reddy Gadekallu

DOI: https://doi.org/10.1109/access.2024.3389497

IF: 3.9

2024-04-30

IEEE Access

Abstract:The Generative Pre-trained Transformer (GPT) represents a notable breakthrough in the domain of natural language processing, which is propelling us toward the development of machines that can understand and communicate using language in a manner that closely resembles that of humans. GPT is based on the transformer architecture, a deep neural network designed for natural language processing tasks. Due to their impressive performance on natural language processing tasks and ability to effectively converse, GPT have gained significant popularity among researchers and industrial communities, making them one of the most widely used and effective models in natural language processing and related fields, which motivated to conduct this review. This review provides a detailed overview of the GPT, including its architecture, working process, training procedures, enabling technologies, and its impact on various applications. In this review, we also explored the potential challenges and limitations of a GPT. Furthermore, we discuss potential solutions and future directions. Overall, this paper aims to provide a comprehensive understanding of GPT, its enabling technologies, their impact on various applications, emerging challenges, and potential solutions.

computer science, information systems,telecommunications,engineering, electrical & electronic
Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet

M. Onat Topal,Anil Bas,Imke van Heerden

DOI: https://doi.org/10.48550/arXiv.2102.08036

2021-02-16

Computation and Language

Abstract:Recent years have seen a proliferation of attention mechanisms and the rise of Transformers in Natural Language Generation (NLG). Previously, state-of-the-art NLG architectures such as RNN and LSTM ran into vanishing gradient problems; as sentences grew larger, distance between positions remained linear, and sequential computation hindered parallelization since sentences were processed word by word. Transformers usher in a new era. In this paper, we explore three major Transformer-based models, namely GPT, BERT, and XLNet, that carry significant implications for the field. NLG is a burgeoning area that is now bolstered with rapid developments in attention mechanisms. From poetry generation to summarization, text generation derives benefit as Transformer-based language models achieve groundbreaking results.
Survey of transformers and towards ensemble learning using transformers for natural language processing

Hongzhi Zhang,M Omair Shafiq

DOI: https://doi.org/10.1186/s40537-023-00842-0

Abstract:The transformer model is a famous natural language processing model proposed by Google in 2017. Now, with the extensive development of deep learning, many natural language processing tasks can be solved by deep learning methods. After the BERT model was proposed, many pre-trained models such as the XLNet model, the RoBERTa model, and the ALBERT model were also proposed in the research community. These models perform very well in various natural language processing tasks. In this paper, we describe and compare these well-known models. In addition, we also apply several types of existing and well-known models which are the BERT model, the XLNet model, the RoBERTa model, the GPT2 model, and the ALBERT model to different existing and well-known natural language processing tasks, and analyze each model based on their performance. There are a few papers that comprehensively compare various transformer models. In our paper, we use six types of well-known tasks, such as sentiment analysis, question answering, text generation, text summarization, name entity recognition, and topic modeling tasks to compare the performance of various transformer models. In addition, using the existing models, we also propose ensemble learning models for the different natural language processing tasks. The results show that our ensemble learning models perform better than a single classifier on specific tasks.
Applications of transformer-based language models in bioinformatics: a survey

Shuang Zhang,Rui Fan,Yuti Liu,Shuang Chen,Qiao Liu,Wanwen Zeng

DOI: https://doi.org/10.1093/bioadv/vbad001

2023-01-01

Bioinformatics Advances

Abstract:Abstract Summary The transformer-based language models, including vanilla transformer, BERT and GPT-3, have achieved revolutionary breakthroughs in the field of natural language processing (NLP). Since there are inherent similarities between various biological sequences and natural languages, the remarkable interpretability and adaptability of these models have prompted a new wave of their application in bioinformatics research. To provide a timely and comprehensive review, we introduce key developments of transformer-based language models by describing the detailed structure of transformers and summarize their contribution to a wide range of bioinformatics research from basic sequence analysis to drug discovery. While transformer-based applications in bioinformatics are diverse and multifaceted, we identify and discuss the common challenges, including heterogeneity of training data, computational expense and model interpretability, and opportunities in the context of bioinformatics research. We hope that the broader community of NLP researchers, bioinformaticians and biologists will be brought together to foster future research and development in transformer-based language models, and inspire novel bioinformatics applications that are unattainable by traditional methods. Supplementary information Supplementary data are available at Bioinformatics Advances online.
Generative AI in the Era of Transformers: Revolutionizing Natural Language Processing with LLMs

Archna Balkrishna Yadav

DOI: https://doi.org/10.55529/jipirs.42.54.61

2024-03-07

Abstract:The advent of Transformer models is a transformational change in the field of Natural Language Processing (NLP), where technologies are becoming rather human-like in understanding and mirroring human language. This paper highlights the impact of Generative AI, specifically the Large Language Models such as GPT, on NLP. The analysis presents the prototypical units fuelling Transformer architectures, with attention given to their applications for complex language tasks and advantages from the angle of efficiency and scalability. However, the evidence highlights substantial progress in MT, text summarization, and SA versus the baseline NLP models. This work, therefore, emphasizes the key role of using a Transformer-based LLM system as a means to grow the NLP field and can lay the foundations for developing more natural and intuitive human-computer interactions.
Investigation of O-glycosylation heterogeneity of recombinant coagulation factor IX using LC-MS/MS.

Youngsuk Seo,G. Park,M. Oh,H. An

DOI: https://doi.org/10.4155/bio-2017-0086

IF: 2.695

2017-09-18

Bioanalysis

Abstract:AIM Recombinant coagulation factor IX (rFIX) has extraordinarily multiple post-translational modifications including N-glycosylation and O-glycosylation which have a drastic effect on biological functions and in vivo recovery. Unlike N-glycosylation extensively characterized, there are a few studies on O-glycosylation due to its intrinsic complexity. In-depth O-glycosylation analysis is necessary to better understand and assess pharmacological activity of rFIX. RESULTS We determined unusual O-glycosylations including O-fucosylation and O-glucosylation which were located at Serine 53 and 61, respectively in EGF domain. Other O-glycosylations bearing core 1 glycan moiety were found on activation peptide. CONCLUSION This is the first comprehensive study to characterize O-glycosylation of rFIX using MS-based glycomic and glycoproteomic approaches. Site-specific profiling will be a powerful platform to determine bioequivalence of biosimilars.
Advancements and Methodologies in Natural Language Processing and Machine Learning: A Comprehensive Review

Kumar Pritam

DOI: https://doi.org/10.22214/ijraset.2024.63359

2024-06-30

International Journal for Research in Applied Science and Engineering Technology

Abstract:Abstract: This review explores notable advancements and methodologies in Natural Language Processing (NLP) and machine learning. Key innovations include convolutional neural networks capable of generating multiple language processing predictions and the Transformer-XL model, which enhances temporal coherence and dependency handling. The importance of logical reasoning and common sense in AI is emphasized, along with practical methods like web mining for building extensive knowledge repositories. The development of NLP tools for specific languages, such as Gujarati, and the utilization of BERT for bidirectional training are discussed, showcasing significant performance improvements across multiple tasks. Sentiment analysis is highlighted as a crucial tool for interpreting unstructured data from social media, achieving high accuracy through rule-based approaches and feature extraction methods. The role of social media in disseminating emotionally charged content and its impact on user behavior is examined, providing valuable insights for businesses to enhance marketing strategies. Additionally, the review addresses the challenge of early risk detection on social media platforms, emphasizing the need for effective preprocessing techniques to handle diverse and unstructured content. The integration of deep learning techniques in sentiment analysis is shown to offer competitive advantages, revealing consumer preferences and product trends.
A Survey on Transformers in NLP with Focus on Efficiency

Wazib Ansar,Saptarsi Goswami,Amlan Chakrabarti

2024-05-15

Abstract:The advent of transformers with attention mechanisms and associated pre-trained models have revolutionized the field of Natural Language Processing (NLP). However, such models are resource-intensive due to highly complex architecture. This limits their application to resource-constrained environments. While choosing an appropriate NLP model, a major trade-off exists over choosing accuracy over efficiency and vice versa. This paper presents a commentary on the evolution of NLP and its applications with emphasis on their accuracy as-well-as efficiency. Following this, a survey of research contributions towards enhancing the efficiency of transformer-based models at various stages of model development along with hardware considerations has been conducted. The goal of this survey is to determine how current NLP techniques contribute towards a sustainable society and to establish a foundation for future research.

Computation and Language,Artificial Intelligence
A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships

Gracile Astlin Pereira,Muhammad Hussain

2024-08-28

Abstract:Transformer-based models have transformed the landscape of natural language processing (NLP) and are increasingly applied to computer vision tasks with remarkable success. These models, renowned for their ability to capture long-range dependencies and contextual information, offer a promising alternative to traditional convolutional neural networks (CNNs) in computer vision. In this review paper, we provide an extensive overview of various transformer architectures adapted for computer vision tasks. We delve into how these models capture global context and spatial relationships in images, empowering them to excel in tasks such as image classification, object detection, and segmentation. Analyzing the key components, training methodologies, and performance metrics of transformer-based models, we highlight their strengths, limitations, and recent advancements. Additionally, we discuss potential research directions and applications of transformer-based models in computer vision, offering insights into their implications for future advancements in the field.

Computer Vision and Pattern Recognition
Transformers and large language models in healthcare: A review

Subhash Nerella,Sabyasachi Bandyopadhyay,Jiaqing Zhang,Miguel Contreras,Scott Siegel,Aysegul Bumin,Brandon Silva,Jessica Sena,Benjamin Shickel,Azra Bihorac,Kia Khezeli,Parisa Rashidi

DOI: https://doi.org/10.1016/j.artmed.2024.102900

Abstract:With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning architecture initially developed to solve general-purpose Natural Language Processing (NLP) tasks and has subsequently been adapted in many fields, including healthcare. In this survey paper, we provide an overview of how this architecture has been adopted to analyze various forms of healthcare data, including clinical NLP, medical imaging, structured Electronic Health Records (EHR), social media, bio-physiological signals, biomolecular sequences. Furthermore, which have also include the articles that used the transformer architecture for generating surgical instructions and predicting adverse outcomes after surgeries under the umbrella of critical care. Under diverse settings, these models have been used for clinical diagnosis, report generation, data reconstruction, and drug/protein synthesis. Finally, we also discuss the benefits and limitations of using transformers in healthcare and examine issues such as computational cost, model interpretability, fairness, alignment with human values, ethical implications, and environmental impact.
A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis

Nimisha Ghosh,Daniele Santoni,Indrajit Saha,Giovanni Felici

2024-12-10

Abstract:In recent times, Transformer-based language models are making quite an impact in the field of natural language processing. As relevant parallels can be drawn between biological sequences and natural languages, the models used in NLP can be easily extended and adapted for various applications in bioinformatics. In this regard, this paper introduces the major developments of Transformer-based models in the recent past in the context of nucleotide sequences. We have reviewed and analysed a large number of application-based papers on this subject, giving evidence of the main characterizing features and to different approaches that may be adopted to customize such powerful computational machines. We have also provided a structured description of the functioning of Transformers, that may enable even first time users to grab the essence of such complex architectures. We believe this review will help the scientific community in understanding the various applications of Transformer-based language models to nucleotide sequences. This work will motivate the readers to build on these methodologies to tackle also various other problems in the field of bioinformatics.

Computation and Language,Artificial Intelligence
Transformers for scientific data: a pedagogical review for astronomers

Dimitrios Tanoglidis,Bhuvnesh Jain,Helen Qu

2023-10-19

Abstract:The deep learning architecture associated with ChatGPT and related generative AI products is known as transformers. Initially applied to Natural Language Processing, transformers and the self-attention mechanism they exploit have gained widespread interest across the natural sciences. The goal of this pedagogical and informal review is to introduce transformers to scientists. The review includes the mathematics underlying the attention mechanism, a description of the original transformer architecture, and a section on applications to time series and imaging data in astronomy. We include a Frequently Asked Questions section for readers who are curious about generative AI or interested in getting started with transformers for their research problem.

Instrumentation and Methods for Astrophysics,Machine Learning
From Turing to Transformers: A Comprehensive Review and Tutorial on the Evolution and Applications of Generative Transformer Models

Emma Yann Zhang,Adrian David Cheok,Zhigeng Pan,Jun Cai,Ying Yan

DOI: https://doi.org/10.3390/sci5040046

2023-12-15

Sci

Abstract:In recent years, generative transformers have become increasingly prevalent in the field of artificial intelligence, especially within the scope of natural language processing. This paper provides a comprehensive overview of these models, beginning with the foundational theories introduced by Alan Turing and extending to contemporary generative transformer architectures. The manuscript serves as a review, historical account, and tutorial, aiming to offer a thorough understanding of the models’ importance, underlying principles, and wide-ranging applications. The tutorial section includes a practical guide for constructing a basic generative transformer model. Additionally, the paper addresses the challenges, ethical implications, and future directions in the study of generative models.

English Else
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

Saidul Islam,Hanae Elmekki,Ahmed Elsebai,Jamal Bentahar,Najat Drawel,Gaith Rjoub,Witold Pedrycz

DOI: https://doi.org/10.48550/arXiv.2306.07303

IF: 5.414

2023-06-11

Machine Learning

Abstract:Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology.
Partial Tensorized Transformers for Natural Language Processing

Subhadra Vadlamannati,Ryan Solgi

2023-10-31

Abstract:The transformer architecture has revolutionized Natural Language Processing (NLP) and other machine-learning tasks, due to its unprecedented accuracy. However, their extensive memory and parameter requirements often hinder their practical applications. In this work, we study the effect of tensor-train decomposition to improve the accuracy and compress transformer vision-language neural networks, namely BERT and ViT. We focus both on embedding-layer compression and partial tensorization of neural networks (PTNN) through an algorithmic approach. Our novel PTNN approach significantly improves the accuracy of existing models by up to 5%, all without the need for post-training adjustments, breaking new ground in the field of tensor decomposition.

Computation and Language,Machine Learning
Transformers in Machine Learning: Literature Review

T Thoyyibah,Wasis Haryono,Achmad Udin Zailani,Yan Mitha Djaksana,Neny Rosmawarni,Nunik Destria Arianti,Thoyyibah T

DOI: https://doi.org/10.29303/jppipa.v9i9.5040

2023-09-25

Jurnal Penelitian Pendidikan IPA

Abstract:In this study, the researcher presents an approach regarding methods in Transformer Machine Learning. Initially, transformers are neural network architectures that are considered as inputs. Transformers are widely used in various studies with various objects. The transformer is one of the deep learning architectures that can be modified. Transformers are also mechanisms that study contextual relationships between words. Transformers are used for text compression in readings. Transformers are used to recognize chemical images with an accuracy rate of 96%. Transformers are used to detect a person's emotions. Transformer to detect emotions in social media conversations, for example, on Facebook with happy, sad, and angry categories. Figure 1 illustrates the encoder and decoder process through the input process and produces output. the purpose of this study is to only review literature from various journals that discuss transformers. This explanation is also done by presenting the subject or dataset, data analysis method, year, and accuracy achieved. By using the methods presented, researchers can conclude results in search of the highest accuracy and opportunities for further research.

[Color choice as psychodiagnostic method; use of the color pyramid test in school children].

Advancements in Natural language Processing: An In-depth Review of Language Transformer Models

Transformers in Natural Language Processing: A Comprehensive Review

Overview of the Transformer-based Models for NLP Tasks

Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions

GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions

Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet

Survey of transformers and towards ensemble learning using transformers for natural language processing

Applications of transformer-based language models in bioinformatics: a survey

Generative AI in the Era of Transformers: Revolutionizing Natural Language Processing with LLMs

Investigation of O-glycosylation heterogeneity of recombinant coagulation factor IX using LC-MS/MS.

Advancements and Methodologies in Natural Language Processing and Machine Learning: A Comprehensive Review

A Survey on Transformers in NLP with Focus on Efficiency

A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships

Transformers and large language models in healthcare: A review

A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis

Transformers for scientific data: a pedagogical review for astronomers

From Turing to Transformers: A Comprehensive Review and Tutorial on the Evolution and Applications of Generative Transformer Models

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

Partial Tensorized Transformers for Natural Language Processing

Transformers in Machine Learning: Literature Review