Abstract:Many applications today, such as NLP, network analysis, and code analysis, rely on semantically embedding objects into low-dimensional fixed-length vectors. Such embeddings naturally provide a way to perform useful downstream tasks, such as identifying relations among objects or predicting objects for a given context, etc. Unfortunately, the training necessary for accurate embeddings is usually computationally intensive and requires processing large amounts of data. Furthermore, distributing this training is challenging. Most embedding training uses stochastic gradient descent (SGD), an "inherently" sequential algorithm. Prior approaches to parallelizing SGD do not honor these dependencies and thus potentially suffer poor convergence. This paper presents a distributed training framework for a class of applications that use Skip-gram-like models to generate embeddings. We call this class Any2Vec and it includes Word2Vec, DeepWalk, and Node2Vec among others. We first formulate Any2Vec training algorithm as a graph application and leverage the state-of-the-art distributed graph analytics framework, D-Galois. We adapt D-Galois to support dynamic graph generation and repartitioning, and incorporate novel communication optimizations. Finally, we introduce a novel way to combine gradients during distributed training to prevent accuracy loss. We show that our framework, called GraphAny2Vec, matches on a cluster of 32 hosts the accuracy of the state-of-the-art shared-memory implementations of Word2Vec and Vertex2Vec on 1 host, and gives a geo-mean speedup of 12x and 5x respectively. Furthermore, GraphAny2Vec is on average 2x faster than the state-of-the-art distributed Word2Vec implementation, DMTK, on 32 hosts. We also show the superiority of our Gradient Combiner independent of GraphAny2Vec by incorporating it in DMTK, which raises its accuracy by > 30%.

Vec2GC -- A Graph Based Clustering Method for Text Representations

Co-Clustering With Manifold And Double Sparse Representation

Topic Modeling Using Distributed Word Embeddings

CGC: Contrastive Graph Clustering for Community Detection and Tracking

Content-driven, unsupervised clustering of news articles through multiscale graph partitioning

Region2Vec: Community Detection on Spatial Networks Using Graph Embedding with Node Attributes and Spatial Interactions

Deep Clustering by Gaussian Mixture Variational Autoencoders With Graph Embedding

Grouped Text Clustering Using Non-Parametric Gaussian Mixture Experts

ScalarGCN: Scalar-Value Association Analysis of Volumes Based on Graph Convolutional Network

Generalized Clustering and Multi-Manifold Learning with Geometric Structure Preservation

Joint unsupervised contrastive learning and robust GMM for text clustering

Clustering-based Image-Text Graph Matching for Domain Generalization

Graph Clustering via Variational Graph Embedding

A New Method for Graph-Based Representation of Text in Natural Language Processing

A Nonparametric Model for Multi-Manifold Clustering with Mixture of Gaussians and Graph Consistency

SGC: A semi-supervised pipeline for gene clustering using self-training approach in gene co-expression networks

Sparse Graph Representations for Procedural Instructional Documents

Distributed Training of Embeddings using Graph Analytics

Community Detection Guarantees Using Embeddings Learned by Node2Vec

Graph InfoClust: Leveraging cluster-level node information for unsupervised graph representation learning