Abstract:Purpose - Despite the extensive academic interest in social media sentiment for financial fields, multimodal data in the stock market has been neglected. The purpose of this paper is to explore the influence of multimodal social media data on stock performance, and investigate the underlying mechanism of two forms of social media data, i.e. text and pictures. Design/methodology/approach - This research employs panel vector autoregressive models to quantify the effect of the sentiment derived from two modalities in social media, i.e. text information and picture information. Through the models, the authors examine the short-term and long-term associations between social media sentiment and stock performance, measured by three metrics. Specifically, the authors design an enhanced sentiment analysis method, integrating random walk and word embeddings through Global Vectors for Word Representation (GloVe), to construct a domain-specific lexicon and apply it to textual sentiment analysis. Secondly, the authors exploit a deep learning framework based on convolutional neural networks to analyze the sentiment in picture data. Findings - The empirical results derived from vector autoregressive models reveal that both measures of the sentiment extracted from textual information and pictorial information in social media are significant leading indicators of stock performance. Moreover, pictorial information and textual information have similar relationships with stock performance. Originality/value - To the best of the authors' knowledge, this is the first study that incorporates multimodal social media data for sentiment analysis, which is valuable in understanding pictures of social media data. The study offers significant implications for researchers and practitioners. This research informs researchers on the attention of multimodal social media data. The study's findings provide some managerial recommendations, e.g. watching not only words but also pictures in social media.

Sentimental showdown: News media vs. social media in stock markets

The impact of social and conventional media on firm equity value: A sentiment analysis approach

How does social media sentiment impact mass media sentiment? A study of news in the financial markets

Social media sentiment and the stock market

Analyzing the Effect of Social Media Sentiment on Stock Prices

The evolution of studies on social media sentiment in the stock market: Insights from bibliometric analysis

Social media investors' sentiment as stock market performance predictor

The Relationship Between Social Media Data and Stock Market Volatility

The Effect of International Media News on the Global Stock Market

Sentiments Affect Stock Returns: Evidence Based on Big Data.

Synergy between stock prices and investor sentiment in social media

The Good, the Bad, and the Social Media: Financial Implications of Social Media Reactions to Firm-Related News

Digital platforms in the news industry: how social media platforms impact traditional media news viewership

Market sentiment dispersion and its effects on stock return and volatility

Exploring the influence of multimodal social media data on stock performance: an empirical perspective and analysis

Weibo Sentiments and Stock Return: A Time-Frequency View.

News sentiment to market impact and its feedback effect

Sentiment Correlation Discovery From Social Media to Share Market

The Impact of Twitter Sentiments on Stock Market Trends

Comparing traditional news and social media with stock price movements; which comes first, the news or the price change?