Calculating latent demand in the long tail

Chris Anderson

DOI: https://doi.org/10.1145/1281192.1281193

2007-01-01

Abstract:An analytical framework for using powerlaw theory to estimate market size for niche products and consumer groups.

What problem does this paper attempt to address?

The Long Tail: Why the Future of Business is Selling Less of More

Chris Anderson

2006-07-11

Abstract:What happens when the bottlenecks that stand between supply and demand in our culture go away and everything becomes available to everyone? The Long Tail is a powerful new force i n our economy: the rise of the niche. As the cost of reaching consumers drops dramatically, our markets are shifting from a one-size-fits-all model of mass appeal to one of unlimited variety for unique tastes. From supermarket shelves to advertising agencies, the ability to offer vast choice is changing everything, and causing us to rethink where our markets lie and how to get to them. Unlimited selection is revealing truths about what consumers want and how they want to get it, from DVDs at Netflix to songs on iTunes to advertising on Google. However, this is not just a virtue of online marketplaces; it is an example of an entirely new economic model for business, one that is just beginning to show its power. After a century of obsessing over the few products at the head of the demand curve, the new economics of distribution allow us to turn our focus to the many more products in the tail, which collectively can create a new market as big as the one we already know. The Long Tail is really about the economics of abundance. New efficiencies in distribution, manufacturing, and marketing are essentially resetting the definition of what’s commercially viable across the board. If the 20th century was about hits, the 21st will be equally about niches.

Business,Engineering,Economics
Shelf space strategy in long-tail markets

R. Alexander Bentley,Paul Ormerod,Mark E. Madsen

DOI: https://doi.org/10.48550/arXiv.0808.1655

2008-08-12

General Finance

Abstract:The Internet is known to have had a powerful impact on on-line retailer strategies in markets characterised by long-tail distribution of sales. Such retailers can exploit the long tail of the market, since they are effectively without physical limit on the number of choices on offer. Here we examine two extensions of this phenomenon. First, we introduce turnover into the long-tail distribution of sales. Although over any given period such as a week or a month, the distribution is right-skewed and often power law distributed, over time there is considerable turnover in the rankings of sales of individual products. Second, we establish some initial results on the implications for shelf-space strategy of physical retailers in such markets.
Long Tails, Automation and Labor

B.N. Kausik

2023-07-27

Abstract:A central question in economics is whether automation will displace human labor and diminish standards of living. Whilst prior works typically frame this question as a competition between human labor and machines, we frame it as a competition between human consumers and human suppliers. Specifically, we observe that human needs favor long tail distributions, i.e., a long list of niche items that are substantial in aggregate demand. In turn, the long tails are reflected in the goods and services that fulfill those needs. With this background, we propose a theoretical model of economic activity on a long tail distribution, where innovation in demand for new niche outputs competes with innovation in supply automation for mature outputs. Our model yields analytic expressions and asymptotes for the shares of automation and labor in terms of just four parameters: the rates of innovation in supply and demand, the exponent of the long tail distribution and an initial value. We validate the model via non-linear stochastic regression on historical US economic data with surprising accuracy.

General Economics
Long Tail in Electronic Markets: Research on Competition Between Niche Products and Featured Products

Yuhui Yang,Yan Zhu

DOI: https://doi.org/10.1109/icsssm.2010.5530193

2010-01-01

Abstract:The emergence of Long Tail in electronic markets illuminates that niche and obscure products which previously sold very little can make up a significant share of total sales. We build a circle city model with horizontal product differentiation to describe the electronic market involving niche products and a featured product. With the model, we analyze the competition between niche products and a featured product. The analytical results demonstrate that it can enhance the consumers' utilities by adopting niche products into the electronic market, and the search cost has an important impact on niche product sales.
Evolutionary Model of a Anonymous Consumer Durable Market

Joachim Kaldasch

DOI: https://doi.org/10.48550/arXiv.1306.3395

2013-06-14

General Finance

Abstract:An analytic model is presented that considers the evolution of a market of durable goods. The model suggests that after introduction goods spread always according to a Bass diffusion. However, this phase will be followed by a diffusion process for durable consumer goods governed by a variation-selection-reproduction mechanism and the growth dynamics can be described by a replicator equation. Describing the aggregate sales as the sum of first, multiple and replacement purchase the product life cycle can be derived. Replacement purchase causes periodic variations of the sales determined by the finite lifetime of the good (Juglar cycles). The model suggests that both, Bass- and Gompertz diffusion may contribute to the product life cycle of a consumer durable. The theory contains the standard equilibrium view of a market as a special case. It depends on the time scale, whether an equilibrium or evolutionary description is more appropriate. The evolutionary framework is used to derive also the size, growth rate and price distribution of manufacturing business units. It predicts that the size distribution of the business units (products) is lognormal, while the growth rates exhibit a Laplace distribution. Large price deviations from the mean price are also governed by a Laplace distribution (fat tails). These results are in agreement with empirical findings. The explicit comparison of the time evolution of consumer durables with empirical investigations confirms the close relationship between price decline and Gompertz diffusion, while the product life cycle can be described qualitatively for a long time period.
A Latent-class Model for Estimating Product-choice Probabilities from Clickstream Data

Naoki Nishimura,Noriyoshi Sukegawa,Yuichi Takano,Jiro Iwanaga

DOI: https://doi.org/10.48550/arXiv.1612.06589

2016-12-20

Abstract:This paper analyzes customer product-choice behavior based on the recency and frequency of each customer's page views on e-commerce sites. Recently, we devised an optimization model for estimating product-choice probabilities that satisfy monotonicity, convexity, and concavity constraints with respect to recency and frequency. This shape-restricted model delivered high predictive performance even when there were few training samples. However, typical e-commerce sites deal in many different varieties of products, so the predictive performance of the model can be further improved by integration of such product heterogeneity. For this purpose, we develop a novel latent-class shape-restricted model for estimating product-choice probabilities for each latent class of products. We also give a tailored expectation-maximization algorithm for parameter estimation. Computational results demonstrate that higher predictive performance is achieved with our latent-class model than with the previous shape-restricted model and common latent-class logistic regression.

Artificial Intelligence,Optimization and Control,Applications
Large Market Asymptotics for Differentiated Product Demand Estimators With Economic Models of Supply

Timothy B. Armstrong

DOI: https://doi.org/10.3982/ecta10600

IF: 6.1

2016-01-01

Econometrica

Abstract:IO economists often estimate demand for differentiated products using data sets with a small number of large markets. This paper addresses the question of consistency and asymptotic distributions of instrumental variables estimates as the number of products increases in some commonly used models of demand under conditions on economic primitives. I show that, in a Bertrand–Nash equilibrium, product characteristics lose their identifying power as price instruments in the limit in certain cases, leading to inconsistent estimates. The reason is that product characteristic instruments achieve identification through correlation with markups, and, depending on the model of demand, the supply side can constrain markups to converge to a constant quickly relative to sampling error. I find that product characteristic instruments can yield consistent estimates in many of the cases I consider, but care must be taken in modeling demand and choosing instruments. A Monte Carlo study confirms that the asymptotic results are relevant in market sizes of practical importance.

economics,social sciences, mathematical methods,mathematics, interdisciplinary applications,statistics & probability
Empirical Methods for Dynamic Power Law Distributions in the Social Sciences

Ricardo T. Fernholz

DOI: https://doi.org/10.48550/arXiv.1602.00159

2016-01-30

ECONOMICS

Abstract:This paper introduces nonparametric econometric methods that characterize general power law distributions under basic stability conditions. These methods extend the literature on power laws in the social sciences in several directions. First, we show that any stationary distribution in a random growth setting is shaped entirely by two factors - the idiosyncratic volatilities and reversion rates (a measure of cross-sectional mean reversion) for different ranks in the distribution. This result is valid regardless of how growth rates and volatilities vary across different economic agents, and hence applies to Gibrat's law and its extensions. Second, we present techniques to estimate these two factors using panel data. Third, we show how our results offer a structural explanation for a generalized size effect in which higher-ranked processes grow more slowly than lower-ranked processes on average. Finally, we employ our empirical methods using panel data on commodity prices and show that our techniques accurately describe the empirical distribution of relative commodity prices. We also show the existence of a generalized "size" effect for commodities, as predicted by our econometric theory.
Expectations Formation with Fat-tailed Processes: Evidence from Sales Forecasts

Eugene Larsen-Hallock,Adam Rej,David Thesmar

DOI: https://doi.org/10.48550/arXiv.2210.10169

2022-10-18

Statistical Finance

Abstract:We empirically analyze a large sample of firm sales growth expectations. We find that the relationship between forecast errors and lagged revision is non-linear. Forecasters underreact to typical (positive or negative) news about future sales, but overreact to very significant news. To account for this non-linearity, we propose a simple framework, where (1) sales growth dynamics have a fat-tailed high frequency component and (2) forecasters use a simple linear rule. This framework qualitatively fits several additional features of data on sales growth dynamics, forecast errors, and stock returns.
Mathematical analysis of long tail economy using stochastic ranking processes

Kumiko Hattori,Tetsuya Hattori

DOI: https://doi.org/10.48550/arXiv.0804.1837

2008-04-11

General Finance

Abstract:We present a new method of estimating the distribution of sales rates of, e.g., book titles at an online bookstore, from the time evolution of ranking data found at websites of the store. The method is based on new mathematical results on an infinite particle limit of the stochastic ranking process, and is suitable for quantitative studies of the long tail structure of online retails. We give an example of a fit to the actual data obtained from Amazon.co.jp, which gives the Pareto slope parameter of the distribution of sales rates of the book titles in the store.
Quantifying the target market for advertisers

Charles Graham,Rachel Kennedy

DOI: https://doi.org/10.1002/cb.1986

IF: 3.199

2021-09-15

Journal of Consumer Behaviour

Abstract:Marketers who want to protect their brand's share or grow it need to know who to reach and nudge with advertising. This paper uses continuous household panel data for 55 leading, advertised brands in 12 CPG categories to quantify their target market over different time frames and conditions (market type, brand size and dynamism). Results demonstrate that the customer base (brand penetration) must swell dramatically over time to maintain, let alone grow, market share. For stable brands, penetration typically doubles from its level in one quarter to a year, then again from 1 to 5 years as brands continue to attract lighter buyers who underpin long run sales. Over 5 years, over a third of brand buyers are so light that they buy the brand just once, but such buyers are vital to sales and critical to growth. As well as quantifying the 5-year target audience for brands across these conditions, we test the predictive accuracy of the NBD-Dirichlet as a benchmark. The implications for advertising and media strategy are detailed. The long-term lessons for targeting become clear—unless brands "target the market", they have adopted a counter growth strategy.

business
Longitudinal Mediation Analysis with Latent Growth Curves

Adam J. Sullivan,Douglas D. Gunzler,Nathan Morris,Tyler J. VanderWeele

DOI: https://doi.org/10.48550/arXiv.2103.05765

2021-03-09

Methodology

Abstract:The paper considers mediation analysis with longitudinal data under latent growth curve models within a counterfactual framework. Estimators and their standard errors are derived for natural direct and indirect effects when the mediator, the outcome, and possibly also the exposure can be modeled by an underlying latent variable giving rise to a growth curve. Settings are also considered in which the exposure is instead fixed at a single point in time.
A Micro-Founded Model for CLV

Wee Chaimanowong,T. Tony Ke,Tony Ke

DOI: https://doi.org/10.2139/ssrn.3954939

2021-01-01

SSRN Electronic Journal

Abstract:A consumer incurs a constant flow cost to keep track of information on a firm's new product, which updates her expected valuation of the product according to a Brownian motion. The consumer's purchase need follows a Poisson arrival over time, and upon an arrival, she decides whether to make a purchase or not. Moreover, at any time, she decides whether to keep tracking information or drop out so as to maximize expected utility. We characterize the consumer's optimal purchase and dropout decisions in closed form. Our model generates a one-parameter formula for the individual-level retention rate that features an initial steep drop and an asymptotic slow decay at the rate of inverse square root of time. Moreover, we prove sub-exponential distribution of purchase instances and provide closed-form formula for the firm's expected revenue. We estimate the model numerically by two methods: random walk simulation and numerically solving a heat equation. A comprehensive study based on the CDNOW data shows that after incorporating consumer heterogeneity in the need arrival rate, our model provides a similar level of fit and prediction accuracy compared with popular existing models such as Pareto/NBD and BG/NBD. For the study, consumers' information tracking cost is estimated to be about $4.5 per annum.
Analytics for an Online Retailer: Demand Forecasting and Price Optimization

Kris Johnson Ferreira,Bin Hong Alex Lee,David Simchi-Levi

DOI: https://doi.org/10.1287/msom.2015.0561

2016-02-01

Manufacturing and Service Operations Management

Abstract:We present our work with an online retailer, Rue La La, as an example of how a retailer can use its wealth of data to optimize pricing decisions on a daily basis. Rue La La is in the online fashion sample sales industry, where they offer extremely limited-time discounts on designer apparel and accessories. One of the retailer’s main challenges is pricing and predicting demand for products that it has never sold before, which account for the majority of sales and revenue. To tackle this challenge, we use machine learning techniques to estimate historical lost sales and predict future demand of new products. The nonparametric structure of our demand prediction model, along with the dependence of a product’s demand on the price of competing products, pose new challenges on translating the demand forecasts into a pricing policy. We develop an algorithm to efficiently solve the subsequent multiproduct price optimization that incorporates reference price effects, and we create and implement this algorithm into a pricing decision support tool for Rue La La’s daily use. We conduct a field experiment and find that sales does not decrease because of implementing tool recommended price increases for medium and high price point products. Finally, we estimate an increase in revenue of the test group by approximately 9.7% with an associated 90% confidence interval of [2.3%, 17.8%].
Price equations with symmetric supply/demand; implications for fat tails

Carey Caginalp,Gunduz Caginalp

DOI: https://doi.org/10.48550/arXiv.1904.00267

2019-03-30

Mathematical Finance

Abstract:Implementing a set of microeconomic criteria, we develop price dynamics equations using a function of demand/supply with key symmetry properties. The function of demand/supply can be linear or nonlinear. The type of function determines the nature of the tail of the distribution based on the randomness in the supply and demand. For example, if supply and demand are normally distributed, and the function is assumed to be linear, then the density of relative price change has behavior $x^{-2}$ for large $x$ (i.e., large deviations). The exponent approaches $-1$ if the function of supply and demand involves a large exponent. The falloff is exponential, i.e., $e^{-x}$, if the function of supply and demand is logarithmic.
Predicting Customer Lifetime Values -- ecommerce use case

Ziv Pollak

DOI: https://doi.org/10.48550/arXiv.2102.05771

IF: 5.414

2021-02-10

Machine Learning

Abstract:Predicting customer future purchases and lifetime value is a key metrics for managing marketing campaigns and optimizing marketing spend. This task is specifically challenging when the relationships between the customer and the firm are of a noncontractual nature and therefore the future purchases need to be predicted based mostly on historical purchases. This work compares two approaches to predict customer future purchases, first using a 'buy-till-you-die' statistical model to predict customer behavior and later using a neural network on the same dataset and comparing the results. This comparison will lead to both quantitative and qualitative analysis of those two methods as well as recommendation on how to proceed in different cases and opportunities for future research.
Scalable Demand and Markups

Enghin Atalay,Erika Frost,Alan T. Sorensen,Christopher Sullivan,Wanjia Zhu

DOI: https://doi.org/10.2139/ssrn.4448281

2023-01-01

SSRN Electronic Journal

Abstract:We study changes in markups across 72 product markets from 2006 to 2018. A growing literature has documented a rise in markups over time using a production function approach; we instead employ the standard microeconomic method, which is to estimate demand and then invert firms’ first-order pricing conditions to infer their markups. To make the method scalable, we propose estimating nested logit demand models, using household panel data to automate the assignment of products to nests. Our results indicate an overall upward trend in markups between 2006 and 2018, with considerable heterogeneity across and within product markets. We find that changes in firms’ marginal costs and households’ price sensitivity are the primary drivers of markup increases, with changes in firm ownership playing a much smaller role.Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.
Dynamic Demand Estimation in Auction Markets

Matthew Backus,Greg Lewis

DOI: https://doi.org/10.1093/restud/rdae031

2024-03-29

Abstract:Abstract We study demand estimation in a large auction market. In our model, a dynamically evolving population of buyers with unit demand and heterogeneous and privately known preferences for a finite set of differentiated products compete in a sequence of auctions that occur in discrete time. We define an empirically tractable equilibrium concept in which bidders behave as though they are competing with the stationary distribution of opposing bids, characterize bidding strategies, and prove existence of equilibrium. Having developed this demand system, we prove that it is non-parametrically identified from panel data. We extend the model to consider a random coefficients demand system akin to workhorse demand models in industrial organization, and show that this too is non-parametrically identified. We apply the model to estimate demand and show how large sellers can exercise market power by using persistent reserve price policies, which induce higher bids and, therefore, revenues. Our analysis highlights the importance of both dynamic bidding strategies and panel data sample selection issues when analyzing these markets.

economics
Information Discovery and the Long Tail of Motion Picture Content

Anuj Kumar,Michael D. Smith,Rahul Telang

DOI: https://doi.org/10.2139/ssrn.1871090

2011-01-01

SSRN Electronic Journal

Abstract:Recent papers have shown that, in contrast to “the Long Tail” theory, movie sales remain concentrated in a small number of hits. These papers have argued that concentrated sales can be explained, in part, by heterogeneity in quality and increasing returns from social effects. Our research analyzes an additional explanation: how incomplete information may skew sales patterns. We use the movie broadcast on pay-cable channels as an exogenous shock to the availability of information, and analyze how this shock changes the resulting sales distribution. Our data show that the pay-cable broadcast shifts the distribution of DVD sales toward “Long Tail” movies, suggesting an information spillover from the broadcast. We further develop a learning-based model of DVD demand to precisely quantify the lost DVD sales due to incomplete information. Our study contributes to the academic literature on information provision and market outcomes, and the dynamics of long tail markets.
Understanding Competitive Action Repertoire, Strategic Group and Performance of E-marketplace Sellers: A Latent Growth Modeling Approach

Huifang Li,Yulin Fang,Youwei Wang,Kai H. Lim

2015-01-01

Abstract:A fundamental question in e-commerce field is how to explain and predict sales performance of e-marketplace sellers. We draw on competitive repertoire perspective and strategic group theory to understand the interaction between competitive repertoire and strategic group membership in influencing sales performance over time. To empirically test these relationships, we will collect a longitudinal dataset consisting of seller-month observations on Taobao. We plan to analyze the data using Latent Growth Modeling and Latent Class Modeling. This study contributes to research in three ways. First, it develops and tests the dynamic relationship between competitive repertoire and performance by focusing on the “dynamic” process rather than “static” characteristics. Second, it extends strategic group theory, which has historically been applied to traditional industrial environment, to the e-marketplace. Third, it responds to Chen and Miller’s (2012) call for study on bridging macro strategy perspective and micro action perspective in strategic management field.

Calculating latent demand in the long tail

The Long Tail: Why the Future of Business is Selling Less of More

Shelf space strategy in long-tail markets

Long Tails, Automation and Labor

Long Tail in Electronic Markets: Research on Competition Between Niche Products and Featured Products

Evolutionary Model of a Anonymous Consumer Durable Market

A Latent-class Model for Estimating Product-choice Probabilities from Clickstream Data

Large Market Asymptotics for Differentiated Product Demand Estimators With Economic Models of Supply

Empirical Methods for Dynamic Power Law Distributions in the Social Sciences

Expectations Formation with Fat-tailed Processes: Evidence from Sales Forecasts

Mathematical analysis of long tail economy using stochastic ranking processes

Quantifying the target market for advertisers

Longitudinal Mediation Analysis with Latent Growth Curves

A Micro-Founded Model for CLV

Analytics for an Online Retailer: Demand Forecasting and Price Optimization

Price equations with symmetric supply/demand; implications for fat tails

Predicting Customer Lifetime Values -- ecommerce use case

Scalable Demand and Markups

Dynamic Demand Estimation in Auction Markets

Information Discovery and the Long Tail of Motion Picture Content

Understanding Competitive Action Repertoire, Strategic Group and Performance of E-marketplace Sellers: A Latent Growth Modeling Approach