Abstract:The majority of online display ads are served through real-time bidding (RTB) --- each ad display impression is auctioned off in real-time when it is just being generated from a user visit. To place an ad automatically and optimally, it is critical for advertisers to devise a learning algorithm to cleverly bid an ad impression in real-time. Most previous works consider the bid decision as a static optimization problem of either treating the value of each impression independently or setting a bid price to each segment of ad volume. However, the bidding for a given ad campaign would repeatedly happen during its life span before the budget runs out. As such, each bid is strategically correlated by the constrained budget and the overall effectiveness of the campaign (e.g., the rewards from generated clicks), which is only observed after the campaign has completed. Thus, it is of great interest to devise an optimal bidding strategy sequentially so that the campaign budget can be dynamically allocated across all the available impressions on the basis of both the immediate and future rewards. In this paper, we formulate the bid decision process as a reinforcement learning problem, where the state space is represented by the auction information and the campaign's real-time parameters, while an action is the bid price to set. By modeling the state transition via auction competition, we build a Markov Decision Process framework for learning the optimal bidding policy to optimize the advertising performance in the dynamic real-time bidding environment. Furthermore, the scalability problem from the large real-world auction volume and campaign budget is well handled by state value approximation using neural networks. The empirical study on two large-scale real-world datasets and the live A/B testing on a commercial platform have demonstrated the superior performance and high efficiency compared to state-of-the-art methods.

Deep Landscape Forecasting for Real-time Bidding Advertising

Deep Landscape Forecasting in Multi-Slot Real-Time Bidding

Functional Bid Landscape Forecasting for Display Advertising.

Scalable Bid Landscape Forecasting in Real-time Bidding

Real-Time Bidding by Reinforcement Learning in Display Advertising

Interpretable Deep Learning for Forecasting Online Advertising Costs: Insights from the Competitive Bidding Landscape

Adaptive Risk-Aware Bidding with Budget Constraint in Display Advertising

Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising

Bid Landscape Forecasting and Cold Start Problem With Transformers

An Intelligent Bidding Strategy Based on Model-Free Reinforcement Learning for Real-Time Bidding in Display Advertising

Know in AdVance: Linear-Complexity Forecasting of Ad Campaign Performance with Evolving User Interest

Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising

An Efficient Deep Distribution Network for Bid Shading in First-Price Auctions

Bidding Machine: Learning to Bid for Directly Optimizing Profits in Display Advertising

Kaplan–Meier Markov Network: Learning the Distribution of Market Price by Censored Data in Online Advertising

Bid-Aware Active Learning in Real-Time Bidding for Display Advertising

Real-time bidding with multi-agent reinforcement learning in multi-channel display advertising

Deep Reinforcement Learning for Sponsored Search Real-time Bidding

HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning

Receding Bid Optimization Method with Real-Time Feedback and Prediction for Sponsored Search Advertising on Taobao

An Extensible Approach for Real-Time Bidding with Model-Free Reinforcement Learning