Dynamic Coupon Targeting Using Batch Deep Reinforcement Learning: An Application to Livestream Shopping

Xiao Liu

DOI: https://doi.org/10.1287/mksc.2022.1403

2022-10-22

Marketing Science

Abstract:We present an empirical framework for creating dynamic coupon targeting strategies using deep reinforcement learning.

business

What problem does this paper attempt to address?

Optimizing Digital Coupon Assignment Using Constrained Reinforcement Learning.

Xinlin Yao,Xianghua Lu

DOI: https://doi.org/10.1145/3310986.3311004

2019-01-01

Abstract:Coupon marketing is a traditional but effective way to retain customers and stimulate new purchases. Recently, digital coupons have been widely used in e-commerce and distributed to almost everyone. However, the decision on when and whom to issue the coupon is often based on managers' experience and calling for optimization and automation. Collaborated with a leading e-commerce platform, we propose an exploratory constrained reinforcement learning modeling to optimize digital coupon distribution policy under the constraint of maximum offering number. Our experimental results showed that the optimal policy could increase the cumulative total sales about 6% comparing to the original policy of the platform. This work enriches the applications of reinforcement learning in real-world business practices and provides useful implications for future study on constrained reinforcement learning.
Deep Reinforcement Learning for Sequential Targeting

Wen Wang,Beibei Li,Xueming Luo,Xiaoyi Wang

DOI: https://doi.org/10.1287/mnsc.2022.4621

IF: 5.4

2022-12-23

Management Science

Abstract:Deep reinforcement learning (DRL) has opened up many unprecedented opportunities in revolutionizing the digital marketing field. In this study, we designed a DRL-based personalized targeting strategy in a sequential setting. We show that the strategy is able to address three important challenges of sequential targeting: (1) forward looking (balancing between a firm’s current revenue and future revenues), (2) earning while learning (maximizing profits while continuously learning through exploration-exploitation), and (3) scalability (coping with a high-dimensional state and policy space). We illustrate this through a novel design of a DRL-based artificial intelligence (AI) agent. To better adapt DRL to complex consumer behavior dimensions, we proposed a quantization-based uncertainty learning heuristic for efficient exploration-exploitation. Our policy evaluation results through simulation suggest that the proposed DRL agent generates 26.75% more long-term revenues than can the non-DRL approaches on average and learns 76.92% faster than the second fastest model among all benchmarks. Further, in order to better understand the potential underlying mechanisms, we conducted multiple interpretability analyses to explain the patterns of learned optimal policy at both the individual and population levels. Our findings provide important managerial-relevant and theory-consistent insights. For instance, consecutive price promotions at the beginning can capture price-sensitive consumers’ immediate attention, whereas carefully spaced nonpromotional “cooldown” periods between price promotions can allow consumers to adjust their reference points. Additionally, consideration of future revenues is necessary from a long-term horizon, but weighing the future too much can also dampen revenues. In addition, analyses of heterogeneous treatment effects suggest that the optimal promotion sequence pattern highly varies across the consumer engagement stages. Overall, our study results demonstrate DRL’s potential to optimize these strategies’ combination to maximize long-term revenues. This paper was accepted by Kartik Hosanagar, information systems. Supplemental Material: The data files and online appendix are available at https://doi.org/10.1287/mnsc.2022.4621 .

management,operations research & management science
Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning: A Field Experiment

Jiaxi Liu,Yidong Zhang,Xiaoqing Wang,Yuming Deng,Xingyu Wu

DOI: https://doi.org/10.48550/arXiv.1912.02572

2021-08-31

Abstract:In this paper we present an end-to-end framework for addressing the problem of dynamic pricing (DP) on E-commerce platform using methods based on deep reinforcement learning (DRL). By using four groups of different business data to represent the states of each time period, we model the dynamic pricing problem as a Markov Decision Process (MDP). Compared with the state-of-the-art DRL-based dynamic pricing algorithms, our approaches make the following three contributions. First, we extend the discrete set problem to the continuous price set. Second, instead of using revenue as the reward function directly, we define a new function named difference of revenue conversion rates (DRCR). Third, the cold-start problem of MDP is tackled by pre-training and evaluation using some carefully chosen historical sales data. Our approaches are evaluated by both offline evaluation method using real dataset of Alibaba Inc., and online field experiments starting from July 2018 with thousands of items, lasting for months on <a class="link-external link-http" href="http://Tmall.com" rel="external noopener nofollow">this http URL</a>. To our knowledge, there is no other DP field experiment using DRL before. Field experiment results suggest that DRCR is a more appropriate reward function than revenue, which is widely used by current literature. Also, continuous price sets have better performance than discrete sets and our approaches significantly outperformed the manual pricing by operation experts.

Machine Learning
Data-Driven Real-time Coupon Allocation in the Online Platform

Jinglong Dai,Hanwei Li,Weiming Zhu,Jianfeng Lin,Binqiang Huang

DOI: https://doi.org/10.48550/arxiv.2406.05987

2024-06-10

Econometrics

Abstract:Traditionally, firms have offered coupons to customer groups at predetermined discount rates. However, advancements in machine learning and the availability of abundant customer data now enable platforms to provide real-time customized coupons to individuals. In this study, we partner with Meituan, a leading shopping platform, to develop a real-time, end-to-end coupon allocation system that is fast and effective in stimulating demand while adhering to marketing budgets when faced with uncertain traffic from a diverse customer base. Leveraging comprehensive customer and product features, we estimate Conversion Rates (CVR) under various coupon values and employ isotonic regression to ensure the monotonicity of predicted CVRs with respect to coupon value. Using calibrated CVR predictions as input, we propose a Lagrangian Dual-based algorithm that efficiently determines optimal coupon values for each arriving customer within 50 milliseconds. We theoretically and numerically investigate the model performance under parameter misspecifications and apply a control loop to adapt to real-time updated information, thereby better adhering to the marketing budget. Finally, we demonstrate through large-scale field experiments and observational data that our proposed coupon allocation algorithm outperforms traditional approaches in terms of both higher conversion rates and increased revenue. As of May 2024, Meituan has implemented our framework to distribute coupons to over 100 million users across more than 110 major cities in China, resulting in an additional CNY 8 million in annual profit. We demonstrate how to integrate a machine learning prediction model for estimating customer CVR, a Lagrangian Dual-based coupon value optimizer, and a control system to achieve real-time coupon delivery while dynamically adapting to random customer arrival patterns.
A two-stage prediction model based on behavior mining in livestream e-commerce

Qinping Lin,Ning Jia,Liao Chen,Shiquan Zhong,Yuance Yang,Tong Gao

DOI: https://doi.org/10.1016/j.dss.2023.114013

IF: 6.969

2023-05-21

Decision Support Systems

Abstract:Livestream e-commerce has been developing at a tremendous pace in recent years. On livestream platforms, such as Douyin, a retailer attracts viewers into the live room through short video advertising, and then streamers promote and sell products in real time. In such a scenario, an accurate prediction of traffic and sales plays an essential role in operation management, including live content strategy and inventory control. However, complex behaviors (follow, share, comment, etc.) of users and long conversion paths (from seeing the advertisement to entering the live room, and to purchasing the goods) lead to poor performance of traditional prediction methods. Additionally, few studies focus on advertising information in evaluating live room performance. Therefore, we propose a two-stage learning model for traffic and sales prediction based on behavior mining, which combines marketing models and deep learning methods. In the first stage, we integrate user behaviors before getting into the live room with short video advertising data for traffic prediction. In the second stage, based on the traditional marketing model, AIDA (Attention-Interest-Desire-Action), we design a funnel convolutional neural network (FCNN) to learn sophisticated behaviors in the live room in both time and behavior orientations, and take the predicted traffic volume as the auxiliary information for sales prediction. Extensive experiments on real-world datasets from Douyin illustrate the efficacy of our proposed method, which shows the value of fusing marketing models with deep learning techniques. Furthermore, the in-depth analysis provides practical insights into user behaviors for livestream e-commerce.

computer science, information systems, artificial intelligence,operations research & management science
Simulation-Based Benchmarking of Reinforcement Learning Agents for Personalized Retail Promotions

Yu Xia,Sriram Narayanamoorthy,Zhengyuan Zhou,Joshua Mabry

2024-05-17

Abstract:The development of open benchmarking platforms could greatly accelerate the adoption of AI agents in retail. This paper presents comprehensive simulations of customer shopping behaviors for the purpose of benchmarking reinforcement learning (RL) agents that optimize coupon targeting. The difficulty of this learning problem is largely driven by the sparsity of customer purchase events. We trained agents using offline batch data comprising summarized customer purchase histories to help mitigate this effect. Our experiments revealed that contextual bandit and deep RL methods that are less prone to over-fitting the sparse reward distributions significantly outperform static policies. This study offers a practical framework for simulating AI agents that optimize the entire retail customer journey. It aims to inspire the further development of simulation tools for retail AI systems.

Artificial Intelligence,Machine Learning,Econometrics
Dynamic Knapsack Optimization Towards Efficient Multi-Channel Sequential Advertising

Xiaotian Hao,Zhaoqing Peng,Yi Ma,Guan Wang,Junqi Jin,Jianye Hao,Shan Chen,Rongquan Bai,Mingzhou Xie,Miao Xu,Zhenzhe Zheng,Chuan Yu,Han Li,Jian Xu,Kun Gai

2020-01-01

Abstract:In E-commerce, advertising is essential for merchants to reach their target users. The typical objective is to maximize the advertiser's cumulative revenue over a period of time under a budget constraint. In real applications, an advertisement (ad) usually needs to be exposed to the same user multiple times until the user finally contributes revenue (e.g., places an order). However, existing advertising systems mainly focus on the immediate revenue with single ad exposures, ignoring the contribution of each exposure to the final conversion, thus usually falls into suboptimal solutions. In this paper, we formulate the sequential advertising strategy optimization as a dynamic knapsack problem. We propose a theoretically guaranteed bilevel optimization framework, which significantly reduces the solution space of the original optimization space while ensuring the solution quality. To improve the exploration efficiency of reinforcement learning, we also devise an effective action space reduction approach. Extensive offline and online experiments show the superior performance of our approaches over state-of-the-art baselines in terms of cumulative revenue.
BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion

Fanglin Chen,Xiao Liu,Bo Tang,Feiyu Xiong,Serim Hwang,Guomian Zhuang

DOI: https://doi.org/10.48550/arXiv.2207.07790

2022-07-16

Abstract:We utilize an offline reinforcement learning (RL) model for sequential targeted promotion in the presence of budget constraints in a real-world business environment. In our application, the mobile app aims to boost customer retention by sending cash bonuses to customers and control the costs of such cash bonuses during each time period. To achieve the multi-task goal, we propose the Budget Constrained Reinforcement Learning for Sequential Promotion (BCRLSP) framework to determine the value of cash bonuses to be sent to users. We first find out the target policy and the associated Q-values that maximizes the user retention rate using an RL model. A linear programming (LP) model is then added to satisfy the constraints of promotion costs. We solve the LP problem by maximizing the Q-values of actions learned from the RL model given the budget constraints. During deployment, we combine the offline RL model with the LP model to generate a robust policy under the budget constraints. Using both online and offline experiments, we demonstrate the efficacy of our approach by showing that BCRLSP achieves a higher long-term customer retention rate and a lower cost than various baselines. Taking advantage of the near real-time cost control method, the proposed framework can easily adapt to data with a noisy behavioral policy and/or meet flexible budget constraints.

Machine Learning,Information Retrieval
Fine-Grained Session Recommendations in E-commerce using Deep Reinforcement Learning

Diddigi Raghu Ram Bharadwaj,Lakshya Kumar,Saif Jawaid,Sreekanth Vempati

DOI: https://doi.org/10.48550/arXiv.2210.15451

2022-10-20

Abstract:Sustaining users' interest and keeping them engaged in the platform is very important for the success of an e-commerce business. A session encompasses different activities of a user between logging into the platform and logging out or making a purchase. User activities in a session can be classified into two groups: Known Intent and Unknown intent. Known intent activity pertains to the session where the intent of a user to browse/purchase a specific product can be easily captured. Whereas in unknown intent activity, the intent of the user is not known. For example, consider the scenario where a user enters the session to casually browse the products over the platform, similar to the window shopping experience in the offline setting. While recommending similar products is essential in the former, accurately understanding the intent and recommending interesting products is essential in the latter setting in order to retain a user. In this work, we focus primarily on the unknown intent setting where our objective is to recommend a sequence of products to a user in a session to sustain their interest, keep them engaged and possibly drive them towards purchase. We formulate this problem in the framework of the Markov Decision Process (MDP), a popular mathematical framework for sequential decision making and solve it using Deep Reinforcement Learning (DRL) techniques. However, training the next product recommendation is difficult in the RL paradigm due to large variance in browse/purchase behavior of the users. Therefore, we break the problem down into predicting various product attributes, where a pattern/trend can be identified and exploited to build accurate models. We show that the DRL agent provides better performance compared to a greedy strategy.

Information Retrieval,Machine Learning
Deep Learning-Based Prediction and Revenue Optimization for Online Platform User Journeys

Tzu-Chien Wang

DOI: https://doi.org/10.3934/qfe.2024001

2024-01-01

Quantitative Finance and Economics

Abstract:In today's digital landscape, businesses must allocate online resources efficiently. Data-driven AI methods are increasingly adopted for customer journey management. This study enhances existing frameworks with three key propositions, integrating deep learning and optimization to create a three-step revenue optimization model using online customer data. First, we apply K-means clustering to analyze online user data, constructing a behavior model. Then, convolutional neural networks (CNN) and long short-term memory (LSTM) networks predict user behavior and conversion values from sequential data. Finally, the heuristic algorithm optimizes revenue within budget constraints based on conversions. From an academic perspective, our study provides an empirical, theory-grounded model for service and marketing management. Technologically, we identify three key findings: stacking LSTM with CNN effectively processes sequential online user data, outperforming traditional machine learning methods; optimization methods and decision trees improve model interpretability and address marketing attribution challenges by understanding user behavior and channel impacts; and traditional integer programming models fall short in solving high-dimensional online channel planning problems, necessitating heuristic algorithms. Our model aids companies in setting online channel standards and budgets, offering valuable insights and practical guidance to decision-makers.
The Online Coupon-Collector Problem and Its Application to Lifelong Reinforcement Learning

Emma Brunskill,Lihong Li

DOI: https://doi.org/10.48550/arXiv.1506.03379

2015-09-22

Abstract:Transferring knowledge across a sequence of related tasks is an important challenge in reinforcement learning (RL). Despite much encouraging empirical evidence, there has been little theoretical analysis. In this paper, we study a class of lifelong RL problems: the agent solves a sequence of tasks modeled as finite Markov decision processes (MDPs), each of which is from a finite set of MDPs with the same state/action sets and different transition/reward functions. Motivated by the need for cross-task exploration in lifelong learning, we formulate a novel online coupon-collector problem and give an optimal algorithm. This allows us to develop a new lifelong RL algorithm, whose overall sample complexity in a sequence of tasks is much smaller than single-task learning, even if the sequence of tasks is generated by an adversary. Benefits of the algorithm are demonstrated in simulated problems, including a recently introduced human-robot interaction problem.

Machine Learning,Artificial Intelligence
Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising

Di Wu,Xiujun Chen,Xun Yang,Hao Wang,Qing Tan,Xiaoxun Zhang,Jian Xu,Kun Gai

DOI: https://doi.org/10.1145/3269206.3271748

2018-10-17

Abstract:Real-time bidding (RTB) is an important mechanism in online display advertising, where a proper bid for each page view plays an essential role for good marketing results. Budget constrained bidding is a typical scenario in RTB where the advertisers hope to maximize the total value of the winning impressions under a pre-set budget constraint. However, the optimal bidding strategy is hard to be derived due to the complexity and volatility of the auction environment. To address these challenges, in this paper, we formulate budget constrained bidding as a Markov Decision Process and propose a model-free reinforcement learning framework to resolve the optimization problem. Our analysis shows that the immediate reward from environment is misleading under a critical resource constraint. Therefore, we innovate a reward function design methodology for the reinforcement learning problems with constraints. Based on the new reward design, we employ a deep neural network to learn the appropriate reward so that the optimal policy can be learned effectively. Different from the prior model-based work, which suffers from the scalability problem, our framework is easy to be deployed in large-scale industrial applications. The experimental evaluations demonstrate the effectiveness of our framework on large-scale real datasets.
Dynamic Retail Pricing via Q-Learning -- A Reinforcement Learning Framework for Enhanced Revenue Management

Mohit Apte,Ketan Kale,Pranav Datar,Pratiksha Deshmukh

2024-11-27

Abstract:This paper explores the application of a reinforcement learning (RL) framework using the Q-Learning algorithm to enhance dynamic pricing strategies in the retail sector. Unlike traditional pricing methods, which often rely on static demand models, our RL approach continuously adapts to evolving market dynamics, offering a more flexible and responsive pricing strategy. By creating a simulated retail environment, we demonstrate how RL effectively addresses real-time changes in consumer behavior and market conditions, leading to improved revenue outcomes. Our results illustrate that the RL model not only surpasses traditional methods in terms of revenue generation but also provides insights into the complex interplay of price elasticity and consumer demand. This research underlines the significant potential of applying artificial intelligence in economic decision-making, paving the way for more sophisticated, data-driven pricing models in various commercial domains.

Machine Learning
Optimizing Item-based Marketing Promotion Efficiency in C2C Marketplace with Dynamic Sequential Coupon Allocation Framework

Jie Yang,Padunna Valappil Krishnaraj Sekhar,Sho Sekine,Yilin Li

2024-09-13

Abstract:In e-commerce platforms, coupons play a crucial role in boosting transactions. In the customer-to-customer (C2C) marketplace, ensuring the satisfaction of both buyers and sellers is essential. While buyer-focused marketing strategies often receive more attention, addressing the needs of sellers is equally important. Additionally, the existing strategies tend to optimize each promotion independently, resulting in a lack of continuity between promotions and unnecessary costs in the pursuit of short-term impact within each promotion period. We introduce a Dynamic Sequential Coupon Allocation Framework (DSCAF) to optimize item coupon allocation strategies across a series of promotions. DSCAF provides sequential recommendations for coupon configurations and timing to target items. In cases where initial suggestions do not lead to sales, it dynamically adjusts the strategy and offers subsequent solutions. It integrates two predictors for estimating the sale propensity in the current and subsequent rounds of coupon allocation, and a decision-making process to determine the coupon allocation solution. It runs iteratively until the item is sold. The goal of the framework is to maximize Return on Investment (ROI) while ensuring lift Sell-through Rate (STR) remains above a specified threshold. DSCAF aims to optimize sequential coupon efficiency with a long-term perspective rather than solely focusing on the lift achieved in each individual promotion. It has been applied for item coupon allocation in Mercari.

Machine Learning
Application and Optimization of Various Machine Learning Models in Social E-Commerce Marketing Strategies

Zhen Li

DOI: https://doi.org/10.62051/bsm4y952

2024-06-20

Abstract:In the context of rapid development in social e-commerce, the optimization of marketing strategies urgently requires new technological approaches. This study investigates the application of four artificial intelligence algorithms—supervised learning, deep learning, unsupervised learning, and reinforcement learning—in Douyin live shopping and Kuaishou platform shopping, proposing a series of innovative marketing strategies. Based on an analysis of 920,000 user behavior records, we evaluate the effectiveness of each algorithm in user behavior prediction, personalized recommendation, advertisement placement optimization, and customer segmentation. The results indicate that the deep learning model achieved a prediction accuracy of 94.8%, enhancing user satisfaction by 19.7%. The supervised learning model achieved a classification accuracy of 89.3%. The reinforcement learning model increased advertisement click-through rates by 24.6%. The unsupervised learning model excelled in customer segmentation. By utilizing hybrid models and improved algorithms, marketing effectiveness was further enhanced, providing new directions and strategies for marketing practices in the social e-commerce sector.
Learning Adaptive Display Exposure for Real-Time Advertising

Weixun Wang,Junqi Jin,Jianye Hao,Chunjie Chen,Chuan Yu,Weinan Zhang,Jun Wang,Xiaotian Hao,Yixi Wang,Han Li,Jian Xu,Kun Gai

DOI: https://doi.org/10.48550/arXiv.1809.03149

2019-09-03

Abstract:In E-commerce advertising, where product recommendations and product ads are presented to users simultaneously, the traditional setting is to display ads at fixed positions. However, under such a setting, the advertising system loses the flexibility to control the number and positions of ads, resulting in sub-optimal platform revenue and user experience. Consequently, major e-commerce platforms (e.g., <a class="link-external link-http" href="http://Taobao.com" rel="external noopener nofollow">this http URL</a>) have begun to consider more flexible ways to display ads. In this paper, we investigate the problem of advertising with adaptive exposure: can we dynamically determine the number and positions of ads for each user visit under certain business constraints so that the platform revenue can be increased? More specifically, we consider two types of constraints: request-level constraint ensures user experience for each user visit, and platform-level constraint controls the overall platform monetization rate. We model this problem as a Constrained Markov Decision Process with per-state constraint (psCMDP) and propose a constrained two-level reinforcement learning approach to decompose the original problem into two relatively independent sub-problems. To accelerate policy learning, we also devise a constrained hindsight experience replay mechanism. Experimental evaluations on industry-scale real-world datasets demonstrate the merits of our approach in both obtaining higher revenue under the constraints and the effectiveness of the constrained hindsight experience replay mechanism.

Machine Learning,Artificial Intelligence
Jointly Learning to Recommend and Advertise

Xiangyu Zhao,Xudong Zheng,Xiwang Yang,Xiaobing Liu,Jiliang Tang

DOI: https://doi.org/10.1145/3394486.3403384

2020-06-19

Abstract:Online recommendation and advertising are two major income channels for online recommendation platforms (e.g. e-commerce and news feed site). However, most platforms optimize recommending and advertising strategies by different teams separately via different techniques, which may lead to suboptimal overall performances. To this end, in this paper, we propose a novel two-level reinforcement learning framework to jointly optimize the recommending and advertising strategies, where the first level generates a list of recommendations to optimize user experience in the long run; then the second level inserts ads into the recommendation list that can balance the immediate advertising revenue from advertisers and the negative influence of ads on long-term user experience. To be specific, the first level tackles high combinatorial action space problem that selects a subset items from the large item space; while the second level determines three internally related tasks, i.e., (i) whether to insert an ad, and if yes, (ii) the optimal ad and (iii) the optimal location to insert. The experimental results based on real-world data demonstrate the effectiveness of the proposed framework. We have released the implementation code to ease reproductivity.

Information Retrieval
Deep Bayesian Multi-Target Learning for Recommender Systems

Qi Wang,Zhihui Ji,Huasheng Liu,Binqiang Zhao

DOI: https://doi.org/10.48550/arxiv.1902.09154

IF: 5.414

2019-02-25

Machine Learning

Abstract:With the increasing variety of services that e-commerce platforms provide, criteria for evaluating their success become also increasingly multi-targeting. This work introduces a multi-target optimization framework with Bayesian modeling of the target events, called Deep Bayesian Multi-Target Learning (DBMTL). In this framework, target events are modeled as forming a Bayesian network, in which directed links are parameterized by hidden layers, and learned from training samples. The structure of Bayesian network is determined by model selection. We applied the framework to Taobao live-streaming recommendation, to simultaneously optimize (and strike a balance) on targets including click-through rate, user stay time in live room, purchasing behaviors and interactions. Significant improvement has been observed for the proposed method over other MTL frameworks and the non-MTL model. Our practice shows that with an integrated causality structure, we can effectively make the learning of a target benefit from other targets, creating significant synergy effects that improve all targets. The neural network construction guided by DBMTL fits in with the general probabilistic model connecting features and multiple targets, taking weaker assumption than the other methods discussed in this paper. This theoretical generality brings about practical generalization power over various targets distributions, including sparse targets and continuous-value ones.
Distributed Dynamic Pricing Strategy Based on Deep Reinforcement Learning Approach in a Presale Mechanism

Yilin Liang,Yuping Hu,Dongjun Luo,Qi Zhu,Qingxuan Chen,Chunmei Wang

DOI: https://doi.org/10.3390/su151310480

IF: 3.9

2023-07-03

Sustainability

Abstract:Despite the emergence of a presale mechanism that reduces manufacturing and ordering risks for retailers, optimizing the real-time pricing strategy in this mechanism and unknown demand environment remains an unsolved issue. Consequently, we propose an automatic real-time pricing system for e-retailers under the inventory backlog impact in the presale mode, using deep reinforcement learning technology based on the Dueling DQN algorithm. This system models the multicycle pricing problem with a finite sales horizon as a Markov decision process (MDP) to cope with the uncertain environment. We train and evaluate the proposed environment and agent in a simulation environment and compare it with two tabular reinforcement learning algorithms (Q-learning and SARSA). The computational results demonstrate that our proposed real-time pricing learning framework for joint inventory impact can effectively maximize retailers’ profits and has universal applicability to a wide range of presale models. Furthermore, according to a series of experiments, we find that retailers should not neglect the impact of the presale or previous prices on consumers’ purchase behavior. If consumers pay more attention to past prices, the retailer must decrease the current price. When the cost of inventory backlog increases, they need to offer deeper discounts in the early selling period. Additionally, introducing blockchain technology can improve the transparency of commodity traceability information, thus increasing consumer demand for purchase.

environmental sciences,environmental studies,green & sustainable science & technology
Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks

Shangxing Wang,Hanpeng Liu,Pedro Henrique Gomes,Bhaskar Krishnamachari

DOI: https://doi.org/10.1109/tccn.2018.2809722

IF: 6.359

2018-06-01

IEEE Transactions on Cognitive Communications and Networking

Abstract:We consider a dynamic multichannel access problem, where multiple correlated channels follow an unknown joint Markov model and users select the channel to transmit data. The objective is to find a policy that maximizes the expected long-term number of successful transmissions. The problem is formulated as a partially observable Markov decision process with unknown system dynamics. To overcome the challenges of unknown dynamics and prohibitive computation, we apply the concept of reinforcement learning and implement a deep Q-network (DQN). We first study the optimal policy for fixed-pattern channel switching with known system dynamics and show through simulations that DQN can achieve the same optimal performance without knowing the system statistics. We then compare the performance of DQN with a Myopic policy and a Whittle Index-based heuristic through both more general simulations as well as real data trace and show that DQN achieves near-optimal performance in more complex situations. Finally, we propose an adaptive DQN approach with the capability to adapt its learning in time-varying scenarios.

telecommunications

Dynamic Coupon Targeting Using Batch Deep Reinforcement Learning: An Application to Livestream Shopping

Optimizing Digital Coupon Assignment Using Constrained Reinforcement Learning.

Deep Reinforcement Learning for Sequential Targeting

Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning: A Field Experiment

Data-Driven Real-time Coupon Allocation in the Online Platform

A two-stage prediction model based on behavior mining in livestream e-commerce

Simulation-Based Benchmarking of Reinforcement Learning Agents for Personalized Retail Promotions

Dynamic Knapsack Optimization Towards Efficient Multi-Channel Sequential Advertising

BCRLSP: An Offline Reinforcement Learning Framework for Sequential Targeted Promotion

Fine-Grained Session Recommendations in E-commerce using Deep Reinforcement Learning

Deep Learning-Based Prediction and Revenue Optimization for Online Platform User Journeys

The Online Coupon-Collector Problem and Its Application to Lifelong Reinforcement Learning

Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising

Dynamic Retail Pricing via Q-Learning -- A Reinforcement Learning Framework for Enhanced Revenue Management

Optimizing Item-based Marketing Promotion Efficiency in C2C Marketplace with Dynamic Sequential Coupon Allocation Framework

Application and Optimization of Various Machine Learning Models in Social E-Commerce Marketing Strategies

Learning Adaptive Display Exposure for Real-Time Advertising

Jointly Learning to Recommend and Advertise

Deep Bayesian Multi-Target Learning for Recommender Systems

Distributed Dynamic Pricing Strategy Based on Deep Reinforcement Learning Approach in a Presale Mechanism

Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks