Approximate Nash Equilibrium Learning for n-Player Markov Games in Dynamic Pricing

Larkin Liu

DOI: https://doi.org/10.48550/arXiv.2207.06492

2024-03-02

Abstract:We investigate Nash equilibrium learning in a competitive Markov Game (MG) environment, where multiple agents compete, and multiple Nash equilibria can exist. In particular, for an oligopolistic dynamic pricing environment, exact Nash equilibria are difficult to obtain due to the curse-of-dimensionality. We develop a new model-free method to find approximate Nash equilibria. Gradient-free black box optimization is then applied to estimate $\epsilon$, the maximum reward advantage of an agent unilaterally deviating from any joint policy, and to also estimate the $\epsilon$-minimizing policy for any given state. The policy-$\epsilon$ correspondence and the state to $\epsilon$-minimizing policy are represented by neural networks, the latter being the Nash Policy Net. During batch update, we perform Nash Q learning on the system, by adjusting the action probabilities using the Nash Policy Net. We demonstrate that an approximate Nash equilibrium can be learned, particularly in the dynamic pricing domain where exact solutions are often intractable.

Computer Science and Game Theory,Artificial Intelligence

What problem does this paper attempt to address?

This paper attempts to solve the problem of finding Nash equilibrium in a dynamic pricing environment. Specifically, the paper focuses on how to find approximate Nash equilibria in a competitive Markov Game (MG) environment where multiple agents (e.g., companies) compete and there may be multiple Nash equilibria. Due to the curse - of - dimensionality, it is difficult to obtain the exact Nash equilibrium in this oligopolistic dynamic pricing environment. Therefore, the author has developed a new model - free method to find approximate Nash equilibria.

The main contribution of the paper lies in proposing a deep - reinforcement - learning - based method. By using gradient - free black - box optimization techniques to estimate $\epsilon$, that is, the maximum reward advantage that a single agent can obtain by deviating from any joint strategy, and at the same time estimating the strategy that minimizes $\epsilon$ in a given state. These strategies and the mapping relationship from states to $\epsilon$ - minimizing strategies are represented by neural networks, and the latter is called the Nash Policy Net. In the batch - update process, by performing Nash Q - learning on the system and adjusting the action probabilities, an approximate Nash equilibrium is learned, especially in the field of dynamic pricing where exact solutions are usually difficult to calculate.

In summary, the paper aims to provide an effective method to deal with the dynamic pricing problem in a multi - agent competitive environment, especially when traditional methods are difficult to find exact solutions, and to achieve the learning of Nash equilibrium through approximation methods. This is not only of great significance for theoretical research, but also provides a possible solution for automatic pricing algorithms in practical applications.