Abstract:Recent empirical studies have identified fixed point iteration phenomena in deep neural networks, where the hidden state tends to stabilize after several layers, showing minimal change in subsequent layers. This observation has spurred the development of practical methodologies, such as accelerating inference by bypassing certain layers once the hidden state stabilizes, selectively fine-tuning layers to modify the iteration process, and implementing loops of specific layers to maintain fixed point iterations. Despite these advancements, the understanding of fixed point iterations remains superficial, particularly in high-dimensional spaces, due to the inadequacy of current analytical tools. In this study, we conduct a detailed analysis of fixed point iterations in a vector-valued function modeled by neural networks. We establish a sufficient condition for the existence of multiple fixed points of looped neural networks based on varying input regions. Additionally, we expand our examination to include a robust version of fixed point iterations. To demonstrate the effectiveness and insights provided by our approach, we provide case studies that looped neural networks may exist $2^d$ number of robust fixed points under exponentiation or polynomial activation functions, where $d$ is the feature dimension. Furthermore, our preliminary empirical results support our theoretical findings. Our methodology enriches the toolkit available for analyzing fixed point iterations of deep neural networks and may enhance our comprehension of neural network mechanisms.

A Modified Iterative Approach for Fixed Point Problem in Hadamard Spaces

Inertial modified S-iteration method for Cayley inclusion problem and fixed point problem

Convergence of algorithms for fixed points of relatively nonexpansive mappings via Ishikawa iteration

Three-Step Iterative Algorithm for the Extended Cayley–Yosida Inclusion Problem in 2-Uniformly Smooth Banach Spaces: Convergence and Stability Analysis

The study of new fixed-point iteration schemes for solving absolute value equations

Modified proximal point algorithms for variational inclusion problems on Hadamard manifolds

A Modified Form of Inertial Viscosity Projection Methods for Variational Inequality and Fixed Point Problems

Inertial iterative method for solving equilibrium problems and fixed point problems

Recent Developments in Iterative Algorithms for Digital Metrics

An efficient iterative procedure in hyperbolic space and application to non-linear delay integral equation

Proximal Point Algorithms on Hadamard Manifolds: Linear Convergence and Finite Termination

A faster iterative scheme for common fixed points of $ G $-nonexpansive mappings via directed graphs: application in split feasibility problems

Convergence Result for Solving the Split Fixed Point Problem with Multiple Output Sets in Nonlinear Spaces

Convergence analysis of inexact proximal point algorithms on Hadamard manifolds

Accelerated iterative splitting methods on Hadamard manifolds

A novel algorithm with an inertial technique for fixed points of nonexpansive mappings and zeros of accretive operators in Banach spaces

Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study

Advancing Nonlinear Problem Solutions: Newton-Like Iterative Techniques for Varied Convergence Orders using Homotopy Perturbation

A fixed-point algorithm for matrix projections with applications in quantum information

Numerical Solution of the Absolute Value Equation Using Modified Iteration Methods

Two Modified Inertial Algorithms for a Class of Split Common Null Point Problems with Demi-Contractive Mapping