Multiobjective Optimization of Neural Network.

JC WANG,WX LU

1995-01-01

Abstract:A theoretical model of multiobjective optimization neural network (MONN) and a new learning algorithm are presented. Experiment results show their effectiveness and advantage.

What problem does this paper attempt to address?

Neural Network Solution for General Nonlinear Optimization Problems

YL Zheng,LH Ma,JX Qian

DOI: https://doi.org/10.1109/tencon.2002.1182664

2002-01-01

Abstract:Nonlinear optimization is the foundation of multiobjective programming. In this paper, our focus is augmented Lagrange multiplier(ALM) method which is employed in constrained nonlinear optimization propositions. A dynamic approach is raised for minimization subproblem in augmented Lagrange multiplier method and a neural network alternative algorithm for general constrained programming is proposed.
An New Evolutionary Multi-Objective Optimization Algorithm

SJ Mu,HY Su,J Chu,YX Wang

DOI: https://doi.org/10.1109/cec.2003.1299764

2003-01-01

Abstract:We introduce a new, simple and efficient evolutionary algorithm to multiobjective optimization problem, which based on neighborhood and archived operation (NAGA). The innovations contain two main parts: neighborhood identify procedure to obtain Pareto optimal solutions from the population and neighborhood crowding procedure to maintain the diversity of Pareto optimal solutions previously found. The neighborhood identify procedure is composed of two steps, first to identify the locally nondominated solutions from the population and then to obtain the global nondominated solutions among the locally solutions. The neighborhood crowding is introduced to maintain a widely distributed set of Pareto solutions along the Pareto optimal front, which through implementing a comparison among the neighborhood bounds of new identified Pareto solutions and those of solutions in the archive. The winners, which are not in any ranges of the solutions in the archive, will be copied to the archive. A well-tuned fitness assignment method is structured to guide the population converging to the true Pareto optimal front. This method is pragmatic compromise between the computational simplicity and efficiency. Four nicely balanced test problems are provided to check the performance of the approach.
A Neural Network Model for General Minimax Problem

Zheng Yong-ling,Ma Long-hua,Qian Ji-xin

DOI: https://doi.org/10.1109/ijcnn.2003.1223806

2003-01-01

Abstract:Minimax problem is a significant topic in signal process (e.g. filter design) and process control (e.g. controller design), which is relevant to robustness, parameters uncertainty, and signal noise etc. However, efficient algorithms are scarce, especially those for general minimax problem with equality and inequality nonlinear constraints. In this paper a novel neural network for general minimax problem has been constructed based on a penalty function approach. The unique request on objective function and constraint functions is that they are first-order differentiable. A Lyapunov function is established for the global stability analysis. The network is simulated and its validity is illustrated by numerical examples. Simulation results show that minimax neural network, which computes in seconds, is more efficient than the previous GA/SGA algorithms, which computes in minutes.
A multi-agent complex network algorithm for multi-objective optimization

Xueyan Li,Hankun Zhang

DOI: https://doi.org/10.1007/s10489-020-01666-8

IF: 5.3

2020-03-20

Applied Intelligence

Abstract:To deal with the multi-objective optimization problems (MOPs), this study proposes a new Multi-Objective Multi-Agent Complex Network Optimization Algorithm called MOMCNA based on the idea of Cellular genetic algorithm (CGA) and the Multi-agent complex network. Compared with the traditional CGA for multi objective problem, the individuals in the population of MOMCNA have more features of intelligent agent, the new form of neighborhood for the population, private archive for individuals, the new strategy of "local-global" genetic operator and the chaotic mutation are proposed in the new algorithm to balance the convergence and diversity of the algorithm. Seventeen unconstrained multi-objective optimization problems and seven many-objective problems are introduced and tested to evaluate the new algorithm, in addition, the classical traffic assignment problem based on different system optimum principle is also established to evaluate the new algorithm. The comparison between MOMCNA and other classical algorithms shows that the proposed MOMCNA proves to be competitive in dealing with multi-objective and many-objective optimization problems and the structure of the complex network made up of population also has effect on algorithm's performance.

computer science, artificial intelligence
Multi-Objective Neural Evolutionary Algorithm for Combinatorial Optimization Problems

Hongchun Zhang,A. Jolfaei,Gautam Srivastava,Yinan Shao,Hu Yi,Dongdong Guo,Jerry Chun-Wei Lin

DOI: https://doi.org/10.1109/TNNLS.2021.3105937

IF: 14.255

2021-09-02

IEEE Transactions on Neural Networks and Learning Systems

Abstract:There has been a recent surge of success in optimizing deep reinforcement learning (DRL) models with neural evolutionary algorithms. This type of method is inspired by biological evolution and uses different genetic operations to evolve neural networks. Previous neural evolutionary algorithms mainly focused on single-objective optimization problems (SOPs). In this article, we present an end-to-end multi-objective neural evolutionary algorithm based on decomposition and dominance (MONEADD) for combinatorial optimization problems. The proposed MONEADD is an end-to-end algorithm that utilizes genetic operations and rewards signals to evolve neural networks for different combinatorial optimization problems without further engineering. To accelerate convergence, a set of nondominated neural networks is maintained based on the notion of dominance and decomposition in each generation. In inference time, the trained model can be directly utilized to solve similar problems efficiently, while the conventional heuristic methods need to learn from scratch for every given test problem. To further enhance the model performance in inference time, three multi-objective search strategies are introduced in this work. Our experimental results clearly show that the proposed MONEADD has a competitive and robust performance on a bi-objective of the classic travel salesman problem (TSP), as well as Knapsack problem up to 200 instances. We also empirically show that the designed MONEADD has good scalability when distributed on multiple graphics processing units (GPUs).

Computer Science,Medicine
Bi-level Multi-objective Evolutionary Learning: A Case Study on Multi-task Graph Neural Topology Search

Chao Wang,Licheng Jiao,Jiaxuan Zhao,Lingling Li,Xu Liu,Fang Liu,Shuyuan Yang

DOI: https://doi.org/10.1109/tevc.2023.3255263

IF: 16.497

2023-01-01

IEEE Transactions on Evolutionary Computation

Abstract:The construction of machine learning models involves many bi-level multi-objective optimization problems (BL-MOPs), where upper level (UL) candidate solutions must be evaluated via training weights of a model in the lower level (LL). Due to the Pareto optimality of sub-problems and the complex dependency across UL solutions and LL weights, an UL solution is feasible if and only if the LL weight is Pareto optimal. It is computationally expensive to determine which LL Pareto weight in the LL Pareto weight set is the most appropriate for each UL solution. This paper proposes a bi-level multi-objective learning framework (BLMOL), coupling the above decision-making process with the optimization process of the UL-MOP by introducing LL preference r. Specifically, the UL variable and r are simultaneously searched to minimize multiple UL objectives by evolutionary multi-objective algorithms. The LL weight with respect to r is trained to minimize multiple LL objectives via gradient-based preference multi-objective algorithms. In addition, the preference surrogate model is constructed to replace the expensive evaluation process of the UL-MOP. We consider a novel case study on multi-task graph neural topology search. It aims to find a set of Pareto topologies and their Pareto weights, representing different trade-offs across tasks at UL and LL, respectively. The found graph neural network is employed to solve multiple tasks simultaneously, including graph classification, node classification, and link prediction. Experimental results demonstrate that BLMOL can outperform some state-of-the-art algorithms and generate well-representative UL solutions and LL weights.

computer science, artificial intelligence, theory & methods
A New Multitask Joint Learning Framework for Expensive Multi-Objective Optimization Problems

Jianping Luo,Yongfei Dong,Qiqi Liu,Zexuan Zhu,Wenming Cao,Kay Chen Tan,Yaochu Jin

DOI: https://doi.org/10.1109/tetci.2024.3359042

2024-01-01

IEEE Transactions on Emerging Topics in Computational Intelligence

Abstract:In this paper, we propose a multi-objective optimization algorithm based on multitask conditional neural processes (MTCNPs) to deal with expensive multi-objective optimization problems (MOPs). In the proposed algorithm, an MOP is decomposed into several subproblems. Several related subproblems are assigned to a task group and jointly handled using an MTCNPs surrogate model, in which multi-task learning is incorporated to exploit the similarity across the subproblems via joint surrogate model learning. Each subproblem in a task group is modeled by a conditional neural processes (CNPs) instead of a Gaussian Process (GP), thus avoiding the calculation of the GP covariance matrix. In addition, multiple subproblems are jointly learned through a multi-layer similarity network with activation function, which can measure and utilize the similarity and useful information among subproblems more effectively and improve the accuracy and robustness of the surrogate model. Experimental studies under several scenarios indicate that the proposed algorithm performs better than several state-of-the-art multi-objective evolutionary algorithms for expensive MOPs. The parameter sensitivity and effectiveness of the proposed algorithm are analyzed in detail.
A Multi-Objective Topology Optimization Methodology Using Deep Learning and Its Application to Electromagnetic Devices

Yilun Li,Shiyou Yang,Zhuoxiang Ren

DOI: https://doi.org/10.1109/tmag.2023.3320490

IF: 1.848

2024-01-01

IEEE Transactions on Magnetics

Abstract:In this paper, a multi-objective topology optimization (MOTO) methodology using Non-dominated Sorting Genetic Algorithm II (NSGAII) and convolutional neural network (CNN) is proposed. The original NSGAII is improved to achieve better global search ability and uniform distribution of Pareto solutions. And CNN is applied as a surrogate model for finite element analysis. The framework of the proposed methodology is elaborated. To validate the proposed methodology, it is applied to the TO of an electromagnetic actuator. Numerical results validate the proposed methodology and demonstrate that computational cost of TO can be reduced.
Multi-Objective Optimisation of Multi-Output Neural Trees

Varun Ojha,Giuseppe Nicosia

DOI: https://doi.org/10.48550/arXiv.2010.04524

2020-10-09

Neural and Evolutionary Computing

Abstract:We propose an algorithm and a new method to tackle the classification problems. We propose a multi-output neural tree (MONT) algorithm, which is an evolutionary learning algorithm trained by the non-dominated sorting genetic algorithm (NSGA)-III. Since evolutionary learning is stochastic, a hypothesis found in the form of MONT is unique for each run of evolutionary learning, i.e., each hypothesis (tree) generated bears distinct properties compared to any other hypothesis both in topological space and parameter-space. This leads to a challenging optimisation problem where the aim is to minimise the tree-size and maximise the classification accuracy. Therefore, the Pareto-optimality concerns were met by hypervolume indicator analysis. We used nine benchmark classification learning problems to evaluate the performance of the MONT. As a result of our experiments, we obtained MONTs which are able to tackle the classification problems with high accuracy. The performance of MONT emerged better over a set of problems tackled in this study compared with a set of well-known classifiers: multilayer perceptron, reduced-error pruning tree, naive Bayes classifier, decision tree, and support vector machine. Moreover, the performances of three versions of MONT's training using genetic programming, NSGA-II, and NSGA-III suggest that the NSGA-III gives the best Pareto-optimal solution.
A Stochastic Multiobjective Optimization Framework for Wireless Sensor Networks

Shibo He,Jiming Chen,Weiqiang Xu,Youxian Sun,Preetha Thulasiraman,Xuemin(Sherman) Shen

DOI: https://doi.org/10.1155/2010/430615

2010-01-01

EURASIP Journal on Wireless Communications and Networking

Abstract:In wireless sensor networks (WSNs), there generally exist many different objective functions to be optimized. In this paper, we propose a stochastic multiobjective optimization approach to solve such kind of problem. We first formulate a general multiobjective optimization problem. We then decompose the optimization formulation through Lagrange dual decomposition and adopt the stochastic quasigradient algorithm to solve the primal-dual problem in a distributed way. We show theoretically that our algorithm converges to the optimal solution of the primal problem by using the knowledge of stochastic programming. Furthermore, the formulation provides a general stochastic multiobjective optimization framework for WSNs. We illustrate how the general framework works by considering an example of the optimal rate allocation problem in multipath WSNs with time-varying channel. Extensive simulation results are given to demonstrate the effectiveness of our algorithm.
Multi-Objective Satisfactory Optimisation Method

Ping Wang,Hongzhong Huang,Xu Zhang

2004-01-01

Abstract:The multi-objective satisfactory optimi ation method is developed in this paper. The proposed method is based on neural networks and genetic algorithm, and it is able to overcome the shortcoming of the single-ob ective optimi ation on the working unit of the loader. An e ample of the working unit optimi ation of the loader is given to illustrate the effectiveness of the proposed multi-ob ective satisfactory optimi ation method.
Surrogate-Assisted Evolutionary Multiobjective Neural Architecture Search based on Transfer Stacking and Knowledge Distillation

Kuangda Lyu,Hao Li,Maoguo Gong,Lining Xing,A. K. Qin

DOI: https://doi.org/10.1109/tevc.2023.3319567

IF: 16.497

2023-01-01

IEEE Transactions on Evolutionary Computation

Abstract:Multiobjective neural architecture search (MONAS) methods based on evolutionary algorithms (EAs) are inefficient when the evaluation of each architecture incorporates parameter learning from scratch. A surrogate-assisted MONAS problem can be tough considering cold-start in surrogate construction, and the evaluation of predicted promising architectures could still be cumbersome. Previously solved MONAS problems are likely to convey useful knowledge that could assist solving the current MONAS problem. To take the benefit from knowledge of these previous practices, a framework tackling large-scale knowledge transfer is proposed. Through sparse-constraint transfer stacking, the surrogate for the current problem gets informative easily. With knee-region knowledge distillation from previously learned parameters of nondominated architectures, evaluation of current architectures could be efficient and credible. To avoid transferring knowledge from irrelevant problems, an iterative source selection algorithm is designed to avoid negative transfer. The proposed framework is analyzed under different source and target MONAS problem combinations. Results show that with the help of this framework, architectures with competitive performance could be found under limited evaluation budget.

computer science, artificial intelligence, theory & methods
Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization

Sean C. Smithson,Guang Yang,Warren J. Gross,Brett H. Meyer

DOI: https://doi.org/10.48550/arXiv.1611.02120

2016-11-07

Abstract:Artificial neural networks have gone through a recent rise in popularity, achieving state-of-the-art results in various fields, including image classification, speech recognition, and automated control. Both the performance and computational complexity of such models are heavily dependant on the design of characteristic hyper-parameters (e.g., number of hidden layers, nodes per layer, or choice of activation functions), which have traditionally been optimized manually. With machine learning penetrating low-power mobile and embedded areas, the need to optimize not only for performance (accuracy), but also for implementation complexity, becomes paramount. In this work, we present a multi-objective design space exploration method that reduces the number of solution networks trained and evaluated through response surface modelling. Given spaces which can easily exceed 1020 solutions, manually designing a near-optimal architecture is unlikely as opportunities to reduce network complexity, while maintaining performance, may be overlooked. This problem is exacerbated by the fact that hyper-parameters which perform well on specific datasets may yield sub-par results on others, and must therefore be designed on a per-application basis. In our work, machine learning is leveraged by training an artificial neural network to predict the performance of future candidate networks. The method is evaluated on the MNIST and CIFAR-10 image datasets, optimizing for both recognition accuracy and computational complexity. Experimental results demonstrate that the proposed method can closely approximate the Pareto-optimal front, while only exploring a small fraction of the design space.

Neural and Evolutionary Computing,Machine Learning
A Swarm Exploring Neural Dynamics Method for Solving Convex Multi-Objective Optimization Problem

Zhijun Zhang,Haomin Yu,Xiaohui Ren,Yamei Luo

DOI: https://doi.org/10.1016/j.neucom.2024.128203

IF: 6

2024-01-01

Neurocomputing

Abstract:Multi-objective optimization problem (MOP) plays an increasingly important role in finance and engineering. In order to obtain more accurate and evenly distributed target solution set to a multi-objective programming, a novel swarm exploring neural dynamics (SEND) method is proposed, analyzed and applied in this paper. Specifically, a scalarization approach is firstly applied to transform the MOP into a group of subproblems. Secondly, each subproblem is solved by a varying parameter recurrent neural network (VP-RNN). By solving these problems, a group of Pareto optimal solutions are obtained. Thirdly, a population evolution weight optimization algorithm is used to diversify the solution set to obtain evenly distributed solutions. Simulation results demonstrate that the proposed SEND method can obtain a more accurate and evenly distributed solution set than some previous methods and the convergence rate is faster than the state-of-art methods, such as collaborative neurodynamic approach (CNA).
A Neural Network for Distributed Optimization over Multiagent Networks.

Jiazhen Wei,Sitian Qin,Wei Bian

DOI: https://doi.org/10.1007/978-3-030-64221-1_8

2020-01-01

Abstract:This paper is concerned with a distributed optimization problem with inequality constraint over a multiagent network. The objective function is the sum of multiple local convex functions, which can be nonsmooth. Based on graph theory and nonsmooth analysis, we propose a neural network with a time-varying auxiliary function. The boundedness of the state solution is demonstrated by using the properties of the auxiliary function. Moreover, it is proved that the designed neural network with any initial conditions reaches a consensus and converges to the global optimal solution. Finally, a numerical simulation is discussed to verify the theoretical results.
A multi-in and multi-out dendritic neuron model and its optimization

Yu Ding,Jun Yu,Chunzhi Gu,Shangce Gao,Chao Zhang

DOI: https://doi.org/10.1016/j.knosys.2024.111442

IF: 8.139

2024-01-26

Knowledge-Based Systems

Abstract:Artificial neural networks (ANNs), inspired by the interconnection of real neurons, have achieved unprecedented success in various fields such as computer vision and natural language processing. Recently, a novel mathematical ANN model, known as the dendritic neuron model (DNM), has been proposed to address nonlinear problems by more accurately reflecting the structure of real neurons. However, the single-output design limits its capability to handle multi-output tasks, significantly lowering its applications. In this paper, we propose a novel multi-in and multi-out dendritic neuron model (MODN) to tackle multi-output tasks. Our core idea is to introduce a filtering matrix to the soma layer to adaptively select the desired dendrites to regress each output. Because such a matrix is designed to be learnable, MODN can explore the relationship between each dendrite and output to provide a better solution to downstream tasks. We also model a telodendron layer into MODN to simulate better the real neuron behavior. Importantly, MODN is a more general and unified framework that can be naturally specialized as the DNM by customizing the filtering matrix. To explore the optimization of MODN, we investigate both heuristic and gradient-based optimizers and introduce a two-step training method for MODN. Extensive experimental results performed on 11 datasets on both binary and multi-class classification tasks demonstrate the effectiveness of MODN, with respect to accuracy, convergence, and generality.

computer science, artificial intelligence
Learning Interpretable Models Through Multi-Objective Neural Architecture Search

Zachariah Carmichael,Tim Moon,Sam Ade Jacobs

DOI: https://doi.org/10.48550/arXiv.2112.08645

IF: 5.414

2021-12-16

Machine Learning

Abstract:Monumental advances in deep learning have led to unprecedented achievements across various domains. While the performance of deep neural networks is indubitable, the architectural design and interpretability of such models are nontrivial. Research has been introduced to automate the design of neural network architectures through neural architecture search (NAS). Recent progress has made these methods more pragmatic by exploiting distributed computation and novel optimization algorithms. However, there is little work in optimizing architectures for interpretability. To this end, we propose a multi-objective distributed NAS framework that optimizes for both task performance and "introspectability," a surrogate metric for aspects of interpretability. We leverage the non-dominated sorting genetic algorithm (NSGA-II) and explainable AI (XAI) techniques to reward architectures that can be better comprehended by domain experts. The framework is evaluated on several image classification datasets. We demonstrate that jointly optimizing for task error and introspectability leads to more disentangled and debuggable architectures that perform within tolerable error.
An Evolutionary Algorithm Based on Fully Connected Weight Networks for Mixed-Variable Multiobjective Optimization

Nanjiang Dong,Tao Zhang,Rui Wang,Xiangke Liao,Ling Wang

DOI: https://doi.org/10.1016/j.ins.2023.120053

IF: 8.1

2024-01-01

Information Sciences

Abstract:Multi-objective optimization research has mostly focused on continuous-variable problems. However, real-world optimization problems often involve multiple types of variables (continuous, integer, and discrete) and multiple conflicting optimization objectives, called mixed-variable multi-objective optimization problems (MVMOPs). Discrete variables make the decision space of the problem discrete. In contrast, while different types of variables need different treatments by the evolutionary algorithm, which poses a challenge to the efficient search of the evolutionary algorithm. Therefore, we propose an evolutionary algorithm based on a fully connected weight network (FCWNEA). The fully connected network structure characterizes the entire decision space, the node access count records the frequency of visits to the node, and the weights of connections and the activity of variables estimate the distribution of the decision space. This information assists in generating offspring solutions. To evaluate the performance of the proposed algorithm, we conduct empirical experiments on different types of problems. The results show that the proposed algorithm has a significant advantage in mixed-variable multi-objective problems. Moreover, the proposed algorithm is also quite competitive in continuous problems and can better handle the correlation between variables in optimization problems.

Multiobjective Optimization of Neural Network.

Neural Network Solution for General Nonlinear Optimization Problems

An New Evolutionary Multi-Objective Optimization Algorithm

A Neural Network Model for General Minimax Problem

A multi-agent complex network algorithm for multi-objective optimization

Multi-Objective Neural Evolutionary Algorithm for Combinatorial Optimization Problems

Bi-level Multi-objective Evolutionary Learning: A Case Study on Multi-task Graph Neural Topology Search

A New Multitask Joint Learning Framework for Expensive Multi-Objective Optimization Problems

A Multi-Objective Topology Optimization Methodology Using Deep Learning and Its Application to Electromagnetic Devices

Multi-Objective Optimisation of Multi-Output Neural Trees

A Stochastic Multiobjective Optimization Framework for Wireless Sensor Networks

Multi-Objective Satisfactory Optimisation Method

Surrogate-Assisted Evolutionary Multiobjective Neural Architecture Search based on Transfer Stacking and Knowledge Distillation

Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization

A Swarm Exploring Neural Dynamics Method for Solving Convex Multi-Objective Optimization Problem

A Neural Network for Distributed Optimization over Multiagent Networks.

A multi-in and multi-out dendritic neuron model and its optimization

Learning Interpretable Models Through Multi-Objective Neural Architecture Search

An Evolutionary Algorithm Based on Fully Connected Weight Networks for Mixed-Variable Multiobjective Optimization