Incorporating momentum acceleration techniques applied in deep learning into traditional optimization algorithms

Bin She,Aimé Fournier,Yaojun Wang,Guangmin Hu

DOI: https://doi.org/10.1190/segam2019-3216012.1

2019-01-01

SEG Technical Program Expanded Abstracts

Abstract:PreviousNext No AccessSEG Technical Program Expanded Abstracts 2019Incorporating momentum acceleration techniques applied in deep learning into traditional optimization algorithmsAuthors: Bin SheAimé FournierYaojun WangGuangmin HuBin SheUniversity of Electronic Science and Technology of ChinaSearch for more papers by this author, Aimé FournierUniversity of Colorado DenverSearch for more papers by this author, Yaojun WangUniversity of Electronic Science and Technology of ChinaSearch for more papers by this author, and Guangmin HuUniversity of Electronic Science and Technology of ChinaSearch for more papers by this authorhttps://doi.org/10.1190/segam2019-3216012.1 SectionsAboutPDF/ePub ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinked InRedditEmail AbstractIn recent years many significant advances have been made in developing numerical optimization algorithms for large-scale machine learning applications, typically deep learning. Momentum techniques (MT) are widely imposed into various optimization approaches due to its efficiency of increasing convergence speed, dampening oscillations, and avoiding local minima or saddle points. However, because of the complexity, time and expense involved in training a deep neural network, research on using MT stays on the framework of the stochastic gradient descent (SGD) algorithm. In this work, we introduce MT into the traditional non-linear conjugate gradient and quasi-Newton optimization methods, which combines the advantages of both MT and traditional optimization methods. Meanwhile, we propose a descent direction memory (DDM) method based on the essential idea of MT. We validate the use of MT and the proposed DDM method using a classical performance test problem and a 1D seismic inversion example. The experiments show off the combined effects of MT, DDM, and traditional optimization methods in generally increasing convergence rate and obtaining a smaller steady-state error.Presentation Date: Wednesday, September 18, 2019Session Start Time: 9:20 AMPresentation Time: 10:10 AMLocation: Poster Station 13Presentation Type: PosterKeywords: optimization, algorithm, inversionPermalink: https://doi.org/10.1190/segam2019-3216012.1FiguresReferencesRelatedDetailsCited byA Hierarchical Prestack Seismic Inversion Scheme for VTI Media Based on the Exact Reflection CoefficientIEEE Transactions on Geoscience and Remote Sensing, Vol. 60 SEG Technical Program Expanded Abstracts 2019ISSN (print):1052-3812 ISSN (online):1949-4645Copyright: 2019 Pages: 5407 publication data© 2019 Published in electronic format with permission by the Society of Exploration GeophysicistsPublisher:Society of Exploration Geophysicists HistoryPublished Online: 10 Aug 2019 CITATION INFORMATION Bin She, Aimé Fournier, Yaojun Wang, and Guangmin Hu, (2019), "Incorporating momentum acceleration techniques applied in deep learning into traditional optimization algorithms," SEG Technical Program Expanded Abstracts : 2668-2673. https://doi.org/10.1190/segam2019-3216012.1 Plain-Language Summary KeywordsoptimizationalgorithminversionPDF DownloadLoading ...