Modes of homogeneous gradient flows

Ido Cohen, Omri Azencot, Pavel Lifshits, Guy Gilboa

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

Finding latent structures in data is drawing increasing attention in diverse fields such as image and signal processing, fluid dynamics, and machine learning. In this work we examine the problem of finding the main modes of gradient flows. Gradient descent is a fundamental process in optimization where its stochastic version is prominent in training of neural networks. Here our aim is to establish a consistent theory for gradient flows ψt = P(ψ), where P is a nonlinear homogeneous operator. Our proposed framework stems from analytic solutions of homogeneous flows, previously formalized by Cohen and Gilboa, where the initial condition ψ0 admits the nonlinear eigenvalue problem P(ψ0) =λψ0.We first present an analytic solution for dynamic mode decomposition (DMD) in such cases. We show an inherent flaw of DMD, which is unable to recover the essential dynamics of the flow. It is evident that DMD is best suited for homogeneous flows of degree one. We propose an adaptive time sampling scheme and show its dynamics are analogue to homogeneous flows of degree one with a fixed step size. Moreover, we adapt DMD to yield a real spectrum, using symmetric matrices. Our analytic solution of the proposed scheme recovers the dynamics perfectly and yields zero error. We then proceed to show the relation between the orthogonal modes {ϕi} and their decay profiles under the gradient flow. We formulate orthogonal nonlinear spectral decomposition (OrthoNS), which recovers the essential latent structures of the gradient descent process. Definitions for spectrum and filtering are given, and a Parseval-type identity is shown. Experimental results on images show the resemblance to direct computations of nonlinear spectral decomposition. A significant speedup (by about two orders of magnitude) is achieved for this application using the proposed method.

Original languageEnglish
Pages (from-to)913-945
Number of pages33
JournalSIAM Journal on Imaging Sciences
Volume14
Issue number3
DOIs
StatePublished - 2021
Externally publishedYes

Keywords

  • Dynamic mode decomposition
  • Gradient flows
  • Homogeneous operators
  • Nonlinear decomposition
  • Nonlinear spectral theory

Fingerprint

Dive into the research topics of 'Modes of homogeneous gradient flows'. Together they form a unique fingerprint.

Cite this