Part VI: Handling Parameter Constraints of Exponential Family In Natural-gradient Methods
Warning: working in Progress (Part VI is incomplete)
Blog posts
2021
Part V: Efficient Natural-gradient Methods for Exponential Family
Warning: working in Progress (incomplete)
Part IV: Natural Gradient Descent and its Extension—Riemannian Gradient Descent
Warning: working in Progress (incomplete)
Part III: Invariance of Natural-Gradients
Goal (edited: 01-Jan-23) This blog post focuses on invariant properties of natural-gradients. We will discuss transformation rules of natural-gradients, ...
Part II: Derivation of Natural-gradients
Goal (edited: 01-Jan-23) This blog post focuses on the derivation of natural-gradients, which are known as Riemannian gradients with the Fisher-Rao metric. W...
Part I: Smooth Manifolds with the Fisher-Rao Metric
Goal (edited: 06-Feb-23) This blog post focuses on the Fisher-Rao metric, which gives rise to the Fisher information matrix (FIM). We will introduce the foll...
Structured Natural Gradient Descent (ICML 2021)
More about this work [1]: Long talk, (Youtube) short talk, extended paper, short paper, poster
2020
GD and NGD
%matplotlib inline from jax.config import config; config.update("jax_enable_x64", True) import jax.numpy as jnp from jax import grad, jit, value_and_grad imp...