What is MVN (Multivariate Normal Distribution)?

1. The Definition

A random vector $\mathbf{x} \in \mathbb{R}^k$ follows an MVN distribution, denoted as $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$, if its joint probability density function (PDF) is:

$$f(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^k |\boldsymbol{\Sigma}|}} \exp\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x}-\boldsymbol{\mu})\right)$$

2. The Core Parameters

While a 1D Gaussian uses a scalar mean ($\mu$) and variance ($\sigma^2$), the MVN uses:
* Mean Vector ($\boldsymbol{\mu}$): A $k \times 1$ vector representing the expected value (the “peak”) in $k$-dimensional space.
* Covariance Matrix ($\boldsymbol{\Sigma}$): A $k \times k$ symmetric, positive semi-definite matrix that defines the “shape” of the data spread:
* Diagonal elements ($\Sigma_{ii}$): The variance of each individual feature.
* Off-diagonal elements ($\Sigma_{ij}$): The covariance (correlation) between features.

3. Geometric Interpretation: The Mahalanobis Distance

The term in the exponent, $D_M^2 = (\mathbf{x}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x}-\boldsymbol{\mu})$, is the Squared Mahalanobis Distance.
* In standard Euclidean distance, we assume all directions are weighted equally.
* In MVN, the “isocontours” (surfaces of equal probability) are ellipsoids.
* The Mahalanobis distance measures how many “standard deviations” $\mathbf{x}$ is from $\boldsymbol{\mu}$, effectively “whitening” the data by accounting for the correlation and scale defined by $\boldsymbol{\Sigma}$.

4. Critical Properties for AI Research

In machine learning, we love the MVN because it is analytically tractable (you can solve the integrals with pen and paper):

5. Why it Matters in Deep Learning

1. The Reparameterization Trick: To backpropagate through a sampling layer, we sample $\epsilon \sim \mathcal{N}(0, \mathbf{I})$ and transform it: $\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \epsilon$.
2. Maximum Entropy: For a known mean and covariance, the MVN is the distribution that assumes the least about the rest of the data, making it a safe “prior.”
3. Latent Spaces: We often force neural networks to map complex data (images/text) into a simple MVN latent space to make generation easier.

• This topic was modified 3 months ago by Wolf.
• This topic was modified 3 months ago by yRocket.