RMS Norm
A special case of layer norm Given a token's embedding or feature vector $\mathbf{x}\equiv [x_1, \cdots, x_d]^T$, the layer normalization first computes the mean and std among its components \begin{equation}\mu = \frac{1}{d}\sum_{i=1}^dx_i\,\,\,\,\sigma =\sqrt{\frac{1}{d}\sum_{i=1}^d (x_i-\mu)^2}\,, \end{equation} and then normalize \begin{equation}\text{LayerNorm}(\mathbf{x})=\frac{\mathbf{x}-\mu}{\sigma}\odot \boldsymbol{\gamma} +\boldsymbol{\beta}\,,\tag{1}\end{equation} where $\boldsymbol{\gamma}$ and $\boldsymbol{\beta}$ are learnable parameters. A common view of RMS norm is to think it as a special case of the layer norm (1) in which we ignore all terms related to bias, i.e. $\mu=0$ and $\boldsymbol{\beta}=0$: \begin{equation}\text{RMSNorm}(\mathbf{x})=\frac{\mathbf{x}}{\sqrt{\displaystyle\frac{1}{d}\sum_{i=1}^d x_i^2}}\odot \boldsymbol{\gamma}\equiv\frac{\mathbf{x}}{\sqrt{\displaystyle\frac{1}{d}\left|\left|\mathbf{x}\right|\right|^2_2}}\odot \boldsymbol{\gamma}\,.\tag{2}\...