Diffusion Model
Denoising Diffusion Probabilistic Models(DDPM)
forward(close-form)
$$\mathrm{If}X\sim\mathcal{N}(\mu_X,\sigma_X^2)\quad Y\sim\mathcal{N}(\mu_Y,\sigma_Y^2)\Z=X+Y\\mathrm{Then}Z\sim\mathcal{N}(\mu_X+\mu_Y,\sigma_X^2+\sigma_Y^2)$$
Accordingly，
$$\begin{aligned}
\boldsymbol{x}{t}& =\sqrt{\alpha_t}\boldsymbol{x}{t-1}+\sqrt{1\boldsymbol{-}\alpha_t}\boldsymbol{\varepsilon}{t-1} \
&=\sqrt{\alpha_t}\left(\sqrt{\alpha{t-1}}x_{t-2}+\sqrt{1-\alpha_{t-1}}\boldsymbol{\varepsilon}{t-2}\right)+\sqrt{1-\alpha_t}\boldsymbol{\varepsilon}{t-1} \
&=\sqrt{\alpha_t\alpha_{t-1}}x_{t-2}+\left(\sqrt{\alpha_t-\alpha_t\alpha_{t-1}}\boldsymbol{\varepsilon}{t-2}+\sqrt{1-\alpha_t}\boldsymbol{\varepsilon}{t-1}\right)\&= \cdots\
&=\sqrt{\alpha_t\alpha_{t-1}\cdots\alpha_1}\boldsymbol{x}0+\sqrt{1-\alpha_t\alpha{t-1}\cdots\alpha_1}\boldsymbol{\varepsilon} \
&=\sqrt{\overline{\alpha}_t}\boldsymbol{x}_0+\sqrt{1-\overline{\alpha}_t}\boldsymbol{\varepsilon}
\end{aligned}$$
$q(x_{t-1}|x_t,x_0)$
假设为马尔科夫模型，如果知道$x_t,x_0$，那么$q(x_{t-1}|x_t,x_0)$是个确定的分布
$$q(x_{t-1}|x_t,x_0)=\frac{q(x_{t-1},x_t,x_0)}{q(x_t,x_0)}=\frac{q(x_t\mid x_{t-1})q(x_{t-1}\mid x_0)q(x_0)}{q(x_t\mid x_0)q(x_0)}=\frac{q(x_t\mid x_{t-1})q(x_{t-1}\mid x_0)}{q(x_t\mid x_0)}$$
Reverse
$$q(\boldsymbol{x}_{t-1}|\boldsymbol{x}t,\boldsymbol{x}0)=\mathcal{N}\left(\boldsymbol{x}{t-1};\frac{\sqrt{\alpha_t}\left(1-\overline{\alpha}{t-1}\right)\boldsymbol{x}t+\sqrt{\overline{\alpha}{t-1}}\left(1-\alpha_t\right)\boldsymbol{x}_0}{1-\overline{\alpha}t},\frac{\left(1-\alpha_t\right)\left(1-\overline{\alpha}{t-1}\right)}{1-\overline{\alpha}_t}\boldsymbol{I}\right)$$
$$\begin{aligned}q(x_t\mid x_{t-1})&\sim\mathcal{N}(x_t;\sqrt{\alpha_t}x_{t-1},1-\alpha_t)\\q(x_{t-1}\mid x_0)&\sim\mathcal{N}(x_{t-1};\sqrt{\overline{\alpha}{t-1}}x_0,1-\overline{\alpha}{t-1})\\q(x_t\mid x_0)&\sim\mathcal{N}(x_t;\sqrt{\overline{\alpha}_t}x_0,1-\overline{\alpha}_t)\end{aligned}$$
如果知道$x_0$，
$$\frac{\sqrt{\alpha_{t}}\left(1-\bar{\alpha}{t-1}\right)\boldsymbol{x}{t}+\sqrt{\bar{\alpha}{t-1}}\left(1-\alpha{t}\right)\boldsymbol{x}{0}}{1-\overline{\alpha}{t}}\boxed{\Longrightarrow}\frac{1}{\sqrt{\alpha_{t}}}\left(\boldsymbol{x}{t}-\frac{1\boldsymbol{-}\alpha{t}}{\sqrt{1\boldsymbol{-}\overline{\alpha}{t}}}\boldsymbol{\varepsilon}{t}\right)$$
training
$$\begin{aligned}&\underset{\theta}{\operatorname*{argmin}}\frac1{2\sigma_q^2(t)}[||\boldsymbol{\mu}_\theta-\boldsymbol{\mu}_q||\frac22]\&\underset{\theta}{\operatorname*{argmin}}\frac1{2\sigma_q^2(t)}\frac{(1-\alpha_t)^2}{(1-\bar{\alpha}_t)\alpha_t}[||\boldsymbol{\varepsilon}_t-\hat{\boldsymbol{\varepsilon}}_\theta(\boldsymbol{x}_t,t)||_2^2]\end{aligned}$$原文对其简化·
sampling
$$\boldsymbol{x}_{t-1}=\frac1{\sqrt{\alpha_t}}\left(\boldsymbol{x}_t-\frac{1-\alpha_t}{\sqrt{1-\overline{\alpha}_t}}\boldsymbol{\varepsilon}_\theta(\boldsymbol{x}_t,t)\right)+\sigma_t\boldsymbol{z}$$