生图生视频模型的前世今生
早期研究：压缩与还原#
视觉信息如图片往往是高维的。一张 的 RGB 图片，便可以看作 的高维向量。
这个问题最初是 EE 的人一直在研究：面向信宿，怎么的图像压缩（从原始图片中提取表征并还原）是合适的？
AutoEncoder#
AutoEncoder 首次引入潜空间（Latent Space）的概念。使用神经网络学习高维图像到低维向量的映射（Encoder）和逆映射（Decoder）。
压缩的问题被解决，我们自然而然想到另外一个问题：假设在 Latent Space 上进行轻微扰动，我们能够得到 unseen 的输出吗？
AutoEncoder 并没有对 Encoder 和 Decoder 做约束，训练的上万张图片只是在 Latent Space 上找到了自己的散点，而没有生成未见图像的能力。
VAE#
Auto-Encoding Variational BayesHow can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions are two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.
变分自编码器（VAE）将 Pixel Space 到 Latent Space 的映射从直接预测点改为了预测分布。
它假定 Pixel Space 图片到 Latent Space 上的点的应该是高斯分布 ，本来无结构的散点在 KL 约束（Encoder 被强制要求学成高斯分布，而非是坍塌成点）下变成了一个又一个晕开的墨点。
这是非常震撼的一件事情，我们真的找到方法能够在高维图片的流形上移动了！后续的扩散和自回归路线都基于 VAE 发展。
尽管 VAE 取得了突破性的进展，它依旧存在致命的缺陷：VAE 的 loss 是 L2 约束的像素损失，为了达到更低的损失，网络会在优化器的推动下更倾向于生成多个图的平均。
GAN#
Generative Adversarial NetworksWe propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.
我们能不能不以重建为 loss 呢？GAN 提出了一种强化学习风格的生图模型，虽然现在 GAN 已经退伍，但是我还是很喜欢这种风格：通过 rollout 和 reward 让网络涌现泛化性。
GAN 的巅峰之作是微软的 StyleGAN。
A Style-Based Generator Architecture for Generative Adversarial NetworksWe propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.
Diffusion 的猪突猛进：从 DD*M 到 flow matching#
Diffusion 的演进充满渐进性，这里就不列举太多论文了。Diffusion 与其叫扩散，不如称为一种更聪明的 VAE：它与 VAE 一脉相承，（通过xx设计得到了更好的网络容量、泛化性什么的）