如何做生成更好的视频图像？Meta_MIT最新《流匹配（FlowMatching,FM）》指南和代码_83页_11mb

Flow Matching Guide and Code Summary

Core Content

Flow Matching (FM) is a modern framework for generative modeling that has achieved state-of-the-art results in various domains such as image, video, audio, speech, and biological structure generation. It is based on learning a velocity field, which defines a flow through the solution of an ordinary differential equation (ODE). The framework allows for the transformation of a source distribution $p$ into a target distribution $q$ by constructing a probability path $p_t$ and training a velocity field to match this path.

FM has been extended to handle more complex scenarios, including non-Euclidean spaces (Riemannian Flow Matching), discrete state spaces (Discrete Flow Matching), and general Continuous Time Markov Processes (CTMPs). These extensions enable the application of FM to a wide range of generative tasks, including language modeling and protein folding.

Main Points

• Flow Matching (FM) is a framework for generative modeling that uses a velocity field to define a time-continuous transformation.
• The core idea is to design a probability path $p_t$ between the source and target distributions and train a velocity field to match this path.
• FM is trained by minimizing a loss function that measures the difference between the predicted and true velocity fields at various time points.
• The framework can be applied to both continuous and discrete state spaces, and can be generalized to Riemannian manifolds and CTMPs.
• FM provides a unified view of many existing generative models, including diffusion models and CTMC models.

Key Information

1. Flow Models

• Flow models are deterministic and bijective transformations of the $d$-dimensional Euclidean space $\mathbb{R}^d$.
• A flow is defined by an ODE with a velocity field $u_t(x)$.
• The velocity field $u_t$ determines the time-continuous transformation $\psi_t(x)$, which maps the source distribution $p$ to the target distribution $q$.
• The probability path $p_t$ is constructed by integrating the velocity field over time.

2. Flow Matching

• Flow Matching aims to find a velocity field $u_t^\theta$ that matches the desired probability path $p_t$.

• The loss function for Flow Matching is defined as:

  $$
  \mathcal{L}{\mathrm{FM}} (\theta) = \mathbb{E}{t, X_t} \left| u_t^\theta (X_t) - u_t (X_t) \right| ^2
  $$

• The loss can be simplified by conditioning on a single data example, leading to the conditional Flow Matching loss:

  $$
  \mathcal{L}{\mathrm{CFM}} (\theta) = \mathbb{E}{t, X_0, X_1} | u_t^\theta (X_t) - (X_1 - X_0) |^2
  $$

• The velocity field is trained using regression to approximate the true velocity field that generates the desired path.

3. Conditional Densities and Expectations

• Conditional densities and expectations are used to compute the transformation of distributions under a diffeomorphism.
• The conditional expectation $\mathbb{E}[X|Y=y]$ is the best function approximating $X$ in the least-squares sense given $Y=y$.
• The tower property of conditional expectations simplifies the computation of expectations over multiple random variables.

4. Diffeomorphisms and Push-Forward Maps

• Diffeomorphisms are invertible functions with continuous derivatives.
• The push-forward map transforms the probability density function of a random variable $X$ to that of $Y = \psi(X)$.
• The change of variables formula allows the computation of the PDF of $Y$ based on the PDF of $X$ and the Jacobian of the inverse function.

5. Extensions to Non-Euclidean and Discrete Spaces

• Riemannian Flow Matching extends FM to Riemannian manifolds, enabling the use of FM in applications involving non-Euclidean data.
• Discrete Flow Matching applies FM to discrete state spaces, such as those found in language modeling.
• Generator Matching generalizes FM to any generative Markov process, unifying various models into a common framework.

6. Relation to Diffusion Models

• Diffusion models are a type of CTMP and are closely related to FM.
• Diffusion models use a forward noise process and a backward reversal process to learn the score function of the marginal probabilities.
• The linear version of FM can be seen as a limiting case of bridge matching, which is a technique used in diffusion models.

Conclusion

Flow Matching provides a flexible and powerful framework for generative modeling, applicable to both continuous and discrete state spaces, as well as non-Euclidean geometries. It offers a unified approach to various generative models and enables the development of new models for different modalities. The accompanying PyTorch package provides a practical implementation of FM, allowing researchers to quickly adopt and extend the framework for their applications.