Diffusion Models

Turn data into noise one small step at a time, learn to undo each step, and you can turn fresh noise into new data. Watch data fan out into a Gaussian, see why the endpoint of a Riemann sum suddenly matters, drag a set across an empirical distribution, and find out why a perfectly trained model would only copy its training set while a smoothed one creates points between them.

Built from the notes and meetings of the Random Training Points project, which studies Z. Chen, “On the Interpolation Effect of Score Smoothing in Diffusion Models” (ICLR 2026). Background follows J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models” (NeurIPS 2020), and Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations” (ICLR 2021). The weight-decay result is from P. Savarese, I. Evron, D. Soudry and N. Srebro (COLT 2019). Results from the paper are labelled with its numbering; the questions and examples are written for this page. For how this subject connects to the others, see the roadmap.

InteractiveHo–Jain–Abbeel (2020); Song et al. (2021)

From data to noise and back

Sampling a new, realistic image directly is hard; sampling Gaussian noise is easy. A diffusion model connects the two. It adds noise to real data, step by step, until nothing but noise is left, and learns to undo each step. To generate, it draws fresh noise and runs the learned undoing backwards.

The idea

Write $p_{0}$ for the unknown distribution of the data (all plausible cat pictures, say) and $x_{0} \sim p_{0}$ for one clean example. We want a procedure that produces new samples from $p_{0}$ , not copies of the training images.

Forward process: adding noise. Fix small numbers

β_{1}, β_{2}, \dots

(0, 1)

, the noise schedule. At each step, shrink the current image a little and add a little fresh Gaussian noise:

x_{k} = \sqrt{1 - β_{k}} x_{k - 1} + \sqrt{β_{k}} ε_{k}, ε_{k} \sim N (0, I) independent .

No learning happens here: it is a fixed random recipe.

Why the end result is Gaussian. Unroll the recursion. Every step leaves its own noise term

ε_{j}

, shrunk by all the steps after it, but independent Gaussians add up to a single Gaussian, so with

{\bar{α}}_{k} = \prod_{j \leq k} (1 - β_{j})

all

k

of them together are equivalent to one:

x_{k} \overset{d}{=} \sqrt{{\bar{α}}_{k}} x_{0} + \sqrt{1 - {\bar{α}}_{k}} ε, ε \sim N (0, I),

where

\overset{d}{=}

means “has the same distribution as”.

Show the computation

After two steps,

\begin{aligned} x_{2} & = \sqrt{1 - β_{2}} (\sqrt{1 - β_{1}} x_{0} + \sqrt{β_{1}} ε_{1}) + \sqrt{β_{2}} ε_{2} \\ = \sqrt{(1 - β_{2}) (1 - β_{1})} x_{0} + \sqrt{(1 - β_{2}) β_{1}} ε_{1} + \sqrt{β_{2}} ε_{2} . \end{aligned}

The noise from step 1 has been shrunk once more by step 2. Continuing the same way, each step multiplies everything so far by $\sqrt{1 - β}$ and adds new noise:

x_{k} = \sqrt{{\bar{α}}_{k}} x_{0} + \sum_{j = 1}^{k} \sqrt{β_{j} a_{j}} ε_{j}, a_{j} = \prod_{i = j + 1}^{k} (1 - β_{i}) (a_{k} = 1) .

The $ε_{j}$ are independent standard Gaussians, so the sum is Gaussian with mean 0 and variance $\sum_{j} β_{j} a_{j}$ . Since $a_{j - 1} = (1 - β_{j}) a_{j}$ , each term is a difference:

β_{j} a_{j} = a_{j} - (1 - β_{j}) a_{j} = a_{j} - a_{j - 1}, so \sum_{j = 1}^{k} β_{j} a_{j} = a_{k} - a_{0} = 1 - {\bar{α}}_{k} .

A mean-zero Gaussian with variance $1 - {\bar{α}}_{k}$ is exactly the law of $\sqrt{1 - {\bar{α}}_{k}} ε$ . Check with two steps: $(1 - β_{2}) β_{1} + β_{2} = 1 - (1 - β_{1}) (1 - β_{2})$ .

Each step multiplies the influence of $x_{0}$ by $\sqrt{1 - β} < 1$ , so $\sqrt{{\bar{α}}_{k}} \to 0$ , while the noise piles up to variance $1 - {\bar{α}}_{k} \to 1$ . Whatever the image was, $x_{k}$ ends up distributed approximately as $N (0, I)$ . The one-line form is also what training uses: to get a noisy $x_{k}$ you don't run $k$ steps, you draw one $ε$ .

The factor $\sqrt{1 - β}$ is chosen so the variance stays put: if $x_{k - 1}$ has variance 1, so does $x_{k}$ , because $(1 - β) + β = 1$ . Practical schedules increase $β_{k}$ from about $10^{- 4}$ to $0.02$ over a thousand steps.

Reverse process: generating. Draw

x_{K} \sim N (0, I)

, which is easy, and apply a learned denoiser step by step back to

k = 0

. The randomness of the generated image comes from the starting noise (and, in some samplers, from noise injected along the way); its plausibility comes from what the denoiser learned.

Training data. Take a clean image $x_{0}$ , pick a step $k$ , draw noise, and form $x_{k}$ by the formula above. The pair (noisy $x_{k}$ , clean $x_{0}$ or the noise $ε$ ) is one training example for the denoiser. The same image is reused many times, each time with a fresh step and fresh noise: that is the standard training loop, and it is what lets the denoiser learn what to do at every noise level.

Aside: the continuous-time version. With many tiny steps,

\sqrt{1 - β} \approx 1 - β / 2

, and the difference

x_{k} - x_{k - 1} \approx - \frac{1}{2} β x_{k - 1} + \sqrt{β} ε_{k}

becomes the stochastic differential equation

d X_{t} = - \frac{1}{2} β (t) X_{t} d t + \sqrt{β (t)} d W_{t},

a deterministic pull toward 0 plus Brownian noise (Song et al. 2021). It is to the recursion what gradient flow is to gradient descent. Chen's paper, used in the last module, takes the simplest version,

d X_{t} = d W_{t}

, so

X_{t} = X_{0} + W_{t}

: no shrinking, just noise of variance

t

Explore

Data

β per step Step k

40 examples being noised, step by step. They start on the data and fan out into noise.

4,000 noised examples at step

k

. exact density, a mixture of

N (\sqrt{{\bar{α}}_{k}} y, 1 - {\bar{α}}_{k})

N (0, 1)

, the target.

Worked example

Constant schedule $β = 0.1$ . Data: $x_{0} = \pm 1$ with probability $\frac{1}{2}$ each, so $x_{0}$ has mean 0 and variance 1.

After 10 steps, how much of $x_{0}$ is left, and how much noise?

${\bar{α}}_{10} = {0.9}^{10} \approx 0.349$ , so $x_{10} = 0.59 x_{0} + 0.81 ε$ . The two clean values have become two overlapping bumps centred at $\pm 0.59$ , each with standard deviation 0.81.
After 50 steps?

${\bar{α}}_{50} = {0.9}^{50} \approx 0.005$ : $x_{50} \approx 0.07 x_{0} + ε$ . The data contribute a shift of $\pm 0.07$ to a unit-variance Gaussian: essentially pure noise.
Check that the variance never changed.

$Var (x_{k}) = {\bar{α}}_{k} Var (x_{0}) + (1 - {\bar{α}}_{k}) = 1$ for every $k$ . The process trades signal for noise without changing the overall scale.

Check your understanding

Which part of a diffusion model is learned?

The noising process.That is a fixed recipe: shrink and add Gaussian noise.
The denoiser, trained on pairs of (noisy, clean) examples made by the noising process.

The forward process manufactures unlimited training pairs from each clean image. The network learns the reverse.

Why multiply by $\sqrt{1 - β}$ before adding $\sqrt{β} ε$ , instead of just adding noise?

It keeps the variance at 1, so the endpoint is the fixed law $N (0, I)$ rather than noise of ever-growing size.
Without it the result wouldn't be random.It would still be random, just with variance growing with $k$ .

Just adding noise (

X_{t} = X_{0} + W_{t}

) also works, with variance

Var (X_{0}) + t

; generation then starts from

N (0, T I)

. That is the “variance-exploding” version used in Chen's paper; the recursion here is “variance-preserving”.

Two different starting noise vectors usually generate two different images. Where does that variety come from?

From the random starting noise (and any noise added during sampling); the denoiser maps each one to a plausible image.
From randomness inside the trained network.A trained network is a fixed function.

With the deterministic (probability-flow) sampler of the later modules, the map from starting noise to output is a fixed, one-to-one function. Whether its outputs are new or copies of the training images is the question of the last module.

Practice

Unroll the recursionInduction

Show that $x_{k}$ has the same distribution as $\sqrt{{\bar{α}}_{k}} x_{0} + \sqrt{1 - {\bar{α}}_{k}} ε$ with $ε \sim N (0, I)$ , by induction on $k$ (an alternative to the telescoping sum).

Solution

If it holds for $k - 1$ , then $x_{k} = \sqrt{1 - β_{k}} (\sqrt{{\bar{α}}_{k - 1}} x_{0} + \sqrt{1 - {\bar{α}}_{k - 1}} ε^{'}) + \sqrt{β_{k}} ε_{k}$ . The two noise terms are independent Gaussians with variances $(1 - β_{k}) (1 - {\bar{α}}_{k - 1})$ and $β_{k}$ , which add to $1 - {\bar{α}}_{k}$ . A sum of independent Gaussians is Gaussian, so they combine into $\sqrt{1 - {\bar{α}}_{k}} ε$ for a single standard Gaussian $ε$ : an equality in distribution, which is all that matters for the law of $x_{k}$ .

From steps to an SDEHeuristic limit

Let $β_{k} = β (t_{k}) h$ for small step $h$ . Show that $x_{k} - x_{k - 1} \approx - \frac{1}{2} β (t) x h + \sqrt{β (t)} Δ W$ , with $Δ W \sim N (0, h)$ .

Solution

$\sqrt{1 - β h} = 1 - \frac{1}{2} β h + O (h^{2})$ , and $\sqrt{β h} ε = \sqrt{β} (\sqrt{h} ε)$ with $\sqrt{h} ε \sim N (0, h)$ , a Brownian increment. Note the noise is of size $\sqrt{h}$ , much bigger than the drift's $h$ : that is why the next module is needed.

InteractiveBackground for Song et al. (2021)

Integrating against noise

An SDE $d X = b d t + σ d W$ means $X_{t} = X_{0} + \int_{0}^{t} b d s + \int_{0}^{t} σ d W_{s}$ . The first integral is ordinary calculus. The second sums a coefficient times random increments of Brownian motion, and one surprise is all you need to know about it: where you evaluate the coefficient in each piece changes the answer.

The idea

Brownian motion.

W_{0} = 0

, continuous paths, independent increments, and

W_{t} - W_{s} \sim N (0, t - s)

. An increment over a step

h

has size about

\sqrt{h}

, not

h

, which is why the paths are continuous but nowhere differentiable.

Riemann sums, endpoints and all. For an ordinary smooth function, $\int_{0}^{t} f d g$ is the limit of $\sum_{i} f (s_{i}) (g (t_{i + 1}) - g (t_{i}))$ , and it doesn't matter whether $s_{i}$ is the left endpoint, the right endpoint or the midpoint of each piece: all three give the same limit. Try the same with $\int_{0}^{t} W d W$ :

\underset{right}{\underset{⏟}{\sum_{i} W_{t_{i + 1}} Δ W_{i}}} - \underset{left}{\underset{⏟}{\sum_{i} W_{t_{i}} Δ W_{i}}} = \sum_{i} (Δ W_{i})^{2} ⟶ t .

The sum of squared increments does not vanish: each $(Δ W_{i})^{2}$ has mean equal to the length of its piece, and they add up to $t$ (the quadratic variation).

Show the computation

Split $[0, t]$ into pieces of length $h_{i}$ , so $Δ W_{i} \sim N (0, h_{i})$ , independent. Then $E [(Δ W_{i})^{2}] = h_{i}$ and $Var [(Δ W_{i})^{2}] = 2 h_{i}^{2}$ (for a Gaussian, $E Z^{4} = 3 σ^{4}$ ). Hence

E [\sum_{i} (Δ W_{i})^{2}] = \sum_{i} h_{i} = t, Var [\sum_{i} (Δ W_{i})^{2}] = 2 \sum_{i} h_{i}^{2} \leq 2 t max_{i} h_{i} \to 0.

A sum whose mean is $t$ and whose variance goes to 0 converges to $t$ . For a differentiable $g$ , $| Δ g_{i} | \leq C h_{i}$ , so $\sum (Δ g_{i})^{2} \leq C^{2} t max_{i} h_{i} \to 0$ instead.

For a smooth path the same sum is of order

h \cdot t \to 0

, which is why ordinary calculus never notices.

Three answers. As the pieces shrink,

\sum_{i} W_{t_{i}} Δ W_{i} \to \frac{1}{2} (W_{t}^{2} - t), \sum_{i} \frac{W_{t_{i}} + W_{t_{i + 1}}}{2} Δ W_{i} = \frac{1}{2} W_{t}^{2}, \sum_{i} W_{t_{i + 1}} Δ W_{i} \to \frac{1}{2} (W_{t}^{2} + t) .

The left-endpoint choice is the Itô integral, the standard one: it only uses information available at the start of each piece, and it has mean zero. The averaged choice is the Stratonovich integral, which obeys the ordinary chain rule. Ordinary calculus would say

\frac{1}{2} W_{t}^{2}

for all three.

Explore

Pieces in the partition

Integrate against

The path

W

[0, 1]

. Each piece shows its left-endpoint and right-endpoint values; the sums use these times the increment

Δ W

across the piece.

Check your understanding

Why does the endpoint matter for $\int W d W$ but not for $\int f d g$ with smooth $f, g$ ?

The difference is $\sum (Δ W)^{2} \to t$ ; for smooth paths $\sum (Δ g)^{2} \to 0$ .
Brownian motion is discontinuous.Its paths are continuous, just very rough.

Increments of size

\sqrt{h}

have squares of size

h

, and there are

t / h

of them. For a differentiable path the squares are of size

h^{2}

The left-endpoint (Itô) sum has mean zero. Why?

$W_{t_{i}}$ is independent of the next increment $Δ W_{i}$ , which has mean zero.
Because $W$ has mean zero.The right-endpoint sum also uses $W$ , and its mean is $t / 2$ .

E [W_{t_{i}} Δ W_{i}] = E [W_{t_{i}}] E [Δ W_{i}] = 0

. With the right endpoint,

E [W_{t_{i + 1}} Δ W_{i}] = E [(Δ W_{i})^{2}] = t_{i + 1} - t_{i}

: it peeks at the increment it multiplies.

Practice

The telescoping trickWhere ½(W² − t) comes from

Show $W_{t_{i + 1}}^{2} - W_{t_{i}}^{2} = 2 W_{t_{i}} Δ W_{i} + (Δ W_{i})^{2}$ and deduce the Itô answer.

Solution

Expand $(W_{t_{i}} + Δ W_{i})^{2}$ . Summing over $i$ , the left side telescopes to $W_{t}^{2}$ , so $\sum W_{t_{i}} Δ W_{i} = \frac{1}{2} (W_{t}^{2} - \sum (Δ W_{i})^{2}) \to \frac{1}{2} (W_{t}^{2} - t)$ .

InteractiveSong et al. (2021); Chen (2026), §2

Empirical distributions and the score

A model never sees the data distribution, only finitely many examples. To say what it is trained on, we need the empirical distribution of those examples, which is easy to understand as a function on sets and awkward as a density. Noise turns it into something smooth, and the denoiser learns its score.

The idea

A distribution is a function on sets. A probability distribution

P

assigns to each (measurable) set

A

the number

P (A) \in [0, 1]

: how likely the random value is to land in

A

. For a coin,

P ({H}) = P ({T}) = \frac{1}{2}

P ({H, T}) = 1

P (\emptyset) = 0

. A density, when there is one, is just a convenient way to compute

P (A) = \int_{A} f

Empirical distribution. Given samples

y_{1}, \dots, y_{n}

, the empirical distribution assigns to each set the fraction of samples inside it:

P_{n} (A) = \frac{# {k : y_{k} \in A}}{n} .

It estimates the whole distribution, set by set, not just a single number such as the mean. Its only possible values are

0, \frac{1}{n}, \dots, 1

Why densities are awkward here. $P_{n}$ puts mass $\frac{1}{n}$ on each sample point and none anywhere else, so its “density” is $\frac{1}{n} \sum_{k} δ (x - y_{k})$ , a sum of Dirac deltas: infinitely tall spikes of zero width and area $\frac{1}{n}$ . As a function of sets it is perfectly simple; as a density it is barely a function. Adding noise fixes this: replace each spike by a Gaussian bump of width $\sqrt{t}$ ,

p_{t}^{(n)} (x) = \frac{1}{n} \sum_{k = 1}^{n} φ_{t} (x - y_{k}), φ_{t} (u) = \frac{e^{- u^{2} / 2 t}}{\sqrt{2 π t}},

which is exactly the law of $y_{K} + W_{t}$ for a uniformly chosen training point $y_{K}$ (the forward process $X_{t} = X_{0} + W_{t}$ started from the training set). As $t \to 0$ the bumps sharpen back into spikes.

Score. The score of a density

p

s (x) = \nabla_{x} \log p (x)

. The logarithm undoes the exponential in Gaussian densities: for

N (μ, t)

s (x) = - (x - μ) / t

, an arrow pointing back to the centre, stronger for smaller

t

. Generation runs

d x / d t = - \frac{1}{2} s_{t} (x)

backwards in time (the probability-flow ODE), so the score is all a sampler needs.

The empirical score (Tweedie's formula). For the mixture

p_{t}^{(n)}

\nabla \log p_{t}^{(n)} (x) = \frac{{\hat{x}}_{t} (x) - x}{t}, {\hat{x}}_{t} (x) = \sum_{k} w_{k} (x) y_{k}, w_{k} (x) = \frac{φ_{t} (x - y_{k})}{\sum_{j} φ_{t} (x - y_{j})} .

{\hat{x}}_{t} (x)

is the best guess of the clean training point given the noisy

x

: an average of the training points weighted by how likely each was to produce

x

. The score points from

x

to that guess.

Show the computation

The score is $\nabla \log p = \nabla p / p$ . Each bump is $φ_{t} (x - y_{k}) = c e^{- | x - y_{k} |^{2} / 2 t}$ , whose gradient is $- \frac{x - y_{k}}{t} φ_{t} (x - y_{k})$ . So

\nabla \log p_{t}^{(n)} (x) = \frac{\sum_{k} φ_{t} (x - y_{k}) \frac{y_{k} - x}{t}}{\sum_{j} φ_{t} (x - y_{j})} = \frac{1}{t} (\sum_{k} w_{k} (x) y_{k} - x \sum_{k} w_{k} (x)) = \frac{{\hat{x}}_{t} (x) - x}{t},

using $\sum_{k} w_{k} = 1$ . The factor $\frac{1}{n}$ and the constant $c$ cancel between numerator and denominator.

Training minimizes

t E ‖ s_{θ} (X) - \nabla \log p_{t}^{(n)} (X) ‖^{2}

, whose exact minimizer is this empirical score.

Explore

Samples n Noise level t

Top: the true density

p_{0}

and

n

samples drawn from it (ticks). Drag the ends of the shaded set

A

. Bottom: the noised empirical density

p_{t}^{(n)}

, one Gaussian bump per sample, and its score

\nabla \log p_{t}^{(n)}

, which points to the nearest training points.

Worked example

Two training points, $y = \pm 1$ , so $p_{t}^{(n)} = \frac{1}{2} φ_{t} (x + 1) + \frac{1}{2} φ_{t} (x - 1)$ .

Compute ${\hat{x}}_{t} (x)$ .

${\hat{x}}_{t} = \frac{φ_{t} (x - 1) - φ_{t} (x + 1)}{φ_{t} (x - 1) + φ_{t} (x + 1)}$ . Since $(x \mp 1)^{2} = x^{2} \mp 2 x + 1$ , the common factors cancel and ${\hat{x}}_{t} = \frac{e^{x / t} - e^{- x / t}}{e^{x / t} + e^{- x / t}} = \tanh (x / t)$ .
Evaluate the score at $x = 0.5$ , $t = 0.1$ .

$\tanh (5) \approx 0.9999$ , so the score is $(0.9999 - 0.5) / 0.1 \approx 5.0$ : a strong push toward $+ 1$ .
What does the score look like as $t \to 0$ ?

$\tanh (x / t) \to sgn (x)$ , so the score approaches $(sgn (x) - x) / t$ : pointing to the nearest training point, with a jump of size $2 / t$ at the midpoint 0. That jump is the subject of the next module.

Check your understanding

You have 4 samples. Which values can $P_{4} (A)$ take?

$0, \frac{1}{4}, \frac{1}{2}, \frac{3}{4}, 1$
Any number in $[0, 1]$ .It counts samples: only multiples of $\frac{1}{4}$ .

The empirical distribution only ever asks “how many of my samples are in this set?”. Compare the empirical mean, which estimates one number; the empirical distribution estimates every probability at once.

For a single point $A = {y_{1}}$ and a continuous $p_{0}$ , how do $P (A)$ and $P_{n} (A)$ compare?

$P (A) = 0$ but $P_{n} (A) = \frac{1}{n}$ .
They are equal.A continuous law gives each single point probability zero.

The empirical distribution concentrates on the samples, so on tiny sets it is completely wrong, while on large sets it is close (by the law of large numbers). Smoothing with noise is a way of not trusting it at small scales.

Why do diffusion models learn the score $\nabla \log p$ rather than the density $p$ ?

The sampler only needs the score, and the score doesn't depend on the unknown normalizing constant of $p$ .
The score is always smaller than the density.Not true, and not the reason.

\nabla \log (c p) = \nabla \log p

for any constant

c

. And the probability-flow ODE

d x / d t = - \frac{1}{2} \nabla \log p_{t}

is driven by the score alone.

InteractiveChen, “On the Interpolation Effect of Score Smoothing in Diffusion Models” (ICLR 2026)

Memorization, and how smoothing creates new points

If a diffusion model learned the empirical score perfectly, it would only ever reproduce its training data. Yet trained models do produce new samples. Chen's paper explains one mechanism in a setting simple enough to solve exactly: the network can't afford the empirical score's sharp jumps, and the smoothed version it learns instead sends some noise to points between the training data.

The idea

The perfect fit is useless. Plug the exact empirical score into the backward flow $d x_{t} = - \frac{1}{2} \nabla \log p_{t}^{(n)} (x_{t}) d t$ . It exactly reverses the noising of the training set, so the output at $t = 0$ is the empirical distribution $\frac{1}{n} \sum_{k} δ_{y_{k}}$ : copies of the training points. This is memorization. A model that generates anything new must be using an imperfect fit, and the question is which imperfection, and why it helps.

Two points. Take $y = \pm 1$ . For small $t$ the empirical score is close to the piecewise linear score ${\bar{s}}_{t} (x) = (sgn (x) - x) / t$ , which jumps by $2 / t$ at the midpoint.

What a regularized network prefers (Savarese et al. 2019; Chen eq. (10)). For a wide two-layer ReLU network on scalar inputs, training with weight decay amounts, in function space, to penalizing the total curvature

R [f] = \int | f^{″} (x) | d x

: for a piecewise linear

f

, the sum of the sizes of its slope changes. A jump costs infinitely much.

The smoothed score (Chen eq. (9)). For

δ \in (0, 1]

, keep

{\bar{s}}_{t}

within

δ

of each training point and replace the jump by a straight line:

{\hat{s}}_{t, δ} (x) = {\begin{cases} - (x + 1) / t, & x \leq δ - 1, \\ \frac{δ}{1 - δ} \cdot \frac{x}{t}, & | x | < 1 - δ, \\ - (x - 1) / t, & x \geq 1 - δ . \end{cases}

Its total curvature is

2 / ((1 - δ) t)

, finite. With

δ_{t} = κ \sqrt{t}

, its score-matching loss stays at a fixed small level as

t \to 0

, and among all functions that fit the empirical score that well it is within a factor

1 + 8 \sqrt{ϵ}

of the least curved (Chen Prop. 1, for

κ

large enough).

The flow, solved exactly (Chen Prop. 3). With

δ_{t} = κ \sqrt{t}

, the backward flow is piecewise linear in

\sqrt{t}

and splits the

(x, \sqrt{t})

plane into three regions. Starting at time

t

from

x

if $x \leq δ_{t} - 1$ (region A) or $x \geq 1 - δ_{t}$ (region B), it ends exactly on the training point $- 1$ or $+ 1$ ;
if $| x | < 1 - δ_{t}$ (region C), it ends at $x / (1 - δ_{t})$ , strictly between them.

So the output is a mixture: atoms at the training points (what is left of memorization) plus a smooth density on

[- 1, 1]

, the noised distribution stretched by

1 / (1 - δ_{t})

How much is new? Only the Gaussian tails that reach region C become new samples. For small starting time this fraction is about $\bar{Φ} (κ) = P (Z > κ)$ : 11.5% for $κ = 1.2$ (the value in the paper's experiment), 2.8% for $κ = 1.91$ . So “interpolation” here means a non-zero smooth part, not mostly new samples. (This count is from the project notes, not the paper.)

Not early stopping. Stopping the exact flow before $t = 0$ leaves Gaussian blur around each training point, spreading in every direction. Smoothing lands samples exactly on the segment between the points. In higher dimensions the smoothed score smooths only along the direction joining the points, so samples collapse onto the line through the data but spread along it (Chen Prop. 5): it recovers the structure without memorizing the points.

Explore

Score used for generating

Smoothing κ Start time t₀

Backward trajectories in the

(x, \sqrt{t})

plane, from

\sqrt{t_{0}}

down to 0. region C, where trajectories land between the points; elsewhere they land on

\pm 1

Where 4,000 samples from

p_{t_{0}}^{(n)}

end up at

t = 0

. Bars at

\pm 1

: copies of the training points. In between: new points.

The two scores at

t_{0}

, scaled by

t_{0}

. empirical (exact

\tanh

form, and its jump in the limit) smoothed, a straight line through the middle.

Worked example

$κ = 1$ , start at $t_{0} = 0.25$ , so $δ_{t_{0}} = κ \sqrt{t_{0}} = 0.5$ .

What is region C at $t_{0}$ ?

$| x | < 1 - δ_{t_{0}} = 0.5$ : the interval $(- 0.5, 0.5)$ .
Where do the points $x = 0.3$ and $x = 0.8$ end up?

$0.3$ is in C: it ends at $0.3 / (1 - 0.5) = 0.6$ , a new point. $0.8$ is in region B: it ends exactly on the training point $+ 1$ .
About what fraction of the samples are new?

A sample from $N (- 1, 0.25)$ lands in region C when $x > - 0.5$ : more than $0.5 = κ \sqrt{t_{0}}$ , one standard deviation, to the right of $- 1$ . That has probability $\bar{Φ} (1) \approx 15.9 %$ (minus a tiny chance of overshooting past $+ 0.5$ ), and the same holds for $+ 1$ . The figure\'s stats show the simulated fraction.

Check your understanding

Why does the exact empirical score only reproduce the training points?

It is the exact score of the noised training set, so running the flow backward exactly undoes the noising of that set.
Because the network is too small.The exact empirical score is what a perfect fit would give, however large the network.

Perfect training means memorization. Whatever produces new samples has to be a systematic deviation from the training objective's minimizer.

Why must the smoothing window shrink like $\sqrt{t}$ ?

The noised data live within about $\sqrt{t}$ of each training point, so only a window of that size keeps the fitting loss at a fixed level.
To make the curvature finite.Any $δ < 1$ makes it finite.

With a fixed

δ

, the smoothed score would be wrong right where the data are as

t \to 0

, and the loss would blow up. With

δ_{t} = κ \sqrt{t}

, rescaling by

\sqrt{t}

leaves a loss depending only on

κ

How is smoothing different from simply stopping the exact flow early?

Early stopping leaves blur around each point in every direction; smoothing puts samples exactly on the segment between the points.
There is no difference.Compare the supports: a neighbourhood of the points versus the segment joining them.

In higher dimensions this is the point: smoothing recovers the line (or manifold) the data lie on, while early stopping produces fuzz off it.

Practice

The middle regionSolve the ODE

In region C the backward flow is $\frac{d x}{d s} = - \frac{1}{2} \cdot \frac{δ_{s}}{(1 - δ_{s}) s} x$ with $δ_{s} = κ \sqrt{s}$ . Check that $x_{s} = c (1 - κ \sqrt{s})$ solves it, and deduce where a point starting at $x$ at time $t$ ends.

Solution

$\frac{d}{d s} \log (1 - κ \sqrt{s}) = - \frac{κ}{2 \sqrt{s} (1 - κ \sqrt{s})} = - \frac{1}{2} \frac{δ_{s}}{(1 - δ_{s}) s}$ . With $x_{t} = x$ , $c = x / (1 - δ_{t})$ , and at $s = 0$ , $x_{0} = c = x / (1 - δ_{t})$ . It is a straight line in $(x, \sqrt{s})$ coordinates, as in the figure.

Counting the new massGaussian tails

Show that for small $t_{0}$ the probability that a sample from $p_{t_{0}}^{(n)}$ lands in region C is about $\bar{Φ} (κ)$ .

Solution

A sample from $N (- 1, t_{0})$ is in C when $x > δ_{t_{0}} - 1$ , i.e. $(x + 1) / \sqrt{t_{0}} > κ$ : probability $\bar{Φ} (κ)$ , minus the exponentially small chance of overshooting past $1 - δ_{t_{0}}$ . The same holds for $+ 1$ , and the mixture weights $\frac{1}{2}$ average to $\bar{Φ} (κ)$ . With $n$ equally spaced points the same count gives about $2 \bar{Φ} (κ) (n - 1) / n$ , since the two end points have only one inner side.