Post

https://twitter.com/docmilanfar/status/1409368685219770371

More from @docmilanfar

Peyman Milanfar

@docmilanfar

Apr 3

We often assume bigger generative models are better. But when practical image generation is limited by compute budget is this still true? Answer is no

By looking at latent diffusion models across different scales our paper sheds light on the quality vs model size tradeoffs

1/5

We trained a range of txt-2-image LDMs & observed a notable trend: when constrained by compute budget smaller models frequently outperform their larger siblings in image quality. For example the sampling result of a 223M model can be better than results of a model 4x larger

2/5

Smaller models may never reach quality levels that large models can. Yet when operating under an inference budget, points reachable by both models may be reached more efficiently w/ smaller ones. We study the tradeoff between model size, compute, quality, & downstream tasks

3/5

Read 5 tweets

Peyman Milanfar

@docmilanfar

Apr 2

It’s been >20 years since I published my first work on multi-frame super-res (SR) w/ Nhat Nguyen and the late great Gene Golub. Here’s my personal story of SR as I’ve experienced it from theory, to practical algorithms, to deployment in product. In a way it’s been my life’s work

Tsai and Huang (1984) were the first to publish the concept of multi-frame super-resolution. Key idea was that a high resolution image is related to its shifted and low-resolution versions in the frequency domain through the shift and aliasing properties of the Fourier transform

This setup assumed no noise, global translation, and a trivial point sampling process: the sensor blurring effect was ignored. But even with this simple model, the difficulty is clear. We have two entangled unknowns: motion vectors and high res image. A bit more realist model is

Read 19 tweets

Peyman Milanfar

@docmilanfar

Apr 1

Motion blur is often misunderstood, because people think of it in terms of a single imperfect image captured at some instance in time.

But motion blur is in fact an inherently temporal phenomenon. It is a temporal convolution of pixels (at the same location) across time.

1/4

Integration across time (eg open shutter) gives motion blur w/ strength depending on the speed of objects

A mix of object speed, shutter speed and frame rate together can cause aliasing in time (spokes moving backwards) & blur in space (wheel surface) all in the same image

2/4

In a video at shutter speed too low to avoid motion blur, but w/ frame rate high enough to avoid temporal aliasing, you can in fact remove motion blur just by deconvolution *in time* with a single 1D point "time" spread function. No segmentation, no motion estimation needed

3/4

Read 4 tweets

Peyman Milanfar

@docmilanfar

Mar 27

This is not a scene from Inception. The sorcery is a real photo was taken with a very long focal length lens. When the focal length is long, the field of view becomes very small and the resulting image appears more flat.

1/4

Here's another example:

The Empire State building and the Statue of Liberty are about 4.5 miles apart, and the building is 5x taller.

2/4

Here's a nice visualizations of how focal length relates to the (angular) field of view.

3/4

Read 4 tweets

Peyman Milanfar

@docmilanfar

Mar 24

What is resolution in an image? It is not the number of pixels. Here’s the classical Rayleigh’s criterion taught in basic physics:

1/5

This concept is important in imaging because it guides how densely we should pack pixels together to avoid or allow aliasing. (Yes, sometimes aliasing is useful!)

2/5

But Rayleigh's criterion is just a rule of thumb - not a physical law. It says we can’t eyeball two sources if they're too close. But this doesn't mean we can't *detect* 1 vs 2 or more sources even in the presence of noise. With proper statistical tests, we absolutely can.

3/5

Read 5 tweets

Peyman Milanfar

@docmilanfar

Mar 12

One of the lesser known ways to compare estimators is "admissibility".

An estimator θ* = g(θ,y) of θ from data y is called *in*admissible if g is uniformly dominated by another estimator g(θ,y) for all values of g(θ,y), say in the MSE sense.

1/6

Being admissible doesn't mean the estimator is good; but it's a very useful idea to weed out the bad ones.

A great example is Stein's:
The maximum likelihood estimate of Gaussian mean is inadmissible in d≥3. The nonlinear "shrinkage" that pulls y towards origin beats it

2/6

The JS story is deservedly famous as a non-linear estimator that dominates a linear one.

But this can happen even with *linear* estimators of the form θ* = G y, of the mean of multivariate normals.

Cohen proves G y is admissible iff G is symmetric, and non-expansive!

3/6

Read 6 tweets

Share this page!

Enter URL or ID to Unroll

Peyman Milanfar

Try unrolling a thread yourself!

More from @docmilanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Did Thread Reader help you today?

Don't want to be a Premium member but still want to support us?

Send Email!