Post

https://twitter.com/docmilanfar/status/1409368685219770371

More from @docmilanfar

Peyman Milanfar

@docmilanfar

Sep 24

Smoothing splines fit function to data as the sol'n of a regularized least-squares optimization problem.

But it’s also possible to do it in one shot with an unusually shaped kernel (see figure)

Is it possible to solve other optimization problems this way? Surprisingly yes

1/n

This is just one instance of how one can “kernelize” an optimization problem. That is, approximate the solution of an optimization problem in just one-step by constructing and applying a kernel once to the input

Given some conditions you can it do much more generally

2/n

If you specialize the regularization to be of the form
φ(x) = ρ( ||Ax|| ) where A= R(|i-j|) is a stationary & isotropic, this gives tidy conversions between φ(x) and the kernel K(x).

3/n

Read 4 tweets

Peyman Milanfar

@docmilanfar

Sep 18

Mean-shift iteratively moves points towards regions of higher density. It does so by placing a kernel at each data point, calculating the mean of the data points within that window, shifting points towards this mean until convergence: Look familiar?

1/n
(Animation @gabrielpeyre)

The first term on the right hand side of the ODE has the form of a pseudo-linear denoiser f(x) = W(x) x. A weighted average of the points where the weights depend on the data. The overall mean-shift process is a lot like a residual flow:

d/dt x(t) = f(x(t)) - x(t)

2/n

Residual on the RHS is an approximation of the “score” -the gradient of the empirical density of x making it a gradient flow

d/dt x(t) ≈ ∇ log p̂(x(t))

So mean-shift a) estimates the empirical density & b) flows points to nearby peaks. Similarly to flow-matching & InDI

3/n

Read 4 tweets

Peyman Milanfar

@docmilanfar

Sep 5

Random matrices are very important in modern statistics and machine learning, not to mention physics

A model about which much less is known is uniformly sampled matrices from the set of doubly stochastic matrices: Uniformly Distributed Stochastic Matrices

A thread -

1/n

First, what are doubly stochastic matrices?
Non-negative matrices whose row & column sums=1.

The set of doubly stochastic matrices is also known as the Birkhoff polytope: an (n−1)² dimensional convex polytope in ℝⁿˣⁿ with extreme points being permutation matrices.

2/n

The extreme points of the Birkhoff polytope (permutations) are sparse matrices, but a typical matrix sampled from inside the polytope is by contrast, very dense

Since rows and columns are exchangeable, the entries of a sampled matrix have the same marginal distribution.

3/n

Read 8 tweets

Peyman Milanfar

@docmilanfar

Sep 1

The perpetually undervalued least-squares:

minₓ‖y−Ax‖²

can teach a lot about some complex ideas in modern machine learning including overfitting & double-descent.

Let's assume A is n-by-p. So we have n data points and p parameters

1/10

If n ≥ p (“under-fitting” or “over-determined" case) the solution is

x̃ = (AᵀA)⁻¹ Aᵀ y

But if n < p (“over-fitting” or “under-determined” case), there are infinitely many solutions that give *zero* training error. We pick min‖x‖² norm solution:

x̃ = Aᵀ(AAᵀ)⁻¹ y

2/10

In either case, the solution can be compactly written in terms of the SVD of A:

A = USVᵀ

where U & V are orthogonal matrices of size nxn & pxp, and S is nxp & contains i = 1 to k nonzero diag elements

x̃ = ∑ σᵢ⁻¹ vᵢ uᵢᵀ

where σᵢ are the nonzero sing vals of S

3/10

Read 10 tweets

Peyman Milanfar

@docmilanfar

Aug 18

Two basic concepts are often conflated:

Sample Standard Deviation (SD) vs Standard Err (SE)

Say you want to estimate m=𝔼(x) from N independent samples xᵢ. A typical choice is the average or "sample" mean m̂

But how stable is this? That's what Standard Error tells you:

1/6

Since m̂ is itself a random variable, we need to quantify the uncertainty around it too: this is what the Standard Error does.

The Standard Error is *not* the same as the spread of the samples - that's the Standard Deviation (SD) - but the two are closely related:

2/6

But this expression isn't practical because we don't know √var(xᵢ) either

Not knowing √var(xᵢ), we are forced to estimate that too. Here, we typically just plug in the (sample) Standard Deviation for it. Therefore:

Standard Error ≈ Sample Standard Deviation/√N

3/6

Read 7 tweets

Peyman Milanfar

@docmilanfar

Aug 15

Did you ever take a photo & wish you'd zoomed in more or framed better? When this happens, we just crop.

Now there's a better way: Zoom Enhance -a new feature my team just shipped on Pixel. Available in Google Photos under Tools, it enhances both zoomed & un-zoomed images

1/n

Zoom Enhance is our first im-to-im diffusion model designed & optimized to run fully on-device. It allows you to crop or frame the shot you wanted, and enhance it -after capture. The input can be from any device, Pixel or not, old or new. Below are some examples & use cases

2/n

Let's say you've zoomed to the max on your Pixel 8/9 Pro and got your shot; but you wish you could get a little closer. Now you can zoom in more, and enhance.

3/n

Read 13 tweets

Share this page!

Enter URL or ID to Unroll

Peyman Milanfar

Try unrolling a thread yourself!

More from @docmilanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Peyman Milanfar

Did Thread Reader help you today?

Don't want to be a Premium member but still want to support us?

Send Email!