SP's can work. But only if you don't try to explain them, define them or formalize them. If you do you'll instantly cause them to collapse into a costly mistake with an inbound paradoxical dimension that will suck all joy out of your existence.
1) They're numbers.
2) Showing Relative Effort
3) Higher Numbers Have Greater Uncertainty
If you believe this then according to the principle of least surprise you've invented a horrible abstraction.
In the small the problem might seem negligible but every single team I've met using SP's have had superstitious beliefs around story selection that could be traced back to misunderstanding that fact.
This is why although your have a numeric "velocity" (a sum of numbers) most will also argue that 100 one points stories are different from one 100 point story.
...
...
Each story is a bet. The size signifies both the cost of the bet and the certainty for a payoff. Problematic.
They're also numbers that really can't be relative.
Or if they are relative. Then they can't be distributions.
And if they are indeed truly relative they're easily converted into hours.
We're now in a distinctively non simple scenario.