Regarding probabilities, it is easy to get confused unless you communicate clearly what it represents.
Your model is the small world and the big world is, well, the true reality.
Big world is how uncertain YOU are about your model. It expresses your confidence of how closely the model tracks the real world.
Model is a model. It’ll do what you ask it to do. It wouldn’t know about assumptions and details that you don’t bake into it.
When you do an A/B test, you should optimize for profit and not for accuracy. That is how @VWO’s statistical engine also works. It gives you a smart decision first and then asks you to shoot for accuracy.
If acting on even a 10% probable event can give you profit, why would you wait for 90% probability?
You don’t get paid (by customers of employers) to predict accurately, you always get paid to take good decisions in a real world context.