@iwangulenko @birdxi1988 @nntaleb I'll give you some background first, then bring in entropy. Note I am not speaking for Taleb here.
@iwangulenko @birdxi1988 @nntaleb Kelly betting is the application of information theory to investing. This conceptual leap makes sense since information theory tells us how to quantify information to make decisions under uncertainty.
@iwangulenko @birdxi1988 @nntaleb Information theory is grounded in Shannon's communication channel analogy. This can in turn be framed WRT to betting by showing how a gambler's growth of capital is equal to the rate of transmission of information over the channel.
@iwangulenko @birdxi1988 @nntaleb In this scenario we are assuming a gambler is receiving information about the outcomes of some chance event. This is the gambler's "edge." A gambler's edge is the side information he can use to gain an advantage (he knows something others do not).
@iwangulenko @birdxi1988 @nntaleb The distinguishing feature of a communication system is that the ultimate receiver (person) is in a position to profit from knowledge of the input. Thus Kelly showed that a gamblers money could grow exponentially using his side information ("edge").
@iwangulenko @birdxi1988 @nntaleb If the channel is noiseless then there is no uncertainty. This means the amount of money a gambler can make depends only on how much he chooses to bet. With no uncertainty the gambler would obviously choose to bet all his money.
@iwangulenko @birdxi1988 @nntaleb In this scenario we have an obvious exponential growth of money after N bets: 
@iwangulenko @birdxi1988 @nntaleb But in real life the "channel" is going to be noisy, which means uncertainty, which means taking into account the probability of error and correct transmission, (p and q). Since we now have imperfect information we can say that continued betting will eventually lead to ...
@iwangulenko @birdxi1988 @nntaleb ...going broke (absorbing barrier).
So the alternative is for the gambler to bet a fraction of his capital each time. Now we can express the growth of the gambler's capital as:
So the alternative is for the gambler to bet a fraction of his capital each time. Now we can express the growth of the gambler's capital as:
@iwangulenko @birdxi1988 @nntaleb Notice how we have extended equation 1 by taking into account the probability of error, the probability of correct communication, and the fraction of the bet. This is the first line on Taleb's whiteboard, but with different symbols:
@iwangulenko @birdxi1988 @nntaleb This (3) is the expected value for log wealth.
So, what happens when we maximize the gambler's growth of capital? By differentiating equation 3 and setting to 0:
So, what happens when we maximize the gambler's growth of capital? By differentiating equation 3 and setting to 0:
@iwangulenko @birdxi1988 @nntaleb This is the second line on Taleb's board. So we can see Kelly expressed the rate of transmission over a noisy communication channel in terms of various probabilities (as Shannon did) but applied to one's wealth when betting.
@iwangulenko @birdxi1988 @nntaleb So where does entropy come in? If we are framing one's growth of wealth in terms of a noisy channel then we are dealing with uncertainty (imperfect information). The gambler's "edge" can be seen as a proxy to the actual betting outcome.
@iwangulenko @birdxi1988 @nntaleb The edge is some incomplete piece of information. Assessing how valuable that incomplete information is can be done through the use of mutual information (MI), since MI quantifies the amount of information we can get about one variable (outcome) by observing another (edge).
@iwangulenko @birdxi1988 @nntaleb Since MI is a way to measure how our knowledge of the proxy reduces the uncertainty around the true outcome, MI is a measure of uncertainty. So is entropy. The connection between entropy and MI is well established.
@iwangulenko @birdxi1988 @nntaleb MI can thus measure the gambler's ignorance and this is how entropy connects to the "side information/edge" the gambler uses to make money.
@iwangulenko @birdxi1988 @nntaleb We can be more obvious about the connection between entropy and Kelly betting by expressing uncertainty in terms of its original mechanical model. The more possible configurations of a system the less we know, the higher the entropy.
@iwangulenko @birdxi1988 @nntaleb In this fashion Boltzmann showed that entropy depends on the information one happens to have available.
@iwangulenko @birdxi1988 @nntaleb We know entropy is an additive quantity, like mass and energy (combine 2 volumes of gas get twice the mass). But the "number of ways" we combine things is multiplicative (2 dice can fall in 36 ways).
@iwangulenko @birdxi1988 @nntaleb Boltzmann used the logarithm to connect entropy to the number of configurations because he knew the logarithm can turn multiplicative quantities into additive ones.
@iwangulenko @birdxi1988 @nntaleb And therein lies the trick first used by Boltzmann, then Kelly, then some smart economists. Kelly showed that it's the logarithm of the gambler's capital which is additive in sequential bets, and to which the law of large numbers applies.
@iwangulenko @birdxi1988 @nntaleb Thus a gambler should try to maximize the expected value of the logarithm of his capital in order to remain additive in repeated bets.
Apologies for the Tweet storm.
Apologies for the Tweet storm.
