![]() ![]() How many yes-or-no questions would it take to transmit each seven-day forecast? For San Diego, a profitable first question might be: Are all seven days of the forecast sunny? If the answer is yes (and there’s a decent chance it will be), you’ve determined the entire forecast in a single question. Louis is more uncertain - the chance of a sunny day is closer to 50-50. San Diego is almost always sunny, meaning you have high confidence about what the forecast will say. Each wants to send the seven-day forecast for its city to the other. The logarithmic formula for Shannon entropy belies the simplicity of what it captures - because another way to think about Shannon entropy is as the number of yes-or-no questions needed, on average, to ascertain the content of a message.įor instance, imagine two weather stations, one in San Diego, the other in St. In information theory, it’s the logarithm of possible event outcomes. In physics, the formula for entropy involves taking a logarithm of possible physical states. There are also formal similarities in the way that entropy is calculated in both physics and information theory. In an analogous way, a random message has a high Shannon entropy - there are so many possibilities for how its information can be arranged - whereas one that obeys a strict pattern has low entropy. ![]() A cloud has higher entropy than an ice cube, since a cloud allows for many more ways to arrange water molecules than a cube’s crystalline structure does. The term “entropy” is borrowed from physics, where entropy is a measure of disorder. “He had this great intuition that information is maximized when you’re most surprised about learning about something,” said Tara Javidi, an information theorist at the University of California, San Diego. He also showed that if a sender uses fewer bits than the minimum, the message will inevitably get distorted. He captured it in a formula that calculates the minimum number of bits - a threshold later called the Shannon entropy - required to communicate a message. Shannon was the first person to make this relationship mathematically precise. More generally, the less you know about what the message will say, the more information it takes to convey. In the second you had a 1-in-4 chance of guessing the right answer - 25% certainty - and the message needed two bits of information to resolve that ambiguity. So, what’s the point? In the first scenario you had complete certainty about the contents of the message, and it took zero bits to transmit it. ![]() There are four possible messages - 00, 11, 01, 10 - and each requires two bits of information. We can communicate the result using binary code: 0 for heads, 1 for tails. In the second scenario I do my two flips with a normal coin - heads on one side, tails on the other. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |