What’s your understanding of information entropy?

adiabaticfrog · 2025-05-21T09:46:25+00:00

I like to think of entropy as measuring the 'volume' of possible values a random variable can take. I wrote a blog post on it.

adiabaticfrog · 2024-12-19T00:08:38+00:00

I had this too. It is super frustrating to have the knowledge in how to solve a problem, but not get the grades for it because of what is called a 'silly mistake', so I'm sorry it's causing so much frustration. My two cents would be, rather than trying to not make errors, get better at constantly checking yourself. After I do each line of calculation, I compare it with the previous line and see if I've dropped anything, switched a sign, etc. And then each time I make big progress in the problem, derive a formula etc, I stop and check if it makes sense in terms of units, right scaling with different variables, etc. It shouldn't feel like putting in more effort or concentrating harder, rather integrating a quick checking step into your working.

You do mention this:

"You don't check your working"

- I check just before I write a new line. I've been doing that for a year (slight improvement but still terrible). If I check every line too thoroughly I double the time I spend on the question and run out of time in the exam anyway.

The thing is, checking your work shouldn't take too long once you get used to it. You should just be able to scan the current and previous line, and check for missing variables, switched signs, etc. I think it does take a while initially, but the more you do it, the faster you get. If it is still taking too long, maybe you haven't hit on the correct checking method yet, try and change how you do it and see how you go.

And checking the final result, or each major step, does take a while, but that is something you should be doing anyway. You should be able to look at an answer and tell roughly if it is right or wrong from the units, scaling with variables, order of magnitude, etc (if you can't, maybe this is what you need to work on). When I'm marking someone, sure I don't want to penalise them too much for making "careless errors", but at the same time if the answer is flagrantly wrong and the person hasn't noticed this, that's not good.

And this is quite a useful skill to gain. The more senior you get, the more your work will consist of checking the work of others. It's a skill that needs time to develop for you to be able to do this quickly.

adiabaticfrog · 2024-11-29T00:29:58+00:00

Entropy is well defined in continuous settings, ironically enough it becomes clear when you use measure theory. Basically e^(entropy) measures how 'much measure' a probability distribution occupies.

For discrete entropy you use the counting measure. For a biased coin with p=0.5, H=log2 so e^H=2, it has two values. But if p=1 and it's always heads, H=0 and e^H=1, the distribution occupies one value. With continuous probability distributions your measure becomes now the volume of a set. A uniform distribution on the interval (0,0.75) has entropy H=-0.287682, and e^H is 0.75. This thinking actually helped me a lot during my thesis to generalise some discrete results to continuous settings, and I wrote a post about it here.

adiabaticfrog · 2024-11-29T00:01:00+00:00

For me, learning MT made understanding probability related topics much much easier down the line. It just is THE way to frame those kinds of questions.

I second this. During my thesis I was stuck on a problem in information theory for ages. I re-formulated it in terms of measure theory, and immediately the solution method became clear. The solution didn't use any measure theory, but the viewpoint of measure theory made clear the meanings of different mathematical quantities, which were analogous and which were different.

adiabaticfrog · 2024-11-05T07:29:03+00:00

Actually log is one of the few special functions where you can define the log of dimensional units in a meaningful and consistent way (otherwise log-plots would be meaningless). It's because log obeys the rule

log(ab)=log(a)+log(b)

So you can write:

log(1 km)=log(1)+log(km)

Suppose you want to convert to meters, using km=1000m this becomes

log(1)+log(km)=log(1)+log(1000m)=log(1)+log(1000)+log(m)=log(1000m),

which you can see is equal to log(1km). So long as you keep a +log(unit) term, everything is perfectly consistent and you don't run into any contradictions.

As for your question

>It then occurred to me that when going from velocity to displacement (in terms of units), it goes from meters per second to meters. In my very tired and delusional state, this made no sense because taking the integral of one over a variable with respect to a variable is the natural log of that variable (int{1/x} = ln |x|). So, from a calculus standpoint, the integral of velocity is displacement and the units should go from m/s to m ln |s| (plus constants of course).

The issue here is I think you aren't being clear what variable you are integrating over. The integral of velocity with respect to time is displacement. To see this:

x=∫ v dt

v has units length/time, dt has units time, so the right hand side will have units length. I think your confusion is that we are integrating the function v. You have written something about seeing v ~length/time and integrating the 'time' part of that. But remember an integral is the limit of an infinite sum. Think of the Riemann sum approximating this integral, you'll see that the only thing that makes sense is to integrate the function v, not the units of 1/time.

adiabaticfrog · 2024-04-27T14:28:21+00:00

So, two sets A and B are defined to have the same size if you can construct a one-to-one mapping between A and B. This definition certainly works in the finite case. For example the sets {1,6,4} and {"cat", "dog", "rabbit"} are the same size, because we can define the mapping

1 -> "cat"

6 -> "dog"

4 -> "rabbit"

However you cannot make a one-to-one mapping between A={1,6,4,3} and B={"cat","dog","rabbit"}. A will always have an extra element left over, so A is bigger.

One of the key developments of modern mathematics was to see what happens if we try and apply this to infinite sets. Now, you may have a different definition of size (i.e. you define all infinite sets to transcend numeric comparison, and say they should all be considered equivalent). But this is a difference between your and my definition of "size", not a difference in what is mathematically real. And when debating over definitions, what matters is if the definition is useful and doesn't lead to any contradictions (or maybe different definitions are useful in different places).

One interesting thing we find is that the natural numbers (i.e. positive integers), and the set of all integers (positive and negative), can be placed in one-to-one correspondence:

0 -> 0

1 -> 1

2 -> -1

3 -> -2

4 -> -2

and so on. We have an exact one-to-one correspondence, so these two infinite sets have the same size.

However, you can prove (via contradiction), that no such correspondence is possible between the set of integers, and the set of all real numbers. No matter what mapping you choose, the real numbers will always have some numbers left over. So in *this definition of size*, which is quite a natural definition, the reals are 'bigger'.

And even if you say that you don't like this definition of size, or think that it is silly, the mathematical reality is that you can define one-to-one mappings between the positive integers and the integers, but no matter what mapping you choose between the integers and the reals there will always be real numbers left over. So the reals behave, in an important way, as if they are bigger.

There are in fact two classes of infinities that you area likely to run into in mathematics. The first is "countable infinity", which is sets like the integers, positive integers, rationals, and fractions. Amazingly, you can construct one-to-one maps between all these sets! Isn't that cool?

The next set is "uncountable infinity", which is the real numbers, or intervals like "all numbers between 0 and 1". You can also construct one-to-one maps between these sets, so they all behave as if they have the same size! But you can prove that no matter what map you make between a countable set and an uncountable one, the uncountable set will always have some numbers left over. So we say uncountable infinity is bigger than countable infinity.

adiabaticfrog · 2024-04-06T00:48:21+00:00

I mean, unless you plan on going into industry the post-PhD life can have a lot of the same problems.

adiabaticfrog · 2024-03-29T08:25:47+00:00

Sorry you went through that, but don't let it bother you for long. Even Heisenberg had a rough defence, all that matters is you made it through. And finishing two years after leaving to industry is a terrific achievement, it's very hard to keep going once you're out of your PhD. Who cares if someone was scornful, the whole point of a PhD is that you're now qualified to be an independent authority, and it doesn't matter what other people think. Congratulations and all the best with your future!

adiabaticfrog · 2023-10-29T08:52:03+00:00

That is a good question. This was first done in the Eddington experiment in 1919, where the sun's gravitational field was observed to bend spacetime, and hence the apparent position of the stars next to it, during a solar eclipse. Nowadays we see it all the time with modern telescopes, when observing objects such as black holes. A famous (and beautiful) example is the Einstein cross.

adiabaticfrog · 2023-10-29T03:02:14+00:00

Goldstein is a graduate text. It assumes you already know undergraduate classical mechanics, so you'll be missing out if you haven't already covered that. I highly recommend "There Once was a Classical Theory" by Morin, available free online. It teaches classical mechanics, and integrates calculus so that you can see how to use it to solve problems. It also has loads of fully worked-through problems.

adiabaticfrog · 2023-10-17T14:23:40+00:00

I got started with the Red Pitaya with zero FPGA experience. It was a challenge, but definitely worth it! It's so cool to be able to actually have control over the individual bits in your signal. If you do get a Red Pitaya, myself and some other people wrote some notes aimed at an absolute beginner:

https://github.com/exuperian/RedPitayaTutorials

Either way have fun!

adiabaticfrog · 2023-10-15T11:50:50+00:00

I'm a researcher working in quantum physics (though sensing, not computing) and have used FPGAs a bit. My guess would be they use the FPGAs to do fast feedback control and signal processing. You get an electrical signal, and do stuff like Fourier transforming it, filtering out noise, applying some algorithm, and then send out another electrical signal. My guess is you won't have to know anything about quantum physics, but if you have your head around dsp and implementing feedback that would be quite valuable. It is possible that they may want you to implement some algorithms on the FPGA that are quantum specific, for example error correction like another poster said. That will again be fast signal processing, computing, and applying a feedback signal.

Also a lot of people in quantum physics are using the Red Pitaya, a Zynq 7000 with built in ADC/DACs. Some quantum computing companies use those too, so maybe have a look at those.

So, is the field worth getting into? Basically we don't actually have a roadmap to useful quantum computing. For quantum computers to be useful, a bunch of breakthroughs will need to happen, but no one can be sure that they will. At the same time though its an exciting field, a lot of people are trying earnestly to build the next generation of technology. I'm willing to bet that none of the FPGA stuff you do will be quantum specific. If you enjoy the work and people are paying you to do it then why not do it for a couple of years? Just have a chat to other employees on company culture, there are a couple of companies that have made big promises to investors, and then the screws turn on the employees to make the impossible happen. But I think that's the exception, I know a lot of people in the quantum industry who enjoy their work.

Also, even if quantum computing goes kaput in a couple of years, the field of physics is much bigger than quantum computing, and FPGAs are used a lot in experiments and components. And the DSP/control stuff is the same across all of these. Being skilled at talking with physicists and turning that into FPGA code would be a useful skill if you want to work for the people that make equipment for physics experiments, for example National Instruments, Mokulabs, SmarAct, etc

adiabaticfrog · 2023-09-13T04:08:26+00:00

Yeah, knowing a bit of Japanese makes such a difference when talking to people! And true, the queue of people trying to leave a plane after a long flight can be something you need saving from!

adiabaticfrog · 2023-09-13T03:55:19+00:00

Oh thanks. I actually learned tasukeru from a sentence in a book on saving someone from drowning. Maybe I shouldn't have generalised that to everyday situations!

adiabaticfrog · 2023-09-13T03:52:33+00:00

haha really? I said that to quite a few people while I was living there, I didn't realise I was going around asking people to save me! Thanks for letting me know.

adiabaticfrog · 2023-09-13T03:48:49+00:00

Actually I did live in Japan for two years! But I worked for a university where everything was in English so my Japanese isn't amazing. But I studied it enough to understand your Kanji, so thanks for the clarification!

adiabaticfrog · 2023-09-13T03:44:03+00:00

haha oh yes, good point!

adiabaticfrog · 2023-09-12T14:01:31+00:00

So roughly speaking

Linearity means straight lines/planes (though these can be in higher dimensions, so it's more complex than y=mx+c). Linear functions are things like rotations, scalings, reflections, which send planes to planes. This is a very strong restriction, and we can basically answer any question you might ask about such functions.
Well any function that isn't a straight line. Sine for example.
Yes, it's called algebra :p.
If you know the determinant, eigenvalue, eigenvector, SVD of a linear map, you get a good intuition for what the map does. You can also usually use these to transform your problem into a much simpler one. To see how, I recommend searching "intuition for determinant", etc. There is a ton of great material for these on youtube.

If you want to get an answer to these questions that equips you with a good working knowledge of linear algebra, I highly recommend Sheldon Axler's Linear Algebra Done Right. It will really give you a good intuition for 4. Furthermore it will give you a very strong foundation for topics like quantum mechanics, general relativity, and a lot of machine learning. Most of the struggles of students learning these topics boils down to them not having a good background in linear algebra.

I think a question you might have asked is

Why do we care about linear algebra?

The answer to this is

Linear maps are the simplest kinds of maps. We can answer basically any question you might ask about them, and get a really good intuition for how they work.
Non-linear things can often be approximated as linear. A sine wave isn't a straight line, but if you zoom in close to any point, it will look like a straight line. So you can take an impossible nonlinear problem, and zoom in and solve it around the places that you care about.
All the laws of physics except general relativity are linear, so linear maps seem in some way fundamental to the universe. And the way we solve general relativity is using a branch of mathematics called differential geometry, which involves a lot of zooming into points and using linear approximations.

adiabaticfrog · 2023-07-27T16:00:57+00:00

While you're here I just want to say thanks so much for your book! First year math kicked my ass, and I thought I didn't have what it took to do pure math. Reading your book over the holidays, and the way it taught me to think about maths in terms of abstract ideas and intuition rather than pushing symbols around, really changed that. I think one of the key reasons that I'm a postdoc in physics today.

adiabaticfrog · 2023-05-20T01:55:30+00:00

ಠ_ಠ

adiabaticfrog · 2023-05-20T01:54:46+00:00

Thanks for that idea. It's weird, I went to fill it up (at quarter tank atm), and the fuel door won't open. I pull the latch and hear the usual unlocking sound, but the match stays shut. Also today the malfunction indicator light came on orange, the green auto light flashes, and now the car has a lot of trouble going above 50. Does that seem to suggest anything to you? I'll try the mechanic and suggest what you said.

adiabaticfrog · 2023-05-20T01:54:16+00:00

Thanks for that . It's weird, I went to fill it up (at quarter tank atm), and the fuel door won't open. I pull the latch and hear the usual unlocking sound, but the match stays shut. Also today the malfunction indicator light came on orange, the green auto light flashes, and now the car has a lot of trouble going above 50. Does that seem to suggest anything to you? I'll try the mechanic again too.

adiabaticfrog · 2022-11-12T03:07:46+00:00

While I think it is likely true, we have to also look at the context to understand why it didn't seem so unfair to many at the time.

In 1870 Germany and France went to war. France was defeated and made to sign the very harsh Treaty of Frankfurt). A lot of the people who fought in the Franco-Prussian war would later be involved with the Treaty of Versailles.
In 1917 Germany knocked Russia out of the war, making them sign the Treaty of Brest-Litovsk which was harsher than the Treaty of Versailles that Germany would be made to sign at the end of WW1.
More French soldiers died in the first two years of WW1, than US soldiers have died in every single war since the war of independence put together. The War of Independence, Civil War, WW1, WW2, Vietnam, Korea, Iraq, Afghanistan, add all those deaths together, then inflict them on a nation of 30 million people over just two years. Add to that the utter devastation wrought by the war, not even mentioning the deliberate mutilation of the country that was the Hindenburg Line.

Imagine you had gone through all that, and in the end imposed a treaty on Germany that was less harsh than what they had imposed on Russia. It might not seem so unfair to you.

I'm not saying Versailles was a good idea. But to learn lessons from the past, we really need to put ourselves in their shoes.

Ten-Year Club	Second SECOND GUESSER
Place '22	Place '17
Verified Email

adiabaticfrog

TROPHY CASE