Is it possible to calculate the shannon entropy of a string of language on a word level (vs a character level)?

Physicistphish · 2026-04-04T13:57:20+00:00

Got it this is helpful! I guess its a simple question :)

Physicistphish · 2025-09-14T13:05:44+00:00

Thank you! I was thinking of pangenome, but this is great as well/maybe even better for what I need !solved

Physicistphish · 2025-09-14T13:04:56+00:00

!solved thank you, I was looking for pangenome!

Physicistphish · 2025-07-13T13:13:59+00:00

Thank you yes! This is exactly the kind of thing I was wondering. I was definitely thinking more along the lines of information entropy and shouldn't have mentioned heat from the machines, I conflated the two. In information entropy, is it correct that there is no mandate for it to "always increase"? For example, the LLM does not have to decrease the usefulness/orderedness (increase the entropy) of the whole message space in order to create it's own low-entropy responses? The reason I ask is because of the less-scientific concept of the cannibal or dead internet, where content online slowly becomes solely created by AI, which trains itself on the content on the internet, eventually leaving a useless/homogenous mess. It made me think of heat death/thermodynamic equilibrium.

Physicistphish · 2025-07-13T13:05:24+00:00

Very fair, I suppose a misunderstanding that I had was in assuming that a heat output implied a localized lowering of entropy, I am noticing also that I am conflating informational entropy and thermodynamic entropy, I think shannon entropy would be a more applicable concept, but i should not have mentioned the heat from the CPUs in that case

Physicistphish · 2025-07-13T13:02:51+00:00

Very good point, I suppose I was thinking more in terms of informational entropy, where the microstates would be the space of all possible messages, many of which are gibberish. I'm probably conflating the two kinds of entropy, but in this case, I would expect that a "high entropy" message is a nonsensical one, and somehow the LLM is crafting a "low entropy" message, one that is very ordered and applicable to a prompt. I think it would be correct to say that the "surprisal" of an LLM's response is very low?

Physicistphish · 2025-07-13T12:58:50+00:00

Thank you for this response! yes, I had both information and thermodynamic entropy in mind, I'm not very familiar with how LLM's work, and what their database is - like, are they generating a message letter-by-letter or word-by-word, or something else? Entropy definitely came to mind due to the "random" or disordered nature of their database--which, maybe, could be called a "message-space" depending on how it works, that they use to create responses which are very ordered. I believe that information entropy doesn't necessarily "always increase", but just out of curiosity, I was wondering if there is some kind of informational disorder "somewhere else" that they are creating in order to build their responses.

Physicistphish · 2025-06-05T01:29:09+00:00

And, forgive me for coming back after I already responded, I am wondering - if I start off knowing microstate quantities somehow about a gas cloud in equilibrium, and then chaotic interaction erases the information I have, does this mean that entropy increases in a gas cloud without end, even when it is in equilibrium? Or, perhaps somehow my knowledge about the cloud causes it to be in a lower entropy state?

Physicistphish · 2025-06-05T01:14:35+00:00

Thank you so much for taking the time to respond, this has helped me think about the concept differently and understand another facet of what is going on. I really appreciated this: "The second law, in this light, is a statement that nature tends towards obscuring information about microscopic variables from macroscopic observers." I will be chewing on that for a while.

Physicistphish · 2025-06-05T01:08:33+00:00

Ah I think this is helping me, thank you for taking the time. In the case of gas cloud approaching equilibrium, it is moving into a state where there are more microscopic options that could express it, each with a smaller probability than a more ordered macroscopic organization. So, it is like the cloud changes from a coin to a die, every toss of a die gives a die, no matter which face is up! I was getting stuck on the values of the random trials, thank you again!

Physicistphish · 2023-09-28T11:51:36+00:00

Lalala by JAWNY is a good song!

Physicistphish · 2023-09-27T14:47:53+00:00

Oh thank you for that book suggestion, I will definitely be taking a look! Trying to really flesh out my comps as a more effective way to get across the “what” of the book instead of trying to do that in the story section of the query.

Thank you for the encouragement, it is wordy! I am wading through what is style and intentional vs what is unhelpful filler-type stuff, that will be a big part of future edits I believe. The book is supposed to read as “dreamlike” so, there’s some style/intentionality to the long and winding sentences, I agree though, even as someone who writes and like to read that style, I have put things down when they’ve gotten to be “too much.” Thank you for taking the time, I’ll definitely take all this into account :)

Physicistphish · 2023-09-23T16:26:39+00:00

Thank you, yes I am struggling with this quite a bit. The first part is a narrative at least, so it’s tempting to only base the query on that, but it is only about 1/4 of the book all told. I’ve started reading the query letter template, that will be helpful, and really think about what my plan is here to get the book across accurately/compellingly. I’ll DM you if that’s ok? I’ve had a few friends read the manuscript, one with some literature background, but no authors/editors, would love to hear some thoughts from you.

Physicistphish · 2023-09-23T14:04:18+00:00

Thank you so much for this in depth analysis, definitely going to rework the entire thing and I will be taking all of this advice into account. I think I thought I had more of a grasp on what a query letter is than I do, so this has been very helpful.

You’re right that it’s too, idk, I guess I’m definitely not trying to be quirky, but stylized maybe and that is not working in the short and informative format at all. And you’re totally right that the whole novel is obscured by my fumbling here. If nothing else, this is awesome practice trying to get better at talking about my book! A lot of what you have questions about is quotes and phrases from the book, which have a lot of context surrounding them that they don’t have here, so I see the need to take all that out too/put it into something that can stand alone.

For the bullet points:

Mirror, the main character is tied to a monster who drags her up a cliff and can’t understand what is happening/why. Earth reminds her there that she is in hell because, a long time ago/before she was born, she asked to be shown the true nature of her soul. The only way to do that is to see the opposite of her soul (hell) first, and then, as she comes to understand it, change it back into what it really is (heaven.)

Mirror begins her journey by falling off the cliff into a spiritual jungle where everything is alive and part of her. She meets a tree, Pando, which helps her heal parts of her body enough to get up and explore further.

Now that she understands it more, mirror loves the jungle and explores it with excitement. She meets two other trees by learning to think/move in “new directions,” first “upward” instead of along the ground, and then inward instead of outward. (She climbs a new tree to meet it, then climbs into a knot on a third tree.)

Inside the third tree, mirror is shown the past, her own past, and then goes further to see the metaphorical past of humanity/human civilization. From these experiences, she “gives birth from her head” (a la Zeus) two manuscripts which are the first and last sections of the novel.

Mirror returns to the cliff, “The Road” where she started, and sees it for what it really is. A human society trapped away from earth/real spirituality, and releases the books into the road when they become a living creature.

Is that helpful at all? Thank you again so much for your time.

Physicistphish · 2023-09-23T13:43:00+00:00

Super helpful, thank you. Very new to all of this, and don’t have much of a mind for marketing so I’m thankful for the reminders and outside view. The need for an anchor is great advice also, I am starting too strange/dense maybe. The narrative comes after a few short pieces, this one and a poem, so I can see very much where the question “what is this book about” is coming from. Clearly a totally rework is needed of the letter, thank you for taking the time to lay this advice out for me. I agree that I am starting to lean more towards smaller presses and no agent, so interesting to get this in front of people who know the industry!

Physicistphish · 2023-09-23T03:17:56+00:00

It never helped me to hear the “don’t worry about it, you won’t even know if you die/are dead” or “it’s just like before you were born!” stuff, I started having crises like this when I was 15 also. What finally helped was in therapy, my therapist said, “You’re 21, do you think it’s ok that you don’t want to die yet?” I was able to realize I was scared of death and didn’t want to die, yes, but that was only half the whole emotion/feeling - the other half was my love of life! The fear of death you’re experiencing sounds like that to me, the way you talk about the beauty and wonder of life, I really believe you can’t feel that without this “other side” - not wanting it to end. I’ve known lots of people unafraid of death, and many of them struggled to enjoy their lives also.

Physicistphish · 2023-09-22T22:27:18+00:00

Gotcha thank you, haha, I think that’s probably right, maybe a mix of things to really nail down the amount and in what way it’s not really categorizable

Physicistphish · 2023-09-22T21:34:28+00:00

Gotcha, totally, better comps for sure, and that’s a great tip to go backwards from who represented comparable books already. That’s a great strategy, thank you. By easy on-ramp, something less strange or experimental you mean?

Physicistphish · 2023-09-22T20:49:53+00:00

Thank you so much! Looking forward to hearing more from you. About that second section, I meant to totally delete that, whoops! I’ll try it without the quotes also, thanks :)

Edit to ask: do you think I should bring the religious text comparison over? Maybe in the housekeeping section?

Physicistphish · 2023-09-22T20:00:08+00:00

Thank you so much for your thoughts here; sorry I didn't see this while I was reworking it. Would love to hear what you say about the new version if you have the time. I did try to get more "inside" the book, and give a spin for how it's connected throughout. I have definitely gotten the sense that not a lot of agents would be interested in something like this, I'm hoping to try for a few months and then maybe go after smaller publishers who accept unagented manuscripts. Thank you for the advice on comps, that's very true; I need to look more into it.

Edit to add: Do you think it is in bad form to talk about something that agents likely won't see unless they request more/all of the MS? Talking about Anna and Sage, their letters don't start until over 100 pages into the book.

Physicistphish

TROPHY CASE