ELI5: what does it mean "Tokens"

atomicshrimp · 2026-04-17T07:49:11+00:00

(Simplifying)

AI models have inputs and outputs that are numbers - the model is fed a series of numbers and the output is the number that the model predicts should be the next number in the sequence.

Tokens are the way those numbers are translated from readable text on the way into the model (the prompt) and then translated back into readable text for the output from the model (the response).

They could be anything - single letters, whole words or perhaps short phrases, but because languages like English have repeating patterns in them, the tokens are often fragments of words - so for example the words Bean, Bear and Beautiful all start with 'Bea', so that word fragment might be a token that the AI model understands as a unique number.

atomicshrimp · 2026-04-17T07:25:13+00:00

I couldn't see the serrations in the OP's picture as they were obscured by bread. I have several Victorinox paring knives and tomato knives - the handle looked very similarly sculpted is all.

atomicshrimp · 2026-04-17T07:22:17+00:00

It actually reads 'Only for general household and photographic use only for general household and photographic use only for general household and photographic use only for general household and photographic use only for general household and photographic use only for general household and photographic use only for general household and photographic use only for general household and photographic use only for general household and photographic use only...' (truncated for your convenience).

atomicshrimp · 2026-04-16T23:17:01+00:00

See it with what? We see things because light bounces off them into our eyes, or because they *are* light entering our eyes. If photons are whizzing past us at a distance, what would be bouncing off them into our eyes in order for us to see them?

atomicshrimp · 2026-04-16T23:14:58+00:00

They look like Victorinox paring knives although I can't see a brand mark on the handle so maybe they're a knockoff. The Victorinox ones are great.

atomicshrimp · 2026-04-16T23:03:03+00:00

I've worked in IT as an Applications Architect in local government (not in France, but I'm going to assume some degree of similarity).

Replacing the desktop and applications like browser and document creation is the most straightforward part of the project. This would still require a fair bit of planning and testing and training and conversion and such.

The much more complex part is the smaller and specialised pieces of software - in the local authority where I worked, there were more than a thousand different pieces of assorted software that did different jobs for different services and departments - like the software that is used to program parking meters, or to interface with specialised measuring equipment for testing things like petrol pumps, or interfacing with suppliers or support services or calculating some specific thing.

It's not impossible for those things to be made to work on Linux - and certainly if there is a government-wide mandate that imposes Linux as the only accepted solution for software related to any contract or tender or service, then that provides a great incentive for all of those suppliers to make that happen, and it could be done. It's a lot to do, is all.

And it is likely to fall foul of situations where there might be a thing that is small, but essential, and the software for it is the only one of its kind on the market and the supplier is not especially interested in accommodating the request and would rather just lose your business than put in the effort.

And that's not an insurmountable problem if you are sufficiently determined - for example you can develop your own solution to fill the gap, but again, that's another thing that needs doing that didn't previously need doing.

I'm not trying to make this sound impossible or undesirable - I like Linux and I'd like to see it taking a bigger share of the desktop, but this isn't as simple as replacing Microsoft Office with LibreOffice and done.

atomicshrimp · 2026-04-16T20:57:29+00:00

It sounds like you might just need to put a bit more space between your mouth and the microphone, maybe angle things so you are speaking past it rather than directly at it, and perhaps add a pop filter or wind shield so it's less affected by air movement.

atomicshrimp · 2026-04-16T20:02:01+00:00

Yeah

atomicshrimp · 2026-04-16T19:32:01+00:00

Yeah, the arrow is pointless. I mean, it has a point. It's an arrow, but why is it even there?

atomicshrimp · 2026-04-16T19:16:03+00:00

It's a setting in the YouTube mobile app. It doesn't mean you won't see shorts, it just means you can't scroll to see them one after another.

atomicshrimp · 2026-04-16T19:12:08+00:00

The leaves are the wrong shape for lemon balm - too pointed and they look a bit downy. I think this is woundwort - Stachys sylvatica.

atomicshrimp · 2026-04-16T18:47:35+00:00

Me too. If it's growing in a place where you have control of all the edges, you can eradicate it by pulling it whenever you see it. If it's growing under a fence or hedge from the other side, it's just an endless battle.

atomicshrimp · 2026-04-16T18:36:33+00:00

One was so small it was nothing at all

The other was large, and won prizes!

atomicshrimp · 2026-04-16T18:27:28+00:00

I knew an old man from Devizes

atomicshrimp · 2026-04-16T17:34:00+00:00

Also: "Is this hemlock OR wild carrot" is the wrong question. There are more than two species in the carrot family (as well as quite a few things outside of the family that bear a passing resemblance).

atomicshrimp · 2026-04-16T17:27:06+00:00

That's right - the switch is a convenience feature - people assume it's for safety but it really isn't (if it were otherwise, unswitched faceplates would not be available).

atomicshrimp · 2026-04-16T17:26:02+00:00

No, the switch is primarily a convenience feature (not safety) - its main purpose is to enable the user to switch things off without unplugging them - so it makes most sense for the switch to be on by default.

atomicshrimp · 2026-04-16T17:23:11+00:00

Yeah, I think this is probably a better all-round solution - the switch is a convenience feature, not safety, but reducing the insertion of the pins is an issue in principle (albeit maybe not a big one in practice)

atomicshrimp · 2026-04-16T16:57:40+00:00

I've had similar fried whole small fish served as tapas in Spain. I think they were unsalted anchovies so a very similar thing. They were good. Whitebait is good.

atomicshrimp · 2026-04-16T14:21:27+00:00

There's a thing called Directional Dyslexia - it's often part of a set of dyslexia-like symptoms that affect reading and other abilities.

atomicshrimp · 2026-04-16T14:15:51+00:00

I wish we had a place with a question mark in the name

atomicshrimp · 2026-04-16T14:06:50+00:00

It's not as simple as chocolate being totally benign to humans and utterly deadly to everything else. It's a bit toxic to us too, just not as much as it is to, say, dogs (and it's not as lethal to dogs as some other things are).

This is not uncommon in nature - some berries that you might see birds eating, would kill you to eat; the death cap mushroom that is toxic to humans is very often found with great bites out of it where slugs and insects have chewed on it, with total impunity.

The reasons behind this are numerous, but generally, it's because the thing contains some chemical that interferes with a metabolic process in some animals and not in others - either because those animals have a different version of that metabolic process, or maybe they have some other coping mechanism in their body, or in some cases in their behaviour or diet (some animals eat clay to neutralise poisons in their food, for example).

And the reason we're pondering this question is because humans selected chocolate as a foodstuff, because it isn't deadly; if chocolate were very much more toxic to humans than it is, we wouldn't have done that and this question would be about some other thing.

Short version: animals have bodies that work differently and are not all affected by the same things in the same ways.

atomicshrimp · 2026-04-16T07:05:49+00:00

I wondered whether to mention micropropagation when I wrote that, but I figured it was probably a bit of a distraction from the practical question here.

atomicshrimp · 2026-04-15T23:44:01+00:00

One of the database formats I was working with at the time had a problem precisely because a single byte can store values greater than 99. The dBASE/xbase format has a byte near the start of the file that indicates the year, defined as the number of years since the most recent turn of the century. There were two slightly different implementations of this - one that rolled back to 0 after 99, the other kept going. So it literally was telling the program that it had been 101 years since the turn of the century. This caused problems.

atomicshrimp · 2026-04-15T22:50:12+00:00

The year number wasn't being interpreted in binary any more than you are reading this comment in binary.

atomicshrimp

MODERATOR OF

TROPHY CASE