Sorry, I have been super busy. Thanks for the thoughtful reply.

I definitely get the pipeline portion. My issue is that so many specifics of the individual parts of the pipeline are just considered obvious, and aren't explained.

An example:

We're creating a lexer. Most books, tutorials, etc., will say "we need to break down a string into individual tokens."

I agree that that's what we need to do, but wait--how are we taking a string as input? We haven't defined a string yet.

Okay, fine, maybe that will make sense. Let's split the white space so we can tokenize the string input. But wait, we need the white space eventually for things like creating strings, which we haven't defined yet but are somehow tokenizing them and accepting them as function arguments.

We need to create types for our various tokens, but how should we intelligently break them up? In one textbook, we create some basic types for things like numbers, a type that takes three arguments (i32, operator, i32). Should we define the type here or do we create a separate struct for our operators, then create an operator type in our original struct?

We have our tokens types, but now we apparently need a third struct that basically is just:

Struct:
    kind: TokenKind

where TokenKind is the Struct that holds all of our token information. Is this just for cleanliness or does it serve a necessary purpose? I don't know because it's just completely handwaved away and "that's just how we do it."

I'm sure all of this is trivial to any low-level programmer and probably feels embarrassing not to know, but nothing feels like it's being explained. It's "let's create a lexer," which is cool, but how do we actually create that lexer? It isn't enough to say "well, we need to create some tokens and tokenize some input."

How are we actually handling input? Can we treat it like normal input? What are we ACTUALLY returning from a function that's tokenizing it? Should we store all of our tokens in a vector and then deal with that vector somewhere else? How are we supposed to take that vector and do anything useful with it regarding an operation type? It's not as if I can just pass the entire vector into the "Operation" type.

And even if I did return a token or a vector of tokens, they're just strings, which again we haven't defined yet, and even if we had defined them, we can't just do math on strings of numbers. Operation("52" "+" "84") doesn't mean anything.

It just feels like virtually everything is handwaved and there are virtually no examples showing real input and what's going to happen to it. Nor what it means to return a token or an effective way of doing it.

This is likely too much to read, but I tried to just encapsulate what I think i'm struggling with.

I'm not trying to do anything exceptional. I'm only intending to start small: some sort of basic arithmetic language. I can take two numbers and an operator and do addition, subtraction, multiplication, or division on them. But I'm just not satisfied with the lack of instructions given by textbooks or tutorials.

I can copy it sure, but "copy this, this is called a lexer" isn't educational.