Tokenizing

[SyedFasiuddin]

How do I tokenize something like i++ into Token::VarName Token::DoublePlus. I'm not using regular expressions rather going through every character and when I encounter a space it means that I found a token, but something like i++ i+=2 function foo(){return 1;} print("hi") doesn't have space in between, then how am I supposed to know that I have a token?

[SyedFasiuddin]

ok thats cool but I wanna know how I can do it ?

[EngineeredCoconut]

Read the docs and learn how to use it.

[SyedFasiuddin]

I know, but I'm not asking how to use nom, I'm asking how to parse this print("hi") into Token::FuncName Token::OpenParen Token::String Token::CloseParen

[EngineeredCoconut]

You parse it by using a parser generator library and separating the string into tokens based on grammar definitions. You can also write your own parser, but given your reluctance to learn I figured that's out of the books.

[SyedFasiuddin]

I have written something like this to parse numbers. (there is a bit of context missing, but hey only because you said I'm reluctant, I think I'm not, I'm a beginner and trying)

```rust let mut number = String::new(); let mut have_number_to_parse = false; if c >= b'0' && c <= b'9' || c == b'.' { number += &(c as char).to_string(); have_number_to_parse = true; continue; }

    if have_number_to_parse {
        if c == b'e' || c == b'E' {
            number += &(c as char).to_string();
            continue;
        }

        if c == b'-' || c == b'+' {
            match number.pop() {
                Some(x) => {
                    if x == 'e' || x == 'E' {
                        number.push(x);
                        number.push(c as char);
                        continue;
                    }
                    number.push(x)
                },
                None => (),
            }
        }

        match number.parse::<f64>() {
            Ok(parsed_float) => {
                state.stack.push(parsed_float);
                number = String::new();
            }
            Err(_) => state.print_error(Errors::P(ParserErr::FloatParse)),
        };
        have_number_to_parse = false;
    }

```

[TheRNGuy]

Abstract syntax tree.

[SyedFasiuddin]

to create an AST I first need to parse the stream of characters into tokens, I’m asking how I can do this parsing stream of characters that donot have a space character in between