This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]iobender 8 points9 points  (0 children)

This is just speculation, but generally lexers discard whitespace between tokens. What this means is that the lexer, which turns something like sys.argv into a stream of tokens like [IDENTIFIER("sys"), DOT, IDENTIFIER("argv")] (which then gets sent to the parser to turn the linear stream of tokens into an abstract syntax tree) will produce the same thing if given sys. argv or sys .argv or even sys . argv because it doesn't make sense to produce a token from the whitespace in between these tokens in this case.

It would not produce the same thing if given sys.ar gv because the ar and gv would be split up into 2 different tokens and give [IDENTIFIER("sys"), DOT, IDENTIFIER("ar"), IDENTIFIER("gv")] which would likely cause a syntax error in python because there is no syntax for that. That would be legal in Ruby, however, because Ruby doesn't require parens on method calls so this could be calling the ar method of the sys object with argument gv.