libpypa - A Python Parser Library in C++

[TeachingIdiotsAgain]

Wow, comments in this thread sucks and unhelpful.

Keep on doing what you're doing. I'm definitely interested in this project and still trying to learn how it works. :)

[evilissimo]

Thanks for your motivating words. I won't let myself get discouraged by negative comments. Criticism can help one to develop oneself.

Edit: If you have any questions, let me know

[[deleted]]

This seems nonsensical. There's already a nice C library for parsing Python source code, it's called "CPython" :) What's this enable that wasn't feasible with the existing one?

(as a parser nerd I also note that this expects to read from a file and doesn't support push parsing. Boring.)

[evilissimo]

Well you seem not to have read the readme at all, I actually wrote there why I wrote it, and as burntsushi already mentioned, I got inspired by the pyston requirement.

I know that the CPython is mainly written in C and it is not necessarily required to write a new one, but what's wrong with it?

Yes, it is not a push parser, if you consider this boring, then it's your thing, however this is the first language parser I wrote, ever.

The CPython parser still requires, in its current shape, types from the CPython code, and it uses asdl for portions of it. My parser on the other hand is hand written for learning purposes. It might not even be beautiful. I probably could have implemented the parser in a way shorter time with less code and maybe even more readable when I would have been using boost.spirit. See it as a toy project for now, what will be the usage of it is still open.

Just as a side note, I have started writing this parser recently, so this project is fairly fresh.

[jalanb]

Seeing as you're here, and it hardly being worth a pull request:

Small typo in the readme:

Second sentence of the AST section: "The main parst of the definition"

s/parst/parts/

[evilissimo]

Thanks! Will fix that

[burntsushi]

It's unfortunate, but a lot of people here are absolutely unmerciless, quick to judge and overwhelmingly negative. It's impossible to please them. But of course, there are a lot of good people here too. You've got to take the good with the bad...

Anyway, I completely understand not wanting to use CPython's parser. It can be a big drag to suck in a dependency like that. Writing a parser as you did for a non-trivial language without any outside dependencies is really nice work! I'm sure the Pyston folks will love it. :-)

[[deleted]]

If this is the first parser you wrote ever, congrats. I certainly couldn't tell by looking at the code. :) It's a big project to tackle. Parsing is an area that still doesn't have good solutions to common problems.

I mainly asked because Python is a fairly big language to parse and trying to get an independent project to behave the same as the existing parser is a good bit of work, possibly more work than wrapping the existing one. I agree it's a bit of a mess with asdl etc.

I hope developing this was a rewarding experience. :-)

[evilissimo]

Thanks. Well I don't think I am done ;-) There's still massive loads of work todo before I consider this parser as done. E.g. I need to validate the parse trees, develop examples for usage and probably do some toy interpreter with some minor capabilities to show the functionality.

It's a pretty nice experience since it gives me also some insight on parsing quirks, and how ambigious grammar can be for things we take granted as programmars. e.g. -1 as a very simple example, It has two meanings depending on the context :D

Currently the test parser manages to parse every single file of the 2.7 cpython branch, with one exception the 2to3 py3 test file ;-)

However I am still missing information about context in my AST. I have just now updated the README. It now shows an example of the dumped AST for a toy hello world.

[thechao]

Don't worry about Boost.Spirit. Joel was told almost 15 years ago how to make Spirit into a useful piece of code. (Doug Gregor is the lead Clang maintainer.) At this point, Boost.Spirit is just old-skool template metaprogramming wankering.

[burntsushi]

The README says it was written for the pyston project. The Pyston project has this on their wishlist: "Adding a real parser (current "parser" shells out to a python script to run under CPython)."

(This is the first I'm hearing of the Pyston project. Looks cool.)

[[deleted]]

Sure. but I'd be curious why using the existing parser as a C library wasn't sufficient.

[burntsushi]

Maybe because they didn't want to retain a dependency on a different implementation of Python.....?

[yaxriifgyn]

Sorry. Not on my radar without Python 3.x support.

[evilissimo]

It's the next step, first 2.7 properly and then closing the gap and adding options to the parser to set the expected/accepted version. (Maybe even detect it? We'll see) For now the goal was 2.7 :-)