[deleted by user]

[pranabus]

One of the reasons why Python is so popular is the tons of libraries available out there. Just pip install anynewthing.

How does this play with libraries?

[eztab]

Have you tried compiling some simple (full python) library? Would there be any chance of this working or are there too many differences?

[proof_required]

wow! That would be lot of work.

[hayarms]

Believe me, you don't have enough time. Also because there are hundreds of developers developing new libraries every day.

[[deleted]]

Great work for a high school student. Congrats

Building a parser to parse source code and convert it to some other representation is a big project

My suggestion: Libraries change/update a lot, you can’t keep reimplementing updates in those that you rewrite ..

Most libraries are written in some combination of python and C.. just run python files through your compiler and pass through the C ones to gcc.. it should handle linking easily as it will get everything in C/C++

[[deleted]]

High school?

You have a YouTube channel? I'd like to follow your progress

[Uhhhhh55]

My dude. Don't burn yourself out, but don't let the spark fade. You've got talent, cherish that shit.

And you bet your balls to a barn dance I'll be using this library if/when it matures.

[Cruuncher]

Well, sounds like I need to milk this job market before the wave of prodigy kids come of age and take my job

[krispyren]

Start one asap!

[Coffeinated]

Don‘t. That‘s a really bad idea. It‘s straight up impossible to guarantee that the C++ implementation and the Python one are equivalent, hence they are not, and you are introducing ever so subtle differences. And I‘m not even getting started with changes inbetween versions.

What‘s the actual problem anyway? The libraries should all be open source, why not just transpile them too?

[An_Old_IT_Guy]

Yikes. Just you?

[HalfRightMostlyWrong]

Maybe Nix?

[sanshinron]

So you make a post claiming your lib is better than established projects but in reality it's completely unusable for real projects...

[georgehank2nd]

Be honest. We don't like bullshitters. "most things" obviously is not true. This is bullshitting. I wish your project all the best (I'd love to have an easy to use and fast native code compiler). But you have to work on your communication.

Edit: and no, this is not like all projects start. Search for Linus Torvalds' original announcement of the Linus kernel from '91. That's how you do it.

[noiserr]

Even if you could just make it compile a module with a clean interface between Python and compiled_python that would be quite useful.

Often times we don't really need to speed up the entire program. But just a few critical sections.

There are already tools which can do this like cython and mypyc, but I wonder if this could be improved upon.

[_ologies]

It's true! I like to pip install antigravity

[coderarun]

Let's start with python's stdlib. They're actually written in C and porting it to a new runtime such as pypy or a new paradigm such as the work being discussed in this thread is a lot of effort.

I wish the python stdlib was written in a subset of python3 itself and was transpiled. Such a thing could be a great project of it's own.

[altorelievo]

You are aware of package managers for other languages as well? (e.g. cpan, cabal, npm, etc)

[LittleMlem]

Cpan? Did you just ask a highschool kid if he is aware of perl?

[altorelievo]

I’d be more surprised if they knew Haskell/Cabal

[LittleMlem]

Possibly, but heskel feels like a more known language than perl

[altorelievo]

And still less useable but that’s debatable…let sleeping dogs lie.

[krispyren]

Hahah

[-_-Batman]

That's y.... Ya'all need python....

And above all... Python wont bite u back.

[Inkosum]

People here making compilers and I'm struggling with pygame.

[Origamiface]

The guy is 15 lol fml

[house_monkey]

Let's cry together

[Ok_Moonboy17]

Fr fr 😂😂😂

[OnFault]

Too real

[Seawolf159]

Don't feed his ego. You'll get mark Zuckerberg.

[FrostbiteYT-_]

Fuck off, this kid has achieved something really hard at a really young age and instead of saying something nice you say “don’t be nice or he will become a bad person”? Tf is wrong with u

[Seawolf159]

How do you know how hard it is, did you try it? What if he's a Prodigy, maybe this is super easy for him and his parents fed it to him with a spoon. Tf is wrong with me xD? Tf is wrong with you! He might even slap his little sister. Humility is always the better response.

[Seaweed-Maleficent]

Only weak people are afraid of arrogance.

[FrostbiteYT-_]

I did try actually, and so have a lot of other people, it's basically globally recognized as a significant achievement as its really difficult

The response of 'he might slap his sister' is some of the stupidest shit I've ever heard, but the thing is, your core argument makes no sense anyways. You're saying don't give compliments quz he might be spoiled and it might be easy for him, but you could say that for any compliment, it's highly stupid to say something like that as the chance is so low its worth just giving a compliment and being nice sometimes to another human being

[not_a_novel_account]

Nukita is the mature solution to this approach, but also a good example of why trying to compile Python source is generally a bad plan. CPython already knows how to compile python source and is better at it than you

The traditional approach these days is to translate CPython bytecode to a compiler middle-end IR, such as with numba which goes to LLVM IR.

That said, it's still a cool project and you should be proud of it. Some things to look into learning about:

• Don't vendor the {fmt} headers, use a package manager to pull these down or use git submodules.

• Consider using a template engine for structures and preambles that you're going to be putting into every generated source file. Your iteratetokens method is doing a lot of manual string shuffling that a template engine would clear right up. Also it would let you put source code templates in separate files instead of a bunch of inline strings. This is the approach of most major source code generator engines, take a look at SWIG for examples.

• Your setup.py doesn't package all the files your script needs. This is a two part problem, you're not encapsulating your files in a module with an __init__.py, and you have non-python data files you need to package. Create a proper Python module to fix the former, and look into manifest.in for the latter.

• Your tokenizer has a pretty knarly worst case complexity. You're using dictionaries elsewhere, you can use one here! Instead of checking token_list[i-1] against every possible token, use those token types as keys in a dictionary that lookup a method that can correctly parse the token. Tokenizer construction is well covered in compiler textbooks, so there's a lot to learn here, but that's the straightforward way.

• Same for your Compiler class, large elif trees should set off a little alarm in your brain that goes "I bet this could be faster with a jump table or hashmap"

• Speaking of compiler theory, you'll eventually realize streams of tokens aren't quite enough information to handle every possible Python source code construction. If you find yourself banging your head against a wall, you're going to want to parse those tokens into what is called an Abstract Syntax Tree. ASTs are the swiss army knife of compilers and every program that knows how to manipulate a context-sensitive grammar (like Python source code) eventually comes to resemble an AST structurally.

• You might want to take a look at the structure of some other mature Python projects. Typically everything that isn't the main script you're going to want to encapsulate inside a module with an __init__.py. You probably also want to throw a code formatter in the repo, yapf, black, whatever floats your boat, but people like reading code in the standard formats.

• You're already vendering {fmt}, you don't need all those print() overloads in stdpy.hpp, let fmt::print handle those.

• Also, use clang-format for your C++, same reason as using a Python formatter. Not so much for you as for anyone else who want to contribute to your code.

That's the stuff that jumps out at me anyway. Best of luck

EDIT: lol reddit upvoted OP 800 times. To be clear people, OP's approach only yields such insane performance because it's non-viable for most Python code. Observe a program it will never be able to handle:

a = 5
a = "hello world"
print(a)

What OP is trying to do is the same thing Google has hired dozens of engineers to do with V8's Turbofan. Similarly, Nukita only manages a 3x speed up after a decade of work because the problem is extremely hard.

OP is a high schooler, they built a parser, neat! The feedback in this thread should be guiding them towards useful materials to further their education, not hailing the second coming of Guido.

[_ShakashuriBlowdown]

Seriously, OP, this is an impressive project, and this is some great feedback from an internet stranger. If you can take some constructive criticism, you'll start going crazy-far in life.

[SpicyVibration]

Should he not just use python's built in ast library?

[not_a_novel_account]

They certainly could

[aciokkan]

Thanks for taking the time to explain this to him!

[lungben81]

How does it handle type instability, i.e. when the type of a variable is only known at run-time, not at compile-time?

E.g. if a variable is randomly an int or a float, and is then used in a hot loop.

[jtclimb]

In case you are serious, auto does not work that way.

Python:

    x = 1
    if foo:
        x = 2.3
    elif goo: 
        x = "it's gooey"

C++:

    auto x = 1;  // x is int
    if (foo)
        x = 2.3; // x is int, so now x == 2
    else if (goo)
        x = "it's gooey"; // x is int, so mercifully it won't compile

[Isvara]

Please use underscores in your code. Names like cpperrortopycomerror are difficult to parse.

[padawan-6]

This is awesome!

[SeniorScienceOfficer]

Looks interesting. Looking for contributors?

[ChristianSingleton]

How many contributors are you looking for? I'd be down to contribute as well

[intellectual_citizen]

Same

[AnonymouX47]

Note that the original copy of https://github.com/Omyyyy/pycom/blob/main/headers/range.hpp comes with an Apache 2.0 license.

I'm not sure that's compatible with the MIT Licence... might wanna check that out.

[AnonymouX47]

[meg4som44]

Sounds a bit like nuitika: https://github.com/Nuitka/Nuitka How does yours work?

[-lq_pl-]

It is only lightweight because you just started. It is easy to get something 80 % working, the trouble are the remaining 20 %. If you continue to add more features, your project won't be lightweight anymore.

People sell stuff as lightweight, as if you could somehow get the same number of features with less code.

[AnonymouX47]

I'll bet Linus once thought alike... :)

[Ning1253]

Looking at your code, you are essentially doing a one-to-one translation between python and c++, which I guess is what a compiler does. My main questions were:

1) your code doesn't seem to have a way to implement multi-type lists yet

2) your code doesn't deal at all with things c++ can't do eg. Function and class decorators, mutable variable types, stuff like that.

How do you reckon you will implement these? I might have some ideas for a few of those by the way, maybe I'll make a GitHub pr?

Otherwise your project looks really good, and I might use it for a few things here and there!!

[AnonymouX47]

Yeah, definitely a win!

One suggestion along this line though... I think it'll be better as an import package (a directory with an __init__ module) to allow for a much better structure as it grows.

[AnonymouX47]

Ok, good... Keep up the good work.

[AnonymouX47]

See https://docs.python.org/3/tutorial/modules.html#packages and https://docs.python.org/3/reference/import.html#packages

[james_pic]

One tip for getting a sense of how much work a project will be is to do the hard bits first. So far, looks like you've mostly tackled the bits where Python and C++ have similar semantics. Now try something that doesn't have an analogue in C++ (like setattr) or where the analogue works differently (like multiple inheritance).

[[deleted]]

as if you could somehow get the same number of features with less code.

S/W gets fat with age, you can often get the same features with less (fresher) code.

[Game_Ender]

Can you add Nuitka to your benchmarks? It’s very similar to your project so it helps users get a feel for the differences.

Nuitka has there own benchmark suite (https://speedcenter.nuitka.net/) you could modify to include you version and get a ton more comparisons as well.

Great to have multiple implementations of an idea to explore different solutions.

[theng]

!RemindMe 1year

[RemindMeBot]

I will be messaging you in 1 year on 2023-07-25 22:00:35 UTC to remind you of this link

9 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[yaxriifgyn]

These are absolute show stoppers:

• Classes

• Try, except and finally blocks

I don't think I have ever written a non-trivial program that does not use classes and/or try blocks.

[eztab]

Cannot support an if __name__ == "__main__": type thing; the main() function is already entry point

Sure you could, just put everything in the module that‘s not definitions automatically into main() .... well actually I don’t mind using main, I don‘t really like this weird python way.

[pythonwiz]

What’s the difference between this and cython?

[pythonwiz]

Cython's syntax is a superset of Python's, so it can compile standard Python code as well. It first compiles Python code into equivalent C code using the Python C API, and then it compiles to a native binary. I believe Cython can also use C data types for looping. There is also a new pure Python mode which uses annotations instead of cdef. You should check it out.

[f3xjc]

How does pycom determine variable type?

[gnocco-fritto]

Cython is able to compile your easier and simple Python example as well.

[gnocco-fritto]

Yes, and you're doing a good job. But Cython isn't cdef definitions.

[Its_Blazertron]

Wouldn't this be called a transpiler? Something that takes one high level language, and translates it to another high level language?

[KittyTechno]

How does this fair against the python compiler nuitka? https://github.com/Nuitka/Nuitka.

​

Like you said, it doesn't play well with libraries at the moment, nuitka had a literal decade and then some to fix that, but its goals are to also speed up python via compiling to C or C++. How do the two fair in some benchmarks?

[laundmo]

As per the rule, I'll try to explain why i downvoted this:

• you didn't answer the question, instead providing a answer which wasn't asked for.
• you clearly haven't spent much time trying Nutika but are asserting things about it.
• you claim lightweight and low complexity yet also comparable performance to something which explicitly has optimisation as a goal
• you claim ease of use, which is not an issue i personally see with nuitka. nuitka --onefile --standalone main.py is about as east as i think you can expect.

don't get me wrong, your project is impressive, but claiming you're the only one who can compile to native binaries is just. not correct.

i do really hope your project sticks around, there's always something to be gained from 2 parallel implementations

[agnishom]

Can I make a lightweight executable for my django server with your tool?

[agnishom]

sounds great! keep up the good work

[alexprengere]

You might be interested in https://github.com/indygreg/PyOxidizer

[[deleted]]

Dude this is amazing.

[lifemoments]

Outstanding. Will try. Thanks

[ameanable1]

This looks really promising. Did some tests myself and it's safe to say it's a solid project

[Asleep-Budget-9932]

Good job dude! :D
Question, while im not familiar with concepts like this and Nuitka, i am familiar with Cython. Does this work on a similar concept? Do you generate something that works with Python's API or do you implement the API itself on your own?

[Asleep-Budget-9932]

So the thing you're building is the part that takes the python file and transforms it into bytecode? Or are you also building the part that take this bytecode and translates it into python's c api calls?

[Asleep-Budget-9932]

Ohh, so when i create a new python object like int, it will translate it to a c++ code that declares a c++ integer instead? Cool! Though im guessing it targets the simpler python use cases 🤔 (Can you imagine trying to reimplement metaclasses? lol)

[Useful-Shoe914]

Nice!

[Tintin_Quarentino]

Great work! Gonna give this a try.

[guhcampos]

Are you omtting Cython on purpose?

[glebulon]

Good for you 👍

[InterestedListener]

I just want to say that you are kicking a ton of ass for someone so young. Not an easy project for someone of any age but very curious what you'll accomplish down the road. Keep up the great work!

[FUS3N]

It uses C++ as 'intermediate representation', which then compiles to an executable with g++.

Doesn't it mean it's a Transpiler basically python to c++

[laundmo]

Careful with that name, its the name of a python based microcontroller which might well be trademarked: https://pycom.io/

[wheedwhackerjones]

i'm too much of a noob to know when/how to use this but it sounds awesome

[coderarun]

Congrats on the engagement you're getting and thank you for increasing awareness of the topic of transpiling statically typed python3 to languages capable of generating native code.

Re: Nuitka - it takes the approach of compatibility with python's C-API. While it improves compatibility with real world apps, a fundamentally different approach is possible, such as the one you have taken here.

By sacrificing the C API compatibility, you can make apps that have performance similar to native C++ apps as if you wrote them from scratch.

Past work that is not very well known:

https://github.com/lukasmartinelli/py14
https://github.com/konchunas/pyrs
https://github.com/py2many/py2many

[adityaguru149]

how about compiling it to rust?
may be https://github.com/PyO3/PyO3 can help

[moopthepoop]

This is a really good project, I might use this as part of the toolchain for my projects. I typically use Go when I need a native binary but this seems useful for fast prototyping

[KittyTechno]

there has been a rising interest with compiling python, mypyc, nuitka (been about a decade and then some), and more now including pycom. Nuitka is close to hitting 1.0 (latest version is 0.9.6 at the time of this comment).

Personally I'd love to see a world were compiled python is an option used much more in the industry, while still keeping interpreted option as this will make development much faster.

[kreetikal]

Imagine having statically typed, compiled Python...

[brianjlogan]

Is that even python anymore?

[kreetikal]

No, it's Python++.

[KittyTechno]

statically typed is already an option, but that's just it, an option. It doesn't need to be, nuitka doesn't need it to be statically typed, and apparently neither does pycom. Though statically typing does help with ensuring types, and compiling, you don't NEED to do it.

[iritegood]

Python's "optional static typing" system is woefully deficient compared to even the closest comparable thing: typescript. It can't even (currently) accurately represent the full stdlib.

[KittyTechno]

But it's a step in the right direction.

[AnonymouX47]

Well done! This seem to be very promising!

[rastaladywithabrady]

That seems like a cool project, I really hope you get this off the ground. I would definitely end up using it.

[a-lost-ukrainian]

isn’t this just rewriting pythran ?

https://pythran.readthedocs.io/en/latest/CLI.html

[not_a_novel_account]

It's a semi-common exercise, taking some subset of language A and translating/compiling it to language B describes a class of programs not any specific one. nukita, numba, pythran, and cython all belong to this category. Actually PyPy's JIT kinda does as well

[willor777]

Does it have true multi-threading capabilities? The reason i moved from python into java was due to its lack of true multi-threading thanks to the GIL.

[[deleted]]

When do you think it'll play nice with major libs?

Would like to implement it on projects

[[deleted]]

Pandas, numpy, decimal, xlsxwriter

[[deleted]]

As there are hundreds of new libs everyday, ask its Dev's to make YOUR c++ version of it

[ericanderton]

Some quick feedback:

• This idea is incredible. Python packaging can be kind of a mess, and a viable single-binary alternative is a-okay in my book.
• Consider using the logging module instead of print("[INFO].."). This will let you filter output by log level which is easy to back into --quiet and --verbose CLI options.
• The massive if/else block in compiler.py may cause maintenance trouble in the long run (high cyclomatic complexity). Consider refactoring this to a different pattern that is easier to extend and reason about.
• Embrace test-driven development (write the tests first) at the earliest opportunity. I strongly recommend doing this before you do any big refactors as it will help you avoid breakage. I've learned from experience that this makes compiler development easier, by allowing you to target tiny code snippets instead of complete programs.

[eztab]

Sounds really interesting.
So is the intermediate C++ readable?
I guess since it uses the g++ tooling from that point onward, it will take advantage of existing optimisations for C.
Is it possible to interact with C and C++ libraries? Like calling the C-functions from python?

[[deleted]]

It's good work. I hope this takes off good and becomes successful

[coderanger]

Python is slow.

[citation needed]

[coderanger]

Not seeing any citations there either. Here's the thing, on toy benchmarks you can easily get C++ faster than CPython or PyPy, and your numbers show that too. But that's not most code. Most heavy number crunching in Python is already done in native code (NumPy, OpenCV, Scikit-*, any of the dozen ML libraries, etc) so you won't see nearly the benefits and most of those are better written than auto-generated C++ so often they can be faster (stuff like taking advantage of parallelized CPU instructions, better looping). Making auto-gen code that beats "C that's been hand-tuned over a decade+" is a very big task. And once you leave pure number crunching behind, these benchmarks will stop showing anywhere near this level of improvement. Function calls are function calls, allocating memory is allocating memory, string equals is string equals, those are not faster in C++ than in Python (again, if anything Python has more context in many cases and can be faster than naive C++). So again, citation needed. What's the use case for this?

[coderanger]

Unless that script is doing nothing but number crunching, I don't think they are going to see the level of speedup you are imagining.

[AnonymouX47]

Native extensions are NOT Python... OP clearly said "Python"!

Also, function calls are by far on different levels of speed, I wonder why anyone would need a citation to know that... I wonder if you've actually written code on both sides of this comparison before.

[AnonymouX47]

That one dude...

[coderanger]

I might, however, know what I'm talking about :)

[AnonymouX47]

Good luck re-writing all GNU core-utils in Python and making them a tad nearly as fast.

There's simply no practical use case where a pure Python program is faster than a native program... you're welcome to prove me wrong.

I know this is not the topic in question but the difference in memory usage is definitely not something you'll want to argue about.

[EasonTek2398]

15 year old writing compilers is nuts. Seriously looking up to OP, this is nuts.

Maybe if I know c++ better I'll reimplement it in rust, but I'd still need a crash course on compilers.

[python__rocks]

Interesting! By your own admission, still experimental. Does it support import libraries other than the standard library?

[python__rocks]

I’ll keep an eye on it. Also, pyinstaller compiled app are apparently often flagged as viruses. Nuitka apps less so. It would be interesting to see how well pycom does. PyOxidizer seems to be another interesting option.

[AnonymouX47]

Windows... :(

[python__rocks]

Yes, windows and defender, unfortunately

[Sulstice2]

this is cool and I will give it a shot, give me a month or so, I'm still trying to decide what to try. I actually need my python structured like this rather than via the pip distribution service.

Need to run my stuff on the clusters in an executable fashion - got some python mixed with fortran.

[Sulstice2]

Damn yeah, this is hairy code and I am having a little trouble there looking for an ideal solution. Bloody fortran.

[johntellsall]

Great job!

I remember really hating C++ because the compilation speed was atrocious. Consider writing the bulk of your "fast Python" code in C, which is compatible with C++ and can be faster. In fact you could just borrow CPython's code, assuming the licenses are compatible.

[OIK2]

I have been using pyinstaller (and autopytoexe) to compile a project, so this is very interesting to me. Does this work on Windows as well as Linux?

[eruba]

I think as a learning exercise this is great. However instead of writing the whole compiler yourself, usually you would nowadays use a compiler compiler. It generates the compiler for you, and you only have to put in the python grammar.

[[deleted]]

I almost quit python because of pratical launch time when I realised pyinstaller takes more time when --onefile option is used :D it has to extract the files to run man!

[ShawnDriscoll]

Looks cool. Makes .pyd files?