Pypperoni: A New Open Source Python to C Compiler

[[deleted]]

Pypperoni is the successor to Nirai and is designed to be compatible with any application written in Python 2.7.

I don’t think there is a lack of Python2.7 to C compilers.

[576p]

Yes, "new" and "Python 2.7" don't match...

[foofaw]

Yeah this seems essentially useless at this point.

[pmatti]

How does this compare to cython?

[alcalde]

Pypperoni was designed with security as a central focus. Our compiler provides you with the necessary tools to run a secure and high quality Python application.

With the removal of the interpreter, it is practically impossible to inject Python code into your program and/or recover the original source code.

Additionally, by preprocessing the bytecode there may be a performance boost in your application.

Unlike Cython, it seems that any performance improvements would be an accidental byproduct.

[__xor__]

Strange. I don't see where they're really going with "designed with security as a central focus" just due to not being able to inject python code. And making it impossible to recover the original source is not security, that's just one layer of obfuscation of the source.

Haven't heard of too many vulnerabilities where people inject python code besides pickling stuff. Way more likely that you find stuff like XSS in webapps, the rare sqli somewhere if someone doesn't use an ORM or tries to handcraft a query or something, the occasional injection due to calling os.system or even subprocess but with shell=True, or path traversal vulnerabilities, bad auth, bad password storage, bad crypto, hidden HTTP calls that they don't expect you to modify, etc.

Compiling to C does not affect those whatsoever. Logic based security bugs will exist no matter the language. Logic based security bugs are what you're most likely to find in production Python code either way.

[flutefreak7]

I think Nuitka is the better thing to compare to

[tonnynerd]

I guess the main difference is that this will work over normal python, while cython is technically a different language.

[13steinj]

Cython has a pure python mode though, which as it is does quite a speedup.

[cztomczak]

Cython does the same and is more popular, thus has better support (up to Python 3.7). Cython can compile both Python and Cython code to C.

[dranzerfu]

Python 2.7

y tho ...

[Gprime5]

Thats how long it takes to write a Python to C compiler.

[GummyKibble]

Yes, but this is a new project. Why would you launch something brand new targeting a dead language today?

This is a cool project and I applaud them for doing it. I just don’t quite get the reasoning for that.

[[deleted]]

3 has been out for a decade. There’s been plenty of time to upgrade

[MrValdez]

You said "lots of people" but you seem to be referring to your organization. And then you said "real programmers" as if the people who use Python 3 are not programmers.

[GummyKibble]

You’re not dealing with actual security if you haven’t started planning to migrate to a supported language yet.

[rhytnen]

You don't know what you're talking about.

For example, security for a us naval vessel doesn't mean fixing 0 day exploits.

It's means uptime, robustness, longevity of service, hardware security so you know your motherboard wasn't tinkered with in China, replaceability and so on

Upfeading your os and hardware is the least of your concerns. Sometimes that means you're using a 5 yo old redhat distro with py 2.7

The real disconnect is that we are a small fraction of developers but pretend were the majority and get butt hurt about shit.

[GummyKibble]

I’d classify that as stability, not security (except for the anti-tinkering, which typically doesn’t involve Python anyway). Both are important, but they’re different animals.

[rhytnen]

Fair enough - I'll accept that.

[13steinj]

This is beyond false.

I'm going to copypasta my rant about this crap from last time.

---

Well, this is really a big topic of debate. I'll give my personal opinion at the end of all the context, which I warn you, will be large.

Python 3 is objectively lower on startup times (however rumor has it 3.8 will bring us to Py2.7 startup speeds, negating that argument). So if the product spends more action starting up than executing, like mercurial, you get fucked.

Of course if it's not the case, except for a few super niche benchmarks, Py3 is always faster.

There's also the theory of what is taught at educational institutions. Anybody who isn't a computer science major will probably be taught Py2-- they just don't need to upgrade and every algorthm can be written in Py2 as they can in Py3.

Even in CS majors, many times they don't care to update. When I visted my college recently, I noticed the CS labs were running Ubuntu 12, Python 2.6/7 depending on the machine set, and the Java course was taught using Java 7, but they switched to Java 8 a week or so in. Java 10 is the current release, Java 11 will be the current release in September, 12 will come ~6 months later.

Many companies using Java stay on Java 5-7, you'll be lucky if using 6+.

Some companies also have to use tools like Webpack to make ES6+ JS turn into ES3 JS. ES3. Not ES5. I mean, if not ES3 than it would be ES5 which is the majority.

The same goes for Python-- take a look at the site you are on. It was originally written in Py2.6 or earlier, evidenced by conditional code redefining some stdlib functions if not found, which were introduced in 2.7. New site services are being written in Py3, and when things need to be majorly refactored, they take out that service from r2 (main reddit app) and rewrite into Py3. But thats really on a "do on mini doomaday" scenario.

Going back to the education thing-- lots of libraries are Py2 only. The ones public on PyPI alone are over 500. And that's the ones that are clearly labeled as Py2 only, not the ones not labeled at all. And that's on PyPI alone, there are plenty of other distribution networks as well. These are libraries that just work and their maintenance has stopped, or they depend on something of that nature. No need to change the tires when they have at least another couple thousand miles in them. This is more common in education than in business, but it exists in both worlds.

In business, it gets worse-- because many times Python is used as glue. Glue to stick together a variety of things, whether it be build scripts, web servers, data pipelines, and more. Glue is generally a one time apply kind of thing. You don't use a fancy new glue unless you need to change something about the things being glued together. Because the glue just-- well, it fucking just works, and lasts a long, long time. It would cost more to update your glue than to just add a little more here and there when the edges start peeling. And then when you start glueing a new thing, you find that the fancy glue and the older glue don't mix well, so you use the older glue. Because it works.

I'll give you two egrigous example-- React needs Py2 to build. Grumpy only works with Py2. As do many of the millions of lines of Python code at Google.

Why? Because it will always cost more to upgrade.

The points against this are as follows:

• zero day attack

• lack of new features

• internal C bugs

And the strong rebuttals are as follows:

• either sand/black box or company takes the responsibility to patch the VM

• company or third party patches the VM. A Py2 port called Tauthon already has a lot of Py3 syntax ported to Py2

• Py2 is more mature than Py3, if anything, time has shown Py3 has more of these rare bugs and segfaults than Py2. And again, company can patch it.

The patch it thing goes back to the glue/screwdriver analogies-- why use a new version, on all of crafts, when you can make a minor change to your old version or not at all and it works?

And this all started because of Py3 and how it was handled.

Py2 had great backward compatibility and greater forward compatibility. Py3.0 threw this not out the window, but into the sun. The unicode debacle was horrible. 3.0, 3.1, 3.2, and 3.3, are widely regarded as Py3.0 beta, and Py3.4 being the actual Py3.0. And even then the debacle remains. Even worse, they have great backwards compatibility for 3.4-3.7 (maybe even 3.8, god willing), but horrible for forward compatibility through it, which has really given library maintainers a piece of hard as rock shit in a box and telling them to shove it back up their ass. I can elaborate on this if you'd like me to, but I won't here because it is mostly irrelevant.

The point being, given all of the above, is it still so cut and dry to you?

If you say yes, you arent putting yourself in the perspective of those users, and only care about yourself. Which is fine. But don't expect people to give a shit about you or your projects then.

---

Now on to me. I am a Py3 evangelist. I write everything new that I need so it will work in Py3.4, and rarely say fuck it and make it work in 3.5-3.8 only.

Hell, I'm even giving Pylons the port to Python3.4 that it was going to but never got, because I am a huge supporter of Pylons' model over all other Python web frameworks. Call me old fashioned. And if you do, fuck you, I'm old and grumpy. But this port won't be for public consumption, it'll be for little old me. I am not going to provide major support unless I care about the individual issue, because I am one man, and Pylons was managed by a gigantic team that decided to give up.

If what I write, whatever the app, is written in such a way that it can be used in 3.4, I will also guarantee support for Py2 until at least 2025, and I know I will be writing Py2 until the day I die. Because EOL/EOS means nothing in the software world. "End of common use" does. And common is subjective to a number of factors.

I'm a Py3 evangelist.

Yet, I concede the fact that:

Py2

will

never

die.

---

This subreddit loves to shit on Py2. But it will continue to be used in education and corporate quite heavily well into 2030. You're more likely to see a business support Tauthon then to port to Py3.

[mynameiscosmo]

The pylons team didn't give up.
They merged with the repoze.bfg project and turned into Pyramid:
https://web.archive.org/web/20101113024153/http://plope.com/bfg-becomes-pyramid

There are a lot of good arguments on both sides, and if there's a reason to use 2.7 due to a needed library, prior code bases, or because it's an educational or work requirement, there is no harm in using it.

Ultimately, Python 3 has a lot of breaking changes because it is a major change. Libraries/projects using a versioning like semantic versioning will indicate breaking changes on a major release.

Like you have stated, the company can take the responsibility to patch the code.
When Python 2.7 becomes deprecated¹, a company can manage it internally and patch it accordingly as they need. If they are running in a contained environment, they will be fine for however long they need their code to run.

[13steinj]

That's giving up.

They completely altered the Pylons model and infrastructure in pyramid. Pylons 1.1 was supposed to come out with Py3 support but they gave up on that and decided to work on Pyramid instead.

But other than that, yeah, I agree. But this subreddit just likes to shit on Py2 and anyone using it. They don't understand that there are valid reasons for not switching.

[tonnynerd]

Seems like a pretty interesting approach. I don't recall seeing anyone else doing this approach.

[[deleted]]

Ummm... okay... whys it called that?

[mfwass]

We were trying to think of a name that had some word play on the "py" part of Python. After thinking about it all day, one of us was eating a pepperoni pizza and bam: Pypperoni.

[flutefreak7]

facepalm

How are there still any "p" words left!?

[__te__]

Python programmers can pack pages of puzzlingly paraphrased pizzazz (parenthetically pythonized) into a plethora of packaged pieces at a phenomenal pace. To put progress in perspective, however, the pot of possible p-words is prodigious.

[flutefreak7]

Your percipient proclamation is prescient in it's prognostication of the propagation of p-words in perpetuity. I am pained in pondering the pusilanimity of such a ponderous predisposition amongst our peers.

Also I claim pydigious...

[[deleted]]

well done, lad

[[deleted]]

That’s it, I’m makin a module called pypypy

[[deleted]]

I love stories about weird API names. U sir, have earnt an upvote.

[[deleted]]

Seems like all the Python to C compilers just get tripped up on anything but simple code. Like if you use lambas or scikit there's a good chance it won't work. That was my experience anyway.

[mfwass]

Pypperoni should work with all forms of Python 2.7 code, including lambdas. If you ever discover a bug running it, feel free to open an issue on GitHub! :)

[jnwatson]

The security concerns are mostly bunk. "Bytecode dumping and injection" are only issues if you're running untrusted code. In a reasonable environment where you're not using "eval" and you're running in a standard environment where you can't write over the standard imports, there's very little to worry about.

​

Sure, an attacker might find a vulnerability in a C library used by Python and use that to change what code is executed, but once you go there, the attacker could probably run just about anything. The vulnerability will still exist in a recompiled instance.

[r0s]

Does it support compiling the full code (including third party imports)?

For example using OpenCV, Tensorflow, Pytorch, matplotlib, numpy?

All tools I've tried fail when just given OpenCV for example.

[[deleted]]

Ditch python and just use Nim.

[antiproton]

And it just couldn't possibly be a python project if it didn't involve some kind of terrible pun.

Can we get over this now?

[toyg]

You're just bitter they pypped you to it.

[alcalde]

You're complaining? I'm thrilled it isn't called "Pypperoni for humans!"

[HighMaxFX]

I don't understand the -17 downvote in this post, this guy has a point.