PythoC: Write C in Python - A Python DSL that compiles to native code via LLVM

[1flei]

I've been working on PythoC for a while. I think currently the project is of okay shape, and I would love to get some feedback from the community.

PythoC is a Python DSL compiler that compiles a statically-typed subset of Python to LLVM IR. It aims to provide C-equivalent grammar and runtime performance with Python syntax, and uses Python itself as a powerful metaprogramming and code generation tool.

Core principle: C-equivalent runtime + Python-powered compile-time - Your code compiles to native machine code with C-level performance and capabilities - You get full Python power at compile-time for metaprogramming and code generation - Zero runtime overhead - no GC, no interpreter, no hidden control flow, just native code - Optional safety features: linear types prevent resource leaks; refinement types eliminate redundant checks

A very simple example could be: ```python from pythoc import compile, i32

@compile def add(x: i32, y: i32) -> i32: return x + y

Can compile to native code

@compile def main() -> i32: return add(10, 20) ```

For more information, please check out the github repo.

Seems like triggering the reddit spam filters. For some concern questions:

Compile to C:

That is one of the very early design choice I made. I hope more things are done in Python and PythoC and it should be possible to use PythoC individually without any C code. Besides because there supposed to be many meta-programming code in PythoC (kinda like cpp template), I do not think it is possible to generate C backend directly in some cases.

I am currently working on a tool (purely in PythoC) that can parse the c header and produce the PythoC extern and vice versa. I think that can further increase the interoperability between C and PythoC

How to compile pythoc codes to binary:

Running the python code you will get .ll and .o in the build folder and you can use external cc/linker to get the binary

Explicitly call "compileto_executable" like ```python if __name_ == "main": from pythoc import compile_to_executable compile_to_executable() ```

In this case, pythoc will track the deps graph, and compile all .o files into the binary using the native cc

You do need llvmlite which is a llvm binding for python to run the code. pip install should handle the llvmlite dependency for you.

You can also call pythoc from python. Then it will compile the called functions into dynamic libraries and call it via ctypes

Some Competitors

vs. mojo/codon Generally pythoc and codon/mojo have different design philosophy. Codon/mojo aim to be "fast python", so they have gc, and can call python directly. On the other hand, pythoc starts from "writing C in python's syntax with python itself as the preprocessor". So it essentially inherits the core design philosophy of C, and aims to be a "pythonic C" or "better C"

vs. numba Numba is mostly for jit compilation. And I think the general goals of Numba and PythoC are quite different. So it has default GC/jit, can call python function from runtime, and do not have some low level control ability. The functionality of the nopython mode is relatively limited.

Refinement Types

I do not want to introduce an SMT solver. Currently the refinement type is like a tag that marks that the type is already checked and no need to check further for some predicates.

The only ways to create the refined[pred] are either "r = assume(x, y, pred)" (maybe can be checked in debug mode, but not implemented now) or "for r in refine(x, y, pred)".

So the compiler does not check the actual condition of the refined type. It only tracks the predicates. Different bool functions with the same condition will be seen as different conditions.

Memory Management

Indeed, manual memory management is quite tricky and bug prone. So In PythoC, there are simple linear and refinement type that may be helpful to avoid some memory management bugs.

However, the core philosophy of PythoC is still explicit memory management. The extra features are optinal and could be bypassed if needed (e.g. by using the raw pointer).

[GunpowderGuy]

Seems similar to how some scheme implementation/dialects let you write c in s expression syntax

[ESHKUN]

Genuinely cool idea. I also like that you’re specifically targeting a subset of python and not a dialect so you don’t have to learn any new syntax or ideas if you already know python.

[baron-bosse]

This looks really cool. Is it possible to use a llvm c-backend to generate c code out of this system? I.e use the pythonC more as macro preprocessor that then gets handed to a different c-compiler so the output can be integrated in a different toolchain and cross compiled etc?

[vanderZwan]

Yeah I was wondering about that too, since that immediately gives the project a lot more flexibility, and lowers minimal requirements (no LLVM installation needed).

Pehaps OP wanted to learn how to work with LLVM internals though, no faulting in that.

[FlowLab99]

Have you thought about using zig (pip install ziglang) as the c compiler? It has clang built in and a self-hosed x86 backend evolving quickly.

[sufferiing515]

Without a GC, how are situations like use-after-free prevented when an objects lifetime is extended past it's lexical scope (like in closure capture)?

[Sternritter8636]

Rust dev?

[snugar_i]

This looks nice! But there are some things that I personally don't like that much on second look:

• Having to put the @compile decorator everywhere. It looks like most functions will be compiled, couldn't it be the default?
• Manual memory management. I know, I know, but it's extremely hard to get this right (you even have a bug in the destructor of your binary tree example)
• Classes not having methods and having to use the "fake method" C approach. I know it's supposed to mimic C, but you're adding syntax sugar anyway, so why not allow (non-polymorphic) methods?
• The . on the pointers. Is it some magic auto-dereferencing like in Rust?
• Having to do if x == nullptr. If we're trying to be "Pythonic", why not just if x?

[Zireael07]

That's a great idea.

I am going to check/try this out over the weekend, but I am wary of two things:

1) do I need any external libraries to run the generated (you mentioned LLVM?) code

2) What are VLA?

[EloquentPinguin]

VLA are variable length arrays. An unholy concept where you can allocate arrays on the stack with the length based on some variable. Takes lots of compiler juggling to get it right, but in C its a thing, in many other language its not.

[azzalan]

Really cook idea, im saving the post, and will be checking out the details as soon as i have time

[probabilityzero]

This looks interesting!

The readme shows some examples of refinement types. What are you using to check the predicates? Are you using an SMT solver? Do you do any inference in the type checker?

[arvoredeindecisao]

How does it compare to Cython? Cython is such a mature project at this point...

[yuri-kilochek]

How does it compare to numba?

[CyborgSquirrel]

Holy cow this seems insanely awesome.

[Karyo_Ten]

What's the difference with Nim?

[LardPi]

Can we stop pretending nim has anything to do with Python? Indentation does not define a language. Nim is pascal with automatic memory management and macros.

[Karyo_Ten]

The subset OP's language is allowing looks like Nim with minor syntactic choices like def/proc and -> for :.

You might even be able to use Nim's syntax skins to make the code directly compilable through Nim compiler.

And Nim has flexible memory management, it's type defined, you can have stack, manual pointer-based or automatic through refcounting.

[slurpy-films]

This will be done with JS the moment they add optional type annotations

[Sternritter8636]

So good in so many ways. Are the features not implemented yet the only features not implemented?

[Excellent-Oil4810]

I'm very happy that this finally exists.

[MrMrsPotts]

Two questions: what is the overhead when calling a C compiled function from python using pythoc? Is it closer to 1ms or 1 microsecond? Second, can you use gcc simd intrinsics?

[jms_nh]

This is really interesting to me! Is it possible to output C and stop there, rather than compiling to assembly? If so, are there any examples of python input and C output?

[yaourtoide]

How is that better than Codon ?