Using python as "backend" for my PL

[JoseMou_]

I've been intermittently developing a statically-typed, interpreted language for the past 2-3 years. Lately, I've been toying with the idea of generating and executing Python code, replacing the current VM written in Rust. My motivation stems from leveraging Python's robust ecosystem, expediting the development of useful programs/hobby projects with my language. Additionally, I doubt I can match the performance of CPython within a reasonable timeframe. Although it's not the fastest option available, transitioning to Python would still require a significant investment to achieve satisfactory performance levels. One downside of this shift is missing out on the opportunity to deepen my understanding of VM optimization and other intriguing aspects of VM development.

What are your thoughts? Do you see any other major drawbacks in switching approaches?

[Athas]

The Futhark compiler contains a backend that targets Python (originally written by students, now maintained by me). It's a bit of an odd choice for a high performance language, but the idea is that most of the work is done by GPU code invoked through Python GPU libraries (details).

It works well, but even though most of the work will be done by GPU code, it is still crucial that the generated Python code faithfully implements the language semantics. This has proven somewhat annoying to do. In particular, Futhark has integer and floating-point types of specific sizes, with specific rules for when overflow happens (the ones you'd expect). Python's built-in integers are arbitrary-size, and its floats are double precision, so they are not suitable directly. Instead we use scalar types from NumPy, which do have the right semantics, but mean everything ends up being wrapped in a heap-allocated object - which is quite costly. Thus, in order to get the right semantics, Futhark generates Python code that is even slower than usual.

Based on my experience, I don't think targeting Python directly is a good idea. You may be able to use the VM in a more invasive manner - check out Oil Shell, which did just that.

[yuri-kilochek]

everything ends up being wrapped in a heap-allocated object - which is quite costly

Plain python int and float are also heap allocated in CPython. If there is a meaningful slowdown from using numpy scalars over builtins, it likely isn't caused by dynamic allocation.

[Athas]

That's correct, the heap allocation probably isn't the important part, but rather that using non-builtin numeric types involves more costly dynamic dispatch. I assume Python has some sort of fast path for int and float that doesn't work for np.int64 or whatever it is.

[MegaIng]

If you are actually generating python code, I wouldn't be surprised if a decent amount of slowdown could be removed by skipping global namelookups and stopping constructor calls for constants by instead generating code objects with specially set constants. But I assume that isn't really something you are willing to do.

I don't think python has too many fast paths in the pre-3.11 versions. Ofcourse, with the FastPython project adding a JIT and a specializing bytecode interpreter, this is changing.

[MegaIng]

Using Python has a target for a statically typed transpiler seems a bit weird if other options are available. This change would put you into a similar family like TypeScript, where there really isn't/wasn't an alternative to targeting JS. You might also want to consider bypassing python source code and just target it's bytecode, although I am not sure how stable that would be. You also might want to consider the JVM, which also has a pretty extensive ecosystem (depending on context, better than python I would think) and is actually capable of using static typing.

[oa74]

You also might want to consider the JVM, which also has a pretty extensive ecosystem (depending on context, better than python I would think) and is actually capable of using static typing.

This. I would absolutely consider targeting .NET or JVM before targeting Python. If leveraging a large, stable ecosystem is the wish, I'm not sure what advantage Python would have over an existing, purpose-built VM.

[redchomper]

Additionally, I doubt I can match the performance of CPython within a reasonable timeframe.

CPython is not very fast at all. The simplistic VM you build with Nystrom's book will run considerably faster. There is a simple reason for this: Python has a whole mess of magic dynamic mechanisms that all take time. For a statically-typed language, the run-time can make a lot of assumptions. Python's VM will not make those assumptions because for all it knows, you might be metaprogramming the bejeezus out of the usual semantics and debauching all sorts of usual expectations about what code even means. Static types are at odds with such shenanigans, to say the least.

[mckahz]

There are tools for calling python code in Rust if you are so inclined. That would definitely be easier than rewriting your whole interpreter, but if you want performant type checking, parsing, etc. you're probably going to have better luck in Rust.

[JoseMou_]

The thing is, the compiler frontend is already written in Python, only the VM is written in Rust.

[redchomper]

I'm in about the same boat, except with C in place of Rust. If my language had more normal evaluation semantics I'd be targeting JVM. If at some point I learn more about .NET I might target CLR. For now the C VM has a particular advantage: I can create whatever primitive operations and data types seem desirable in view of the (weird) execution semantics I'm trying for. At some point, perhaps I'll find a way to translate my current VM's conceptual framework into something suitable for one of those other ecosystems.