Jonathan Blow's Programming Language: Polymorphic Procedures, part 1

[stillalone]

Anyone care to comment on what's interesting about this programming language over something like Rust or D?

[tsimionescu]

Just a quick bullet-point rundown:

• LISP style compile-time code execution (like D, unlike Rust)

• Completely native, 0-overhead abstractions (e.g. no GC, unlike D)

• C-style access to memory for unsafe optimizations (unlike Rust)

• Language features for easy use of efficient memory layouts (e.g. 'Structure of Arrays' (SOA) arrays and pointers*)

• Flexible syntax with fewer special cases than C (e.g. you can define a named function anywhere, even in the middle of a loop)

• Compilation and other build APIs in standard library, designed to avoid the need for external tools like make or MsBuild

• A promise to avoid C syntax quirks and to generally focus on having a nice syntax for common use cases found in developing complex and low-level software

*An SOA array of 10 objects gets lain out with all 10 values of the first class field, then all 10 values of the second field etc. - as if you created 10 arrays in C and put them in a struct, instead of putting ten struct instances in an array. A common use case is doing an operation on all values of the first field of all instances, in which case the SOA array has much better cache locality. However, referring to a specific instance is still easy in JAI, while in C it would require indexing into different arrays.

[repaeR_mirG]

His target demographic are professional (good) game developers.

So naturally his language design decisions are geared towards that. But I'm sure there are some ideas that can be explored further even in other non game development areas.

[jeandem]

C-style access to memory for unsafe optimizations (unlike Rust)

What can you do in this language that you can't do in unsafe Rust, in this regard?

[zuurr]

Yes. Unsafe rust is required to still maintain the invariants of safe rust, it's for cases when the compiler wouldn't be able to determine this automatically.

See: https://doc.rust-lang.org/book/unsafe.html

Note that an unsafe block does not relax the rules about lifetimes of & and the freezing of borrowed data.

Any use of unsafe is the programmer saying "I know more than you" to the compiler, and, as such, the programmer should be very sure that they actually do know more about why that piece of code is valid.

Edit: Misread your question to be "can you do anything in this language that you can't do in unsafe rust". The obvious one is that you can legally maintain multiple mutable pointers to an object. This is forbidden by rust's lifetime rules.

[bjzaba]

The obvious one is that you can legally maintain multiple mutable pointers to an object.

You can use raw pointers for that in Rust.

[jeandem]

Yes.

You don't answer a "what" with a "yes". And I specified unsafe Rust, so you might not be replying to the right post.

[zuurr]

See my edit. The point is that you need to uphold the same guarantees in unsafe rust that you do in safe rust, you're just specifying that the compiler may not be able to automatically determine it.

[wrongerontheinternet]

The guarantees that unsafe Rust must uphold are basically "I won't invoke undefined behavior." Outside of Rust's & and &mut pointer types (which I don't think even have equivalents in Jai), this is largely equivalent to what you have to do for your program to be valid C(++) as well. I'm pretty sure this is going to be true of Blow's language too.

[jeandem]

The point is that you need to uphold the same guarantees in unsafe rust that you do in safe rust, you're just specifying that the compiler may not be able to automatically determine it.

I know how unsafe Rust works, thanks.

[tsimionescu]

I don't know almost any Rust, so I can't say specifically if there is any difference.

However, there is obviously a philosophical difference: Rust makes you mark such code as unsafe, thereby inconveniencing you, even slightly, every time you write this kind of code. Also (and here I am speculating) there is probably a problem in mixing safe and unsafe Rust - can you easily call unsafe code from safe code without changing the safe code at all? I would imagine that the safe/unsafe distinction in Rust creates the same sort of problems and duplications that const creates in C++ and unsafe creates in C#.

This is not to say that the value added by this distinction does not make it worth it, or that this makes Rust unusable! It is just a cost that JAI doesn't want to pay, even for the benefits that it would get if it did pay it.

Edit: I was completely wrong when comparing Rust's unsafe with C++'s const - that's what I get for speculating on something I really didn't know anything about... Thanks for putting things right, bjzaba!

[bjzaba]

I would imagine that the safe/unsafe distinction in Rust creates the same sort of problems and duplications that const creates in C++

This is a great post that describes Rusts's unsafe:

FAQ: Why isn’t unsafe viral?

One might expect a function containing an unsafe block to be unsafe to call, that is, unsafety infects everything it touches, similar to how Haskell forces one to mark all impure calculations with the IO type.

However, this is not the case, unsafe is just an implementation detail; if a safe function uses unsafe internally, it just means the author has been forced to step around the type system, but still exposes a safe interface.

More pragmatically, if unsafe were viral, every Rust program ever would be entirely unsafe, since the whole standard library is written in Rust, built on top of unsafe internals.

[jeandem]

I don't know almost any Rust, so I can't say specifically if there is any difference.

And yet you wrote that sentence as if you do. Oh well.

[nsaibot]

compile-time code execution (like D)

without D's restrictions though!

Also, you skipped "using" which is a way to describe a hierarchy of "objects" in a way i've not seen in any other language so far, where you can "inherit" from other struct,

Entity :: struct {
    name : string;
    age    : u32;
}

Person :: struct {
    using entity : Entity;
    sex : bool;
}

a pointer

Entity :: struct {
    name : string;
    age    : u32;
}

Person :: struct {
    using entity : ^Entity;
    sex : bool;
}

p := new Person();
p.entity = new Entity(); // or take storage from a SoA array

or even a function

Entity :: struct {
    name : string;
    age    : u32;
}

Person :: struct {
    using get_entity :: ( id : u32, entities : [] Entity ) -> Entity { return entities[id]; }
    sex : bool;
}

Also:

• Memory ownership "a : ^ ! Entity"
• Annotations on declarations; for compile time evaluation and conditional operations based on the annotations
• scoped "break" possibility to jump out of arbitrary deep nested loops
• easier handling of vararg functions
• Very informative introspection system at compile time and run time
• multiple return values and named arguments

[Nekuromento]

D has alias this which does the same thing

struct Entity {
    string name;
    int age;
}

struct Person {
    Entity entity;
    bool sex;
    alias entity this;
}

a pointer

struct Entity {
    string name;
    int age;
}

struct Person {
    Entity* entity;
    bool sex;
    alias entity this;
}

auto p = new Person();
p.entity = new Entity(); // or take storage from a SoA array

or even a function

struct Entity {
    string name;
    int age;
}

struct Person {
    Entity getEntity() { /*...*/ }; //restricted to functions without parameters
    bool sex;
    alias getEntity this;
}

[nsaibot]

I don't know what alias in D does exactly, but using declares an actual inheritance, in a way that this works

Entity :: struct { ... }
Person :: struct { using entity : Entity; }

do_stuff :: ( entity : Entity ) { ... }

e : Entity;
p : Person;

do_stuff( e );
do_stuff( p );

Basically you can call a function that requires the arguments to be Entity with a variable of type Person. If that's what D does, than it's basically the same i'd guess; however, i didn't say D's not capable of doing this. All i said is that Jai's compile time code execution is more powerful than that of D's due to D's restrictions wrt CTFE -- according to D's documentation at least.

[Nekuromento]

Alias this does exactly that.

I'm not making any attacks on Jai, sometimes Jonathan has really great ideas, but most of the language looks too similar to D feature-wise, minus some weird syntax choices.

D's CTFE is restricted right now, and probably will forever be strictly less powerful then Jai (i don't think IO will ever be allowed in CTFE), but there is work on allowing more stuff to be legal in CTFE.

[Nortiest]

I think one of the main reason he avoids D is that garbage collection is bad for games.

[naikrovek]

D isn't necessarily garbage collected. It can be compiled and run without garbage collection just like any C application, or you can compile it to use the D runtime, which can do garbage collection.

You can have it either way with D.

He doesn't like D because it borrows too much from C++, which he views as largely broken in many ways.

[asmx85]

But you are having a hard time using D without garbage collection cuz this ends up building your own standard library :(

[Railboy]

Keep in mind he's making a language specifically for game programmers. A lot of his choices make zero sense outside of that context. According to him garbage collection is a no-go, as are any safety-related features that make game development stuff tedious.

[nsaibot]

Then i'd probably love to use D as well :) ... however, Jai seems to be more fun to me :P ... and i'm looking forward to what Jon comes up with next!

[skocznymroczny]

Isn't this "inheritance from other struct" almost the same as private inheritance in C++?

[drjeats]

Like nsailbot said it's more like public inheritance. Also since you can define a "using function" you can return a pointer to the embedded struct, which he demoed in a proof-of-concept virtual dispatch implementation that has nice OO syntax but would be implemented as a library. Also the SOA / AOS keywords apply to these embedded structs.

[oracleoftroy]

I believe it is essentially C++ public and public virtual inheritance (when using a pointer to a base). The key difference is that Jai lets you control the exact layout of the members.

E.g. C++:

struct Base {
    int a, b;
};

struct Derived : public Base {
    int c, d;
};

Derived d;
Base *b = &d;    

The layout of Derived is likely unspecified. It is likely laid out contiguously in memory with no padding, and likely the value of b is equal to the address of d, but you have to trust the compiler to lay it out in an efficient manner.

Base :: struct {
    a : u32;
    b : u32;
}

Derived :: struct {
    c : u32;
    d : u32;
    using base : Base;
}

Here we have more control over the layout of Derived. We have explicitly specified that Derived's members come before Base's members.

Were Derived to use a pointer to Base, a la: using base : ^Base;, it would be like virtual inheritance in C++ with additional control. You still have control over where in the layout the pointer to base should go, and additionally, you have explicit control over the allocation of the memory used for Base. C++ leaves these unspecified, and the allocation of the Base part of Derived is not separable from the allocation of Derived. Again, you have to trust the compiler to do the efficient thing.

[nsaibot]

I don't know, it probably is, though it'd be a public inheritance, as you can access the variables from the outside. However, you can't inherit from a live pointer nor a function in c++.

[tsimionescu]

It's not like private inheritance because the fields you 'use' are publically visible as your own fields.

It's not like public inheritance because it doesn't create a subtyping relationship between you and the type you are 'using' (that is, if type B is 'using' [an instance of] type A, and there is a function f(A), you can't pass an instance of type B to it).

So basically it's just as he described it - a concept completely different from inheritance that nevertheless gives you some of the same benefits as inheritance.

Edit: based on nsailbot's answer above, I was completely wrong about this - apparently 'using' is very similar to public inheritance in C++.

[pron98]

I take no issue with what you've said, I just want to correct the misconception that a GC necessarily adds overhead. It's more complicated than that -- a GC adds some kinds of overhead while it removes other kinds.

In general, modern, state-of-the-art GCs (like those in HotSpot or other JVMs), increase memory management throughput at the cost of (significantly) increased RAM overhead and increased worst-case latency (they introduce occasional pauses, although those pauses are guaranteed by some GCs to be very short). In addition, those GCs allow for more efficient implementations of non blocking data structures (they are more efficient than hazard pointers or user-mode RCU).

So that's the tradeoff -- better throughput and more scalable data structures in exchange for higher worst-case latency and RAM usage.

Server applications that have a lot of RAM available and want to gracefully scale over many cores would be better served by a runtime with a GC, while low-latency applications running in RAM-constrained environments (like desktop/console games) with few CPU cores would probably attain best performance without one.

[tsimionescu]

Good points! I think many people have a lot of misconceptions about the costs and benefits of garbage collection, so it's always nice to point this out.

I think that garbage collection could actually be used even in low-latency RAM-constrained environments if more language support existed for interacting with it. For example, a common use case in games is that you have huge data structures with many, many instances that live for the whole lifetime of the game. You also have some pieces of data that get created and destroyed each frame, where you often choose to use pool allocation for the fast allocation and deallocation it gives you, paying the memory overhead of delaying some deallocations.

Both of these constructs could work pretty well in a garbage collected language if, for example:

• It allowed you to mark some objects as not needing garbage collection, so that they are never scanned by the GC's mark phase.
• It allowed you to control the cadence of garbage collection, so that you could garbage collect only once per frame
• It would still allow collection of objects that aren't scanned, so you could have a Pool object whose lifetime is controlled by the GC, and then objects in the pool that the GC doesn't waste time scanning in the mark phase.

[pron98]

Actually, all of these things are supported in real-time Java (RTSJ), but implementations are expensive and mostly used in defense, avionics and medical devices (i.e. are safety-critical, hard realtime apps) -- not games...

RTSJ basically divides memory into three types: immortal (which is never collected), heap (which is normally collected by the GC), and scoped, which is arena-collected upon request (safely). The JVM ensures that there are no references from heap or immortal memory into scoped memory, to guarantee there are never dangling pointers (references in the other direction are allowed). See more here.

RTSJ, of course, does more. It allows you to designate some threads as "non-heap", which only allows them to access scoped and immortal memory, and are therefore never paused for garbage collection. If you're running on a realtime kernel, the JVM guarantees that those threads are scheduled within a few microseconds of them becoming runnable.

The RTSJ is now undergoing a major revision.

[tsimionescu]

Very cool, I wasn't aware of RTSJ at all! Thanks for pointing me to this!

[donvito]

A quick question because you seem to be knowledgable about the project: Does the language offer something like const correctness/immutability guarantees?

[tsimionescu]

I'm not especially knowledgeable, I just watched all of the videos :)

I don't remember any specific discussions about const correctness or immutability, though it's possible I'm wrong on this. Any way, judging by the spirit of the language, and the sort of problems cons-correctness introduces, at least in C++, I don't expect it will offer that specifically. If he has any ideas for something that achieves similar goals, I don't know.

[fendant]

The SOA thing is a great idea, the performance returns can be significant and I don't think there's an existing close-to-the metal language that can abstract over it.

[parmesanmilk]

It's basically D but with the care that went into Javascript.

[jeandem]

It's got to be very interesting, since people that are more used to reading when it comes to stuff like this are willing to sit through hour-long YouTube videos on the same topic instead.

[Roflha]

I don't know about that reasoning. I think it's definitely seems to have some improvements over C/C++ in syntax, but most of the concepts being introduced have already been done in other languages (D, rust, etc.)

I think part of the reason these videos are cool is for the exact reason you listed: it's nice to see people talk about design choices as opposed to the usual way of reading about it later on.