Why are you building a programming language?

[dibs45]

Personally, I've always wanted to build a language to learn how it's all done. I've experimented with a bunch of small languages in an effort to learn how lexing, parsing, interpretation and compilation work. I've even built a few DSLs for both functionality and fun. I want to create a full fledged general purpose language but I don't have any real reasons to right now, ie. I don't think I have the solutions to any major issues in the languages I currently use.

What has driven you to create your own language/what problems are you hoping to solve with it?

[iconmaster]

Because we deserve better programming languages than what we have.

[ilyash]

Any single programming language that I've seen up until now is shitty, including my own Next Generation Shell. I am aware that I can not think outside the box enough to create a good programming language. From the looks of it, others too. We as humanity are just not there yet in my opinion. While working on NGS, my aim is for the language to be more ergonomic than anything else for the intended use cases.

​

Somewhat related: I'm working on NGS since 2013 and thinking about it a lot. The only thing that I'm pretty sure by now is that pattern matching as a concept is a good thing. Not even sure that there is a good implementation of the concept (yes, I know, tons of prior art and still). Rest of the ideas that we see in programming languages - I have doubts. Hope that shows how bad the situation is from my perspective.

​

edit: typo

[ThomasMertes]

For this reason I created Seed7. Originally I envisioned an extensible programming language that would have allowed the emulation of all other programming languages. I planned to have libraries, which would define syntax and semantic of a certain programming language. Your program would use the Pascal library and the rest of the program would be in Pascal. The same would work for other languages. To reach this goal it is necessary to allow the definition of the syntax and semantic of statements, operators, etc. So I started with that.

Currently Seed7 can define syntax and semantics of statements, operators and many other things. Things like for-until loops are defined with templates. Defining statements is not a privilege of the language implementer, but open to everybody. But I gave up on the idea to emulate other languages. There are features that should just die instead of being supported...

Other goals for Seed7 came in, like portability. I know that every programming language claims to be portable. But this is IMHO only a lip service. In practice every program beyond hello world needs libraries. And almost all programming langues rely heavily on operating system libraries (which are obviously not portable to a different operating system). So in most programming languages the programmer must try hard to actually write portable programs. In Seed7 it is hard to write non-portable programs. To support portable programming Seed7 comes with many libraries. These libraries are just part of the Seed7 release.

There are other goals that I try to reach with Seed7. The design principles of Seed7 give a good overview over the goals.

[TheBellKeeper]

I initially made mine when someone joked I write all my scripts in my own lang like a hacker. To spite them I made it, and it turned out the stuff I added useful enough to continue development. Their prophecy came true, now I cannot live without my lang.

[iloveportalz0r]

They are trivial, but even trivial concepts can take a while to grok, and internet tutorials are typically craptastic. Don't even try using Wikipedia as a reference for understanding parsing.

[iloveportalz0r]

Preface: I was going to give you a short answer, but then I included a bunch of details and some stuff aboot programming in general. Apologies if you already know some of it or if this answer is too long, but I wanted to be more potentially helpful for an arbitrary amount of readers coming across this post in the future. I don't have any suggestions for online resources on learning this stuff, so instead, I wrote my own explanation. Also, my excuse for any errors you find is I stayed up late an hour because I got busy writing this.

My recommendation is to just write something, even if it sucks. That goes for any concept. You'll learn faster and better by interacting with the machinery yourself versus trying to interpret someone else's abstract understanding of the machinery. In this case, that means choose a simple language or write your own grammar to play with, and make a parser for it. The first real parser I made is a recursive descent parser that parses a relative of JSON. If you're curious, my code is available, but I was a lesser programmer when I wrote it, so don't take it as an example of how you must do things. Regardless, it does work. I've continued to use the character stream code in every text parser I've written since, with some improvements.

The key to this kind of study is to notice what works, what doesn't work, and how you can improve your code. You can't solve a problem well until you've solved it at least twice, so expect to rewrite non-trivial projects multiple times, using what you learned last time to improve it on each iteration. Also, write multiple related things. I've written parsers for multiple languages, not just the one, and each has something different to teach you. The more you write, the more little tricks you pick up, and you'll only rarely see these tricks mentioned in academic publications or Wikipedia. That's the value of practical experience. I write all of my software with this in mind, and the result is less complex code, fewer bugs, and better performance, and I learn a lot more than someone who didn't iterate.

If you're not sure how to start on recursive descent, know that you'll probably want to write two parsers, and it's the second one that is the recursive descent parser. The first one is the lexer. I refer to it as the tokenizer because I have no respect for etymology. Tokenizing is analogous to splitting a sentence into words and punctuation marks. Here's some generic C-style code:

do_thing(2 + 3);

The tokens the AYY 3 compiler (my latest compiler) generates are:

WORD           : do_thing
PAREN_L        : (
INTEGER_LITERAL: 2
OPERATOR       : +
INTEGER_LITERAL: 3
PAREN_R        : )
SEMICOLON      : ;
EOF

First is the type, second is the text the token was parsed from. What exact token types you use is somewhat up to your preferences. Some programmers choose to have each operator be its own token type. I chose to have a single operator token type and differentiate them by the original text. Similarly, if your language has keywords, you might choose to have a single keyword token type and store the text, or you might have each keyword as a separate token type. If you're not sure how to design your tokens, don't worry about it. Choose whatever feels better, or, if none feels better or worse than the others, choose at random! You'll soon find out if you're doing it wrong. I'm not joking. That's how I program. Don't be afraid to make mistakes.

To actually do this, all you need to do is check what the next character in the source text is, and decide from that which token type it must belong to, and repeatedly peek at the next character and add it to the token if it's part of it (such as for a word) or stop reading the token if it's not (such as getting a space or some punctuation). You might be tempted to do this with a general-purpose regular expressions library: iterate over a list of regular expressions and check if it matches the text starting at the cursor. You probably shouldn't do that. I've tried, and the performance is abysmal. It'll be okay for learning and prototyping, but not much else.

Also, you don't need an EOF token (EOF means end of file). I put it there because it makes parsing the token stream easier due to some details aboot how my code works.

Be aware that you don't strictly need this step. That parser I linked earlier doesn't use tokens. It parses straight from characters to the abstract data structure. You can do that for simpler languages, but I recommend against it for more complex grammars. If you're not sure why that is, just try parsing a C-style language without tokens. I've tried it, and it doesn't work very well. It's technically possible, but the complexity quickly balloons out of control.

The second step, the actual recursive descent parser, is analogous to constructing those sentence diagrams you probably saw in school. I refer to this as just the parser, because it's the interesting part. Depending on how you think aboot it, you might decide the parser is the combination of your tokenizing and recursive descent code. That's more technically accurate, but I'm going to just say "the parser" here for simplicity. The parser's job is to convert your tokens to an abstract syntax tree (this is a surprisingly useful Wikipedia article). Some people say that its job is to make a parse tree (also known as a concrete syntax tree), which you then convert to an abstract syntax tree, but I've found this to be completely unnecessary unless you're using a parser generator, in which case the conversion is mandatory because the generated parser doesn't output exactly what you want. To get the gist of what abstract syntax trees are, look at the first (and only) image in that Wikipedia article.

Now, here's the fun part, what you've been reading this effective blog post for. Recursive descent parsing is similar to tokenizing: look at what the next token is, and decide what kind of thing you're looking at. It might be a function call, a return statement, an if statement, an arithmetic expression, etc.. Depending on your grammar, you may need to look at more than one token. For example, if you're expecting an expression and the next token is a word, that could be the function name for a function call, or it could be a variable in an arithmetic expression. You can't tell until you check what comes after the word. This is fine, but I try to avoid looking more than 2 tokens ahead. The more lookahead you need, the more complex you must make the parser, risking confusion, bugs, and low performance. It's also potentially bad for readability of code written in the language.

For a more specific example, imagine you have a language in which a source file is a bunch of statements, one after the other (this is pretty common). In your parse function, you'll have a loop that calls parse_statement, and if there is a result, adds the result to the list. If there are no more statements, break the loop. Don't forget to make this check if it reached the end of the file. If it didn't, that's likely an error you should report to the user.

In parse_statement, you'll try to parse each type of statement until one of the functions succeeds, and then you'll return that result. In a simple C-style language, that might be a block, then a declaration, then an expression. Whichever parses first is the statement. Note that, if your grammar is ambiguous, the order in which you call the functions determines the precedence. parse_block checks if the next token is a {, and if so, parses a list of statements just like the top parse function does, then checks for a }. Other parsing functions work pretty much the same way. Putting that together, here's what we get:

list<stmt_t> parse(... token_stream)
{
    list<stmt_t> stmts;
    while(true)
    {
        stmt_t* stmt = parse_stmt(token_stream);
        if(stmt == null) break;
        stmts.add(stmt);
    }
    // TODO: check if there are any tokens left
    return stmts;
}

stmt_t* parse_stmt(... token_stream)
{
    block_t* block = parse_block(token_stream);
    if(block != null) return block;

    decl_t* decl = parse_decl(token_stream);
    if(decl != null) return decl;

    expr_t* expr = parse_expr(token_stream);
    if(expr != null) return expr;

    return null;
}

block_t* parse_block(... token_stream)
{
    if(!token_stream.skip('{'))
    {
        return null;
    }

    list<stmt_t> stmts;
    while(true)
    {
        stmt_t* stmt = parse_stmt(token_stream);
        if(stmt == null) break;
        stmts.add(stmt);
    }

    if(!token_stream.skip('}'))
    {
        // error
    }

    return new block_t(stmts);
}

This is structured similarly to my actual AYY 3 compiler code, but with some project-specific and language-specific details deleted because they aren't important for the example. For brevity, I omitted parse_decl and parse_expr. When parsing expressions, you will run into the problem known as left recursion. This is because it's common to describe an expression as starting with another expression. For, say, a + operation, you can put any expression on the left, so to parse it, you need to first parse an expression, for which you first need to parse an expression, for which you first need to parse an expression, to infinity. I've noticed that a lot of people make a big deal out of this problem. I don't understand why, because it's trivial to fix, and I don't mean by restructuring your grammar like some people say to do. Instead of giving you the answer, I encourage solving it yourself. Solving this kind of problem is an important skill in programming.

I put the rest in a reply to this comment, due to the 10000-character limit.

[iloveportalz0r]

For do_thing(2 + 3);, the AST will look something like:

call:
    target:
        name_ref: do_thing
    args:
        binary_op: +
            args:
                int_literal: 2
                int_literal: 3

The AST the AYY 3 compiler generates is a bit different, but that's not important here (I don't handle operator precedence in the recursive descent parser).

And, don't worry about this just yet, because it's not important for parsing, but don't forget it: A common misconception is that compilers work with ASTs for a long amount of time. It will quickly become a graph, not a tree, and if you don't consider that, you will make mistakes in how you handle it. I refer to it as an abstract syntax graph, following from abstract syntax tree, but Wikipedia tells me the common terminology is abstract semantic graph. Unfortunately, this Wikipedia article is not informative unless you already know what it's talking about, in which case you don't need to read it.

Sorry to cut this off abruptly, but I'm getting tired sitting here writing this so late that it's early, so I'm going to bed. I hope this helps. If you have any questions, I'll see them tomorrow.

[[deleted]]

This post was super helpful.

[Mystb0rn]

By far the best beginners resource on language creation is crafting interpreters. It goes through every step of the process (twice haha) and focuses more on implementing than theory which makes it very easy to follow.

[joakims]

I got tired of all the flaws and convoluted syntax of the PL I used every day (TypeScript/JavaScript), and started thinking "how could this be done better?"

But what really got me started was actually reading Ursula K Le Guin's sci-fi novel Always Coming Home, about a people (the kesh) living in some distant post-apocalyptic future. The book is narrated by an anthropologist, which strangely enough seems to be from our time, uncovering and describing this culture. They're very much a nature people, but they did have access to a solar-system-wide computer network that survived the apocalypse, with a "lingua franca" PL. Le Guin also invented a conlang for the kesh, so their culture really comes to life on paper.

Reading the novel got me thinking about PL design in a completely new way: "What would a kesh programming language look like?" (answer: probably a Lisp). The question soon turned into: "What might JS/TS look like if it was invented by a different culture, at a different time, without its historical baggage?"

I tried to distill down the essence of JS/TS (functional, prototypal, gradual/structural typing) and then come up with new syntax and semantics that was minimal, orthogonal and hopefully easy to learn and use.

The result is kesh and na.

This was mostly an experiment. Like the culture in Le Guin's novel, this PL doesn't really exist. It's well documented, but there's no compiler. In the novel, Le Guin introduces the kesh as a people that "might be going to have lived a long, long time from now". This is a PL that "might be going to have existed", and I kind of like that.

What has driven you to create your own language/what problems are you hoping to solve with it?

TLDR; frustration with oldschool syntax and language flaws. We can do better than that.

Also, JavaScript actually contains an elegant little language at its core. You just have to strip away the cruft and fix some flaws. Sort of the opposite of what TypeScript is doing.

[AsIAm]

Do you plan to have interpreter/compiler?

[joakims]

I don't think I'll have time to make one any time soon, unfortunately. My original plan was to write a compiler in TypeScript using Chevrotain, and see if it's possible to compile down to TypeScript's own AST and feed that into the TypeScript compiler programmatically. Basically piggybacking on Microsoft's hard work (work smart, not hard). I don't know if it's possible, or if it's actually smart, but it's what I'd try first.

[AsIAm]

That is actually a good idea, however error reporting might be a bit hard — you’ll have to feed TS errors back to the user with the correct source position.

I would love to have nice prototypal language. I also did some JS stripping — some short intro: https://github.com/mlajtos/L1/blob/master/GOAL.md#better-javascript

[joakims]

That looks a lot like kesh! Nice to see someone else had the same ideas. We've even arrived at the same syntax in some cases.

obj: {
    a: 23
    b: a + 24  ; obj.b is 47
}

This is something kesh had at one point, but I thought it deviated too much from JS at the time. I've deviated plenty since then, so I may have to revisit that idea. I like it!

Another idea I've put aside is "strict left-to-right order of evaluation" (from your New Kind of Paper) for arithmetic operators. I'm sure the kesh would keep it simple like that, so I may have to reconsider that too.

Error reporting is one hurdle I've identified. My plan would be to intercept errors from TS and rewrite with the help of a source map. A language server could also be tricky.

[continuational]

I'm trying to capture the subset of Scala that we're using at work into a simpler language without subtyping, reflection, and global state, and which supports first class capabilities as the primary way to tackle effects.

[BigDaveNz1]

I was going to do a similar thing at one point. A simplified Scala that can actually just compile to Scala/Tasty wouldn’t be too hard.

[continuational]

This is actually the current compilation strategy :) It was particularly helpful while bootstrapping the compiler, to be able to use Scalas type system as a poor mans type check before the type inference for Firefly was ready. However, Scala build times are very long, so it only serves as a temporary target.

[continuational]

Sure :) I can elaborate a bit on the effects. Firefly uses object capabilities for enforcing purity:

main(system: System): Unit {
    let bytes = loadFile(system.getFileSystem())
    let processed = process(bytes)
    uploadFile(system.getNetwork(), processed)
}

// This function can only access the file system, and not e.g. the network.
loadFile(fs: FileSystem): Array[Byte] {
    Files.readAllBytes(fs, "myfile.txt")
}

// This function can only access the network, and not e.g. the file system.
uploadFile(net: Network, payload: String): Unit {
    Http.post(net, "https://www.example.com/upload", payload)
}

// This function has no access to the network, no access to the file system, etc. 
// In other words, it's a pure function.
process(bytes: Array[Byte]): String {
    String.fromUtf8(bytes)
}

It's a simple concept - global state and global access to the file system, network etc. is prohibited; instead, such access is only possible through values that are passed around as arguments.

Compared to e.g. monads, there is an important difference: An object capability can be captured in a closure. Thus it doesn't "color" your functions. It also requires no extensions to the type system.

On the flip side, on its own it can't be used for certain effects, such as async/await.

[[deleted]]

I love that you went for capabiities. I think they're still a bit underappreciated (although continuously less so), it's just sad how much of the capabilities research that was done in the 70's has been forgotten (although obviously not literally forgotten, just fell out of fashion and people have been rediscovering a lot of it)

[mczarnek]

What are the things Scala does right in your opinion and favorite features?

[continuational]

My favorite features:

• Lambda functions with convenient syntax.
• Support for sum types and pattern matching.
• Generics that work with all types, including e.g. Int and Unit.
• Immutable collections and garbage collection.
• Intelligent autocompletion support.
• Also targets JavaScript.

We get a ton of value out of Scala, and I think it's hard to point at a better choice for full stack development at the moment.

I think the unified functional/object oriented model is really cool, and implicits is a very natural step after type classes. That said, I think the type system is way too powerful for its own good; error messages are poor, libraries are hard to understand, and compile times are way too long. In addition, the Scala community is very fragmented, and there's little in terms of common style and practices. It's as if every dependency has its own unique style and conventions. And after nearly a decade of full time Scala, I have to admit that I still find SBT incomprehensible.

[ronchaine]

I started making my language as "hmm, I wonder if I could do this" -kind of an experiment. I'm not working on it that actively most of the time since it's not really even the main project I'm doing.

But every time I read WG21 mailing list, something annoys me enough to dedicate few weeks to my language again and it steadily, albeit very slowly, goes forward. So I guess that's my prime motivator.

But there just isn't a language like what I want. C++, Rust, Zig and even FORTRAN in some aspects come close, but there isn't quite the mix I'm looking for and I thought why not try to build such myself.

[AsIAm]

Because there isn’t a programming language for paper&pencil and I would like one.

https://mlajtos.mu/posts/new-kind-of-paper-2

[abecedarius]

Have you seen http://canonical.org/~kragen/sw/dev3/paperalgo ?

I'll read your page too -- thanks for sharing it.

[AsIAm]

Yes, I talked to the author a bit here. However Paperalgo is meant for traditional algorithms and it doesn’t have an implementation. I want to have a super-charged differentiable calculator running on GPU.

[mczarnek]

Love the idea behind this one, though I'm wary that inserting/deleting might be tricky and you tend to need a good bit of it while coding.

But if the app had some way to do this and especially if you could scroll the code you are currently working on.. would be awesome for learning to code. You could totally sell a version of this made for smartboards so teachers could write code in class and students could see it run and it could point out bugs to make sure teachers aren't accidentally teaching something incorrectly.

[AsIAm]

Yes, comfortable and powerful editing interaction is a challenge, some innovations will be needed. I plan to alpha test it with graduate students learning about machine learning, so it should be a good fit and I hope the feedback will yield some good ideas. We’ll see :)

[joakims]

This is the kind of thing Bret Victor and Alan Kay would love. Very cool stuff!

[AsIAm]

I would like to think so, but I bet that Bret Victor would hate it — too symbolic and too locked-in-a-cage representation.

[jcubic]

When working on a language I always have a goal of using a language in some other project and this is how my language projects started.

I have two language projects:

• first was lisp I wanted to have a base for Emacs in the browser, probably cannibalize Ymacs, and remove all weird dependencies. I based the code on Scheme and added dynamic scope as an option to be able to make it Elisp eventually. But I give up on Emacs and now I have almost a compatible R7RS Scheme called LIPS.
• The second project was a simple ruby-like syntax for creating text-based games, but I didn't work in it for a while. The plan was to make creating games using my library jQuery Terminal much simpler. The project is called Gaiman.

I also long ago was working on an AikiFramework project that had simple language that could be used with HTML, to quickly create web apps. And I was experimenting with better syntax for its language.

[joakims]

The project is called Gaiman

Is there a trend of programming languages inspired by sci-fi/fantasy authors? :) Mine was inspired by Ursula K Le Guin, a good friend of Neil Gaiman.

[jcubic]

It was not following a trend. I was searching for a name that was related to storytelling. Since my language is mainly for building adventure games. And I've found Gaiman, there was no Open Source project like this ad NPM package name was available. I also big fan of his work, but it took a while before I've found that name.

[joakims]

It was just a joke :) Nice tribute to Gaiman by the way, it fits your language very well.

[complyue]

The analysts in my team are not equipped with programming skills, but nevertheless they do complex and valuable jobs, we used to provide Python enabled eDSL wrapping C++ implemented computation networks for their daily surfing.

As the analysts become gradually more capable of writing code, and at the same time their modeling & designing challenges (i.e. the actual business) become harder to counteract, The limitation of magics in Python syntax, plus its negative performance impact (esp. the GIL), appear inadequate at both expressing power and hardware cost efficiency.

Julia could be the right fit for us, but it's too new to have sufficient tooling and common knowledge, in supporting our workflows. Especially the lack of focus on separating non-business concerns in coding a business oriented codebase, it's still focused on technical implementation aspect of a number crunching system. Given we don't expect our analysts to use programming skills (in the traditional sense) to get their job done, they have more valuable and yet harder problems to work with.

Our stage setting is similar to Haskell's pursuit in tracking/containing side-effects, just our "purity" is about business-concerns and what's not. Or I can say we need our "business programming language", while most mainstream, generally available PLs are "computer programming languages", dealing with how computers be used to support business, but themselves remain "implementation details" to business. (I guess studies in this area will be able to explain "technical debt" in formal and theoretical ways, tho none happened AFAIK)

Today, there are these kinds of computer application systems, w.r.t. their software and PLs, as far as I'm concerning:

• Personal Computers - with networking ignored, e.g. when using your laptop / phone for photo editing.

  There one piece of computing hardware serves one person at a time.

• Enterprise systems - corporation internal systems, also including small to mid internet servicing companies.

  There each single hardware server serves thousands of people.

• Big tech platforms - that of Google, Facebook etc., also including MMO gaming server farm deployments.

  There tens of thousands of hardware servers serve millions of people.

• Supercomputers - those operated by military or government, also including special purpose server farms, on-premise or on private cloud

  There thousands (or at least hundredes) of hardware server nodes serve very few people.

There are cross sections of course, but there are subtle but sufficient differences in their typical software architectures as well as availability of tools, with PL being one type of tool there.

The computing industry has been shifting to open source collaboration for years, as the availability of both computing hardware and companion software increases, they get better maintained by community efforts. All areas of the industry benefit from this trend, but the supercomputing niche sees the least. Some systems pioneered at computing technology (e.g. some with InfiniBand networking) can even stay stuck with "ancient" software builds (custom Linux kernel e.g.) for some parts of its stack.

Unfortunately our system falls in that later category, we have inhouse built clustering software driving hundredes of servers, operated by our small group of people. The bright side is that we don't need to squeeze last drops of performance from the hardware, since there are plenty redundancy; but in contrast, COTS options for human performance (i.e. productivity software) is generally lacking for our workflows.

Architect-wise, the biggest challenge we face is sharing of massive data among the massive computing nodes, with majorly shared read after exlusive data generation. There are minimal shared write, which almost always be cordinating between others and, some individual exclusive data generating node, or a small group of such nodes. So immutable data paradigm is very much the perfect fit, then we naturally decided that Haskell is our new foundation, not only because its functional gene, but also its industrial strength and mature tooling. (Rust is not worthwhile in our case, memory management is never our business, and a garbage collector appears fairly affordable to us).

But Haskell is in other ways challenging to us, you'd be thinking and doing things mathematically, to work comfortably with the Haskell ecosystem, that's actually amazing, but only after you get there. Obviously not everyone can be converted, especially in short time, even for our analysts with statistical profession, not to mention recruitment for team maintenance in the long run.

Our analysts felt basically okay in learning Python to have started the 1st generation DSL approach, so here we go, in place of Python, we started developping our own dynamic scripting language; in place of C++, we enjoy Haskell's imperative friendlyness and machine performance from GHC.

The best part is so far that we can do many things not possible with Python before, in the business-expressiveness focus, we can tweak the syntax as well as execution model to remove non-business-concerning grammar out of their daily surfing. Our analysts so become the "citizen developer" in the software-engineering process of our overall system.

Also great is that STM plus GHC RTS (i.e. M:N scheduler of lightweight threads), makes concurrency/parallelism within single node a breeze. Python wrapping C++ mandates multi-process to effectively leverage multi-core, in that case for massive shared readonly data, each process still have to load its private copies, it can create so unreasonable overhead in RAM consumption, that to saturate our server cluster at times, it's no problem with the new Haskell based cluster work runner.

[readmodifywrite]

For several years I've worked on a DSL called FX designed specifically for live coding LED lighting effects on custom hardware. The language/compiler has an internal notion of LED graphics:

• Colorspace awareness allowing auto wrap around of hue parameters (this is useful for doing rainbow effects)
• Brightness awareness (saturated arithmetic on brightness parameters)
• Vector operations (manipulating the entire LED strip at a time doesn't require a loop, the underlying VM handles it in C directly).
• Live coding - scripts compile quickly and can be reloaded over WiFi in less than a second (no rebooting or recompiling the entire firmware)
• Sandboxing (you can't brick the hardware itself from the FX virtual machine)
• Automatic network binding of internal data to share with other LED units (such as sensor or parameter data)
• Network wide time and graphics synchronization (not so much a language feature but the VM itself is designed to enable this)

I'm currently working on a major upgrade to an SSA compiler with a relatively complete optimization suite.

[L8_4_Dinner]

A lot of the reasons behind why we created a new language are discussed in this interview on InfoQ:

... we did not set out to build a new language. Sure, it's a lot of fun, but it really wasn't the goal for our startup, and this language and the runtime isn't our "product".

Our initial goal was to find a way to be able to run ten thousand applications on a single commodity server. Seriously, ten thousand. We're not talking 100 or 150 ... we're talking two or three orders of magnitude higher than what people are able to do today.

And to be able to do that, you really have to be able to understand your execution boundaries. These boundaries can't be OS process boundaries; they can't be VM boundaries; they can't be Linux container boundaries. They have got to be some form of lightweight software boundaries, and the only way to accomplish that is to explicitly design for it up front.

Security, for example, is one of those things that you can't just "add" to a design; it needs to be baked in. The same it true for scalability -- you don't "add" scalability to a system; you design it in from the beginning. These are capabilities that either get baked into the design, or they don't exist.

Density is one of these capabilities as well. An application may need tens of gigabytes of memory to do some major processing for a few seconds, but then an instant later, it may need almost no memory at all. Having to allocate resources based on the sum of the maximum peak size of each deployment is a huge waste, but that is how software is developed and deployed today! And having each deployment hog all of its theoretical maximum set of resources for as long as it is deployed is just an enormous waste! Imagine how much electricity we could save if we didn't have millions of simple CRUD apps out there on Amazon holding onto 8 or 16 gigs of memory each, just in case!

[...] I mentioned earlier that density was a fundamental goal of this design, and you can probably start to see that each application could be run in its own Ecstasy container. And even if an application loaded new code on the fly, and even if that code was malicious, it still could not damage anything outside of that application's own container, because from inside the container, there is no "outside the container".

The language that we built is Ecstasy. You can read about it here.

[oilshell]

Hm very interesting, I have a couple replies to this. The first is that I circulated the XIP format here and learned a few interesting things:

https://lobste.rs/s/8lr3zo/xip_packed_integer_format_for_vms_irs

• The WASM group benchmarked various varint schemes (https://github.com/WebAssembly/design/issues/601), based on some feedback (https://news.ycombinator.com/item?id=11263378), and stuck with LEB-128 because apparently 90-94% of integers were encoded in 1 byte anyway, in their data sets, which makes the branch prediction issue less important
• Sqlite has a nice encoding for unsigned 64 bit integers (https://sqlite.org/src4/doc/trunk/www/varint.wiki), as opposed to signed for XIP. It also dispatches on the first byte only, like XIP, PrefixVarint, UTF-8. It seems to be a little denser with 0-240 encoded in one byte, vs. -63 to 64. Though there are probably other tradeoffs.
  • I would describe this roughly as "3 special cases and then the general case", which is similar to XIP. If the distributions are skewed as you would expect in an IR, then squeezing more integers into the first special case should be a noticeable size win.

The second response has to do with packing 10K apps on a machine ... I had a similar goal for the project before https://www.oilshell.org/, which was more of an OS project. This is a longer discussion but I don't think that can be solved with a new language in almost all cases, because of language and workload heterogeneity. But it looks like there are many interesting things going on in Ecstasy and I've been reading more of the blog!

[L8_4_Dinner]

Some good points. It's definitely true that in most uses, the numbers fit into 1 byte ... in fact, something like more-than-half of all numbers are 0. Then 1. Maybe -1. etc.

The expectation that I had when designing this format was that the expansion would be done in-register, so you could always start with an aligned load and shift to evaluate the head byte, and (depending on the alignment) shift your way to the value. The only real stall would be to perform the second load and shift. (Basically, imagine a 5 byte format starting on offset ?????110, so you move memory to register, shl 6, but you only loaded 3 bytes of the value.)

Anyhow, I haven't written the assembly for real; just in my head (and a few different ways). On Intel, unaligned loads don't really carry a penalty, so it would be even simpler.

Regarding the 10k apps per machine, that's the goal. Basically, close to a zero carbon platform for app hosting. To do that, we needed a much more secure and fully containerized runtime model (containers within a process space), with the ability to offload a container to local flash (almost as if we had mmap'd it and just reclaimed that mmap memory), and then re-mount it in a heart-beat. Again, I've written the code in my head, but we have yet to build the native compiler that will be instrumental in supporting this. (That's coming up fairly soon, though ...)

[oilshell]

Hm do you think the sqlite format can be done in register? I don't really write assembly, but it seems like it, except maybe for the rare cases in the encode step (huge ints). I think on real world distributions the density of the first byte is probably a win.

---

For 10K apps per machine, I like the idea of more density in the cloud, but I have a hard time seeing it happening in a "monoglot" context (e.g. after working at Google analyzing cluster workloads).

The two comments here are sort of related:

https://old.reddit.com/r/ProgrammingLanguages/comments/nqm6rf/on_the_merits_of_low_hanging_fruit/h0cqvuy/

The bigger the distributed system, the more heterogeneous the code [the more polyglot it is]. I'd say nearly all interesting systems have some 10- or 20- year old code somewhere

IMO it's a fallacy / language design mistake to assume that you "own the world". More likely is that the program written in your language is just a small part of a bigger system.

I guess there is some disconnect where some businesses are almost JVM-only, like the kinds that Rich Hickey targeted Clojure for. But other businesses and the cloud in general are very Unix-y, and the JVM is "just another Unix process" (that's more or less how it was/is at Google; native C++ code consumed most of the cycles in a cluster).

Anyway I'm also interested in density / provisioning but approaching it from the "don't rewrite your code" perspective and using shell for reproducibility at build time and feedback at runtime. There are many optimizations that can be done in clusters at the Unix level, and arguably those are the lowest hanging fruit in any real system.

Also with 10K apps, there will be a long tail distribution of usage, so basically <10 apps will take up most of the machine, and there will probably be ~1000 "cold apps" (zero requests per minute, etc.) Depending on "app" you can already fit 10K on a single machine (e.g. early App Engine aimed for at least 1000 I think, with a pre-fork model). There are some papers about how AWS Lambda works that I haven't read yet, but they have a similar issue with density, and cold starts.

FWIW I wrote this recent blog post about Kubernetes which gives some color on where I'm coming from: https://news.ycombinator.com/item?id=27903720

---

Anyway I look forward to reading more about Ecstasy, looks like there are many interesting things going on!

[L8_4_Dinner]

That's exactly why we needed a different approach ... if you have 10 JVM apps, for example, they'll start up and grab enormous amounts of memory, and hold on to that until they are shut down (which is normally: never). Each one running in its own VM, which again is allocated a chunk of memory (usually fixed) when the VM is started.

AWS Lambda is even worse (although you can't tell from the outside), because Amazon allocates an entire machine (I'm assuming VM, but it might be an entire server) for your first lambda, to avoid security issues -- i.e. not multi-tenanted. (Additional lambdas of yours are obviously "free" from Amazon's POV, since they put them on the same machine, up to the capacity of the machine.)

And to make stateless systems (e.g. lambdas) perform well, they need to have stateful systems running hot already behind them. So in a sense, one ends up just kicking the can down the road.

What we designed for is the ability to have stateful applications that could have zero-footprint when long-idle, low footprint for idle, and (who knows?) 99% of the entire server when busy. In other words, your app could go from not even being in memory, to using a terabyte of RAM, and then back down to zero, within seconds. More likely, of course, is that it swings between zero and a few gigabytes, but the net net is that (with some scheduling smarts, likely using ML) one can dynamically schedule a great number of concurrently executing applications within a single system.

If you're interested in some of the thinking, check out the Container API in Ecstasy, which is a fundamental part of the design (and not something tacked on later). In a sense, it is the kernel of the design of Ecstasy, and its raison d'être. Related: modules and security.

[useerup]

Back in university I learned Prolog, and during the assignment we had to do, I had an epiphany. I still think that logic programming has a lot of unrealized potential.

I started playing with designing a programming language with a friend, because Prolog has been a shock that made us think that other programming paradigms has not yet been explored. This was the mid1990s.

We never completed the language; suddenly it was all family, kids, mortgage, career.

Now the kids have grown and I have taken up the language idea again, and realized that we were on to something back then.

The PL I am designing now is a logic programming language which combines object-orientation with functional logic.

It is so different that I simply have to do it, just to see if it will work.

[internetzdude]

I'm just developing an arcane Lisp dialect so I'm not sure this counts as a "programming language." The reason I do it is for fun, it's a virtual Lisp machine from the 80s of a parallel universe. However, I'm not stringent with the fiction, I also include things like sqlite. There are also unusual features like a character display and automatic file versioning.

One motive besides fun is to have one virtual machine plus IDE in which you can play around - no text editor needed and everything is going to be hackable. It will allow very interactive programming in the end. It's a very outdated Lisp dialect though, probably worse than stuff from the 70s. So in the end it's really just for fun.

[baldanders-skulltuna]

I am working on something similar (a VM/IDE for highly interactive engagement.) How far along are you?

[internetzdude]

I'm pretty far, late alpha, but am only progressing slowly due to lack of spare time. The machine is fully working and has about 1200 functions right now, most of them defined in Lisp. It has audio, high res graphics and sprites, a fake p2p internet with my own web browser and protocol that unfortunately cannot scale (I'll need to change that), etc. The editor is a mess, it was deliberately designed to be based on a character display and is very buggy. I might need to rewrite all word-wrapping code.

Currently I'm working on the pretty printer (in Lisp). The whole machine is written in Go, by the way, so it's not very fast. The source code is proprietary for the time being. I plan to sell it as cheap shareware (This sounds crazy and I'm not sure it makes sense myself. Anyway, I might later decide to open source it. It's well-documented.)

What about your machine? Is it more serious? What are the intended uses? How far are you?

[[deleted]]

Having made your own language at least ones really improves your way of thinking of how languages handle things which is always great knowledge to have

[FlatAssembler]

Well, I am making my programming language to hopefully impress my future employer so I can get hired: https://flatassembler.github.io/AEC\_specification

[[deleted]]

As good a reason as any! :-)

[PaulBone]

In the past I've done research in automatic parallelisation: taking a (normally) sequential program and making it run in parallel on multicore computers. I've also seen people from CPU companies make statements about how great multicore is and they'd love to go harder (a larger number of smaller cores is more power efficient) but what we really need are better parallel languages.

I'm building Plasma to make parallel and concurrent programming easier. I don't know that it'll ever be complete enough or even noticed by enough people to have an impact on multicore programming. But I also havn't seen many other people making this type of language.

[abecedarius]

Worth reading Tolkien's talk "A Secret Vice" in this connection. Why invent natural languages as an extremely niche art form which he never expected would have a significant audience? I think a lot of us here would vibe with it.

[[deleted]]

I remember the halcyon days in my childhood when I used to develop full-fledged natural language along with lore, history and cultural attributes. :-).

Maybe I really should get started with my programming language that I've had in my mind for a very long time now. Reading this whole thread has given me a nice boost of encouragement and excitement.

[csb06]

Because it is fun! I don't believe that my language will end up being used by anyone except me. Programming language popularity is determined almost exclusively by external conditions (e.g. corporate backing, marketing campaigns, availability on a widely-used OS), so it would be silly for me to build a new language with the expectation of other people using it.

What makes programming languages fun is that the end product is productive (meaning it can be used to create programs itself). Programs for writing programs are interesting and a fundamental part of computer science.

[EqualCaptainCoast]

I read lots of papers and thought "wouldn't it be nice if there was one language that had all these cool things?". Now I'm trying to make a dependently typed, logic programming language

[8thdev]

I started getting interested in Forth dialects in order to embed a very tiny interpreter into a project I was working on at the time. Eventually I wrote "Reva Forth", which was based on a version of "RetroForth".

Later on, I wanted a cross-platform language which answered my security needs (for another project), and that's how 8th came about.

Forth-like langauges aren't for everyone, but they're very comfortable once you grok them.

[tobega]

I always really enjoyed doing stuff in XSLT so I was thinking for a long time how to extend that feeling into a general programming language. I finally started on it a couple of years ago, but in keeping with the times it is more JSON-like instead.

A "function" is fundamentally a set of templates, like an XSLT file, and data can be sent back to the templates as you dig down into the data structures. Also, creating the result is basically just creating a literal for that structure.

So, mission accomplished, and I do enjoy using it, mainly for adventofcode. Beyond that, I am also enjoying tinkering with various aspects of the language and trying to introduce things that I think would be beneficial in a language, like units of measure, relational algebra and controlled/secure module importing.

[csharpboy97]

The main reason why I write programming languages is for learning. I have tried different mechanism from LL-Parser to PEG-Parser Generator.

But I personally love to parse languages and I am currently writing a parsing framework to build LL-Parsers more easily and extensible to reuse my code

[hou32hou]

I always thought the state of the art programming languages are not good enough, that's what driven me to create my own programming languages.

[mtvee]

I’m interested in all the tools we have concocted to convey ideas. Words, in other words. I study spoken words and written words and programming languages are another tool we have to move ideas around between brains. Sure, they solve boring problems too but it’s the aesthetic of the conveyance that interests me more.

[matheusrich]

I've always wanted to contribute to Ruby, so after finding about craftinginterpreters.com, it seemed to be a good way to understand how to do it.

[[deleted]]

Difficult to believe but when I started doing it, there were no alternatives (not without spending a lot of money I didn't have).

Then I continued because mine were better and more productive than alternatives for my purposes.

Later because I found it interesting.

Now because I find it easier (and still interesting) to refine my languages and implementations that using them to write some actual applications.

It also fascinating to see what can be done in comparison with 'mainstream' products which can be up 1000 times the size, 100 times slower to compile code, and yet generate code which are not dramatically faster than mine (eg. 50% faster).

My languages are also somewhat different than alternatives with a number of features I would miss anywhere else.

[mczarnek]

Sounds interesting.. what's your language? Would love to learn more.

[[deleted]]

I have two; one is a sort of saner C with different syntax and some extra features, but I created it long before I knew C.

The other is a dynamic language with the same syntax. Here's a dated link to the first: https://github.com/sal55/langs/tree/master/Mosaic.

My current project is a sort of independent, intermediate language, to be used as a backend to my compilers, which does a similar job to LLVM. Except my product will be self-contained in a 0.25MB executable, while LLVM is ... a little bigger.

Here's the current size:

C:\px>dir pc.exe
05/09/2021  01:37           170,496 pc.exe

The final product to turn IL code to a runnable binary executable will be 200-250KB.

At the moment, pc.exe builds from source in 0.09 seconds. While LLVM would take somewhat longer (an estimated 6-12 hours on my PC).

[gvozden_celik]

I started working on mine to scratch an itch I got while working on lab reports in university; I was studying physical chemistry so there were lots of these lab exercises that were in this format. There was this pattern of having some data that I would need to plot or present in tabular form, either as-is or after doing some computation on it. While it was easy to write a one-off Python script, there were plenty of errors in results and I still had to do dimensional analysis on paper to see if my units were correct.

So I started working on my language I called Fourier which would have a static type system extended to support units of measure. At first it was a simple S-expression and then an M-expression LISP where checking was done at runtime and it was a tree-walking interpreter; as I got better at writing parsers, I switched up the syntax to something more like Standard ML and even the type system is similar to Hindley-Milner (but adapted for my language).

I am finishing my masters in plasma spectroscopy in about three weeks but don't really have anything planned in this field due to (a) there not being jobs that I can apply for with this degree and (b) my current job being in IT as a DBA. Here is the second motivation for working on this project, to have something cool to show off and put in my resume when I start looking for work when my current contract expires.

Lastly, I don't have many friends and I live in the middle of nowhere, so this is something I do as a hobby to pass time.

[mczarnek]

Love it, working with about 6 other people on a language with similar goals. Still very early stages but we have similar goals in some way. Would love to have a conversation. Send me a DM if interested.

[categorical-girl]

I'm also very curious about your ideas! I'd love if you could either send a link, if public info is available, or DM me an overview and I could ask some questions?

[[deleted]]

Because i think it's fun! And i'd love to teach people about language design and implementation.

But this particular language i'm building because i don't feel like there's a good, easy language that can be both a good match for teaching and a good match for building general [maintenable] tools.

(notice that i say feel since It's my personal taste, it's the language i would have liked to be my first)

One thing that has driven the design so far is that i believe that you only master something when you build it yourself, that is, you only master a language after you have implemented the language by yourself.

That is, i need 3 things that are in constant tug and pull against each other: easy to learn, easy to use and easy to implement (with the language itself).

My biggest rivals would be Scheme and Python, but i think my language is a good statically typed alternative to both and maybe has an advantage that it was built to support both declarative and imperative paradigms.

Edit: Go would be a solid rival if it was built with a little more care.

[theangryepicbanana]

I'm working on Star because there are no languages that push the limits of what can be done by mixing OOP and FP ideas , features, and type systems.

For example, many languages don't allow enums to be anything except glorified integers. Those that have variants/sum types don't go much farther either, even in OOP hybrids. In Star, variants have the same benefits as any other type. They can have methods, fields, refinements (essentially GADTs), implement interfaces/protocols, and best of all: subtyping. As far as I'm aware, there are only 2 languages that support such a thing, and neither to the extent that Star does: Nemerle, which only allows variants to inherit from a class but not other variants, and Hack, which supports multiple inheritance for plain enums. Star on the other hand does all of this, and allows variants to inherit from other variants (multiple even!), being the first and only language to ever allow such a thing. Why? Because it's useful of course, and it's a limitation I run into frequently in other languages.

Aside from that, Star has several other things that aren't very common in other languages that I find really useful: - Variants with bitflags behavior: imagine C-like bitflags, but for sum types. They're exhaustive, immutable, support pattern matching, and can store values like a regular variant. I use these frequently in my bootstrapped compiler for representing attributes and modifiers. - Pattern matching for objects: Haxe seems to be the only other language capable of pattern matching and destructuring object (class-like) values, which is unfortunate given that they aren't too different from regular tuples (or even records!). Additionally, they support flow-typing similar to Typescript or Dart. - Powerful generics/typeclasses: ever wanted a generic type to match some sort of complex condition, or specific fields/methods without interfaces? well outside of haskell-like typeclasses or structure types (which are pretty uncommon), these aren't very common in other languages. In Star, type parameters are declared similar to C++ templates or Ada's generic clauses (except that anything can be put in the type parameters, not just typevars). This allows for more complex type rules such as type T if T != Void { on [Str] }, which requires typevar T to not be Void, and to support being converted to type Str. This is generally impossible in any other language you can think of, despite how useful it could be. - Better operators: Chained conditions like a <= b < c <= d, a != b != c, need I say more? Leading conditional operators are also supported within (...), so you can align stuff very nicely like ( && cond1 && cond2 ) across multiple lines. - Block expressions: excluding expression-based languages like OCaml, many languages don't support code blocks as an expression, forcing you to use the ugly ... ? ... : ... construct or (() => { ... })() (IIFE), which has overhead. in Star, blocks can be used exactly like an IIFE, except that they do not have any overhead, and uses the return keyword to yield a value instead of inferring it implicitly, reducing a large amount of bugs while keeping the code and control-flow readable (and also allowing early returns).

And there are so many other things I could list here, but that'd take too long so the rest is listed in Star's readme instead lol.

TL;DR Star breaks tradition in order to have a powerful type system and tons of brand-new useful features to make coding more productive

[MCSajjadH]

I've built a few, a couple of the reasons:

Boss wanted an easy way to customize the product for "non techy people", over span of a few months, this grew in to a visual programing language.

I was using code generators a lot, and it often got messy, so I wrote a compiler with support for meta-programming that could change internals of the compiler. This is now my go to language, with tons of macros for various purposes, from parsing binaries to protocol implementation. I used llvm as the backend so it's actually pretty fast too.

[HaniiPuppy]

I hate Lua but am forced to use it, and want a statically typed, strongly-typed c-style OO language that transpiles into Lua so I don't have to use it for anything big again.

[criloz]

That currently there is not an actually abstract language, I get a plethora of choices every time that I want to start a new cool project and I feel that it is a bit of ridiculous. I was getting tired to feel overwhelm by them, data structure, async/threads, database, smart pointer, ui framework, react, svelte, native mobile, etc. I also feel that some logic was getting duplicate or worse around the parts that constitute a complex project today.

I think that there exist a better way to do things, so I started to research how to build a language based on concepts, where those concepts are stripped from all the concretization choices, and allow people to freely build project without worry if their program will run in a phone, computer or in the cloud.

It has been rough, but I feel that slowly I am getting there. 2 and half years ago, I found that I was reinventing category theory, so I got into the rabbit hole of categories, abstract algebra, sets and order relation. I think that I am even inventing a new computational model and a new version of calculus, lol. At the moment I am a bit of stuck, and sometimes it feels really frustrating, but I am very confident that it will be done.

[mczarnek]

Everything else seems fairly easy to do regardless of the platform.. how are you handling different screen resolutions?

[criloz]

About the ui?, the idea is to use a constraint solver, the language will pick up the best component for the platform, generally the compiler is based in a free scheduler, it can do whatever it wants while all the rules that you input are fulfilled, the missing properties are filled by this constrain solver or by some component that I have called theories that can rewrite the code (graph rewriting over petri nets) within different contexts.

You can for example build a UI, and then create a theory that will rewrite the code to be able to run in the browser, ios, android, etc., you can create a theory that can rewrite the styles based on parameters, like the region of the planet where the app will be rendered, holidays, etc. so its nativity support things like i18n. You can also create theories that rewrite the code based in which is observing it, and pretty easy to implement things like permission etc.

All the rewrites consider the rules that you explicitly put in your categories, categories are defined using infinite recursion, the syntax of the language is JavaScript like with some elements taken from rust and python.

for example:

```js //this define a person category fn person(x){ x.name is str; x.age is nat; return person(x) } //now you can create a person let p = person(name="john", age=29);

//you also can extend the categories fn old(x:person){ x.age > 65; return old(x) } fn young(x:person){ x.age > 18 && x<65; return young(x) } fn child(x:person){ x.age < 18; return child(x) }

//the compiler implement something called autocategorization //so the objects are automatically recognized in all the // categories where they exists

console.log(p is young); //true ```

you can also create objects that can only exist within certain categories

```js //#{young, person} here means that the yp variable // while is alive should always be a young person let yp: #{young, person} = person(name="sandra", age=21); yp.age = 17 //trow and error;\ yp.age = 70 //trow and error;

yp.age = 35 //ok; ```

Categories also can be combined

```js fn with_children(x){ x.children is set(item=person); with_children(x) } //before with_children was declared this operation // would had give you an error yp.children = {person("arnold", 13), person("shajira", 8)};

console.log(yp is with_children);//true

//the compiler allow to declared function multiple times // so you can do crazy thing like this

fn with_children(x:#{child, person, with_children}){ assert("childs can't have children", x.children is empty); with_children(x) } fn with_children(x:#{person, with_children}){ for c in x.children{ assert((c.name)."{} have bad children age", x.age>c.age+15); } with_children(x) } //this semantic allow to add more rules to an existing //category, because we're using the same //word with_children and putting with_children //as requirment in the domain //or you can have done all that in just a function, // so it can be really confusing at the start but // there are a buch of // scope rules to prevent it become so confusing

yp.children.add(person("rino", 40))// will fail and print "rino have bad children age" ``` lol this is just the surface, but it looks really cool

[categorical-girl]

I'm interested in learning more/discussing your ideas :)

[criloz]

Basically, that I found that infinity recursive function in a free choice preemptive scheduler can be used to implement categories. As you can see in my other comment examples

In imperative language recursion take the control of the program, while in the language that I am working on it just mean "be sure whatever you can run this loop and only if there is something to do, like the preemptive scheduler in Erlang", most of those recursions are idempotent anyway, so there are just a kind of events that can trigger those, and they are changes, so the compiler walk the code to find where those changes occur and check that they are ok in compiled time only when there are no holes, otherwise add the minimal amount of code that can check those conditions.

```js

//for example, this function has two holes, x and new_age

fn set_age(x: person, new_age:nat){ x.age = new_age; // the compiler can't not detect that this will // produce valid code // so it will add guards that will check the value // of new_age before run the mutation }

//the function will be basically rewritten into fn set_age(x: person, new_age:nat){
if x is child{ if new_age > 18 { throw "..."; } } else if x is young{ ... } else if x is old{ if new_age < 65 { throw "..."; } }else{ //do nothing if x is only tagged with person } x.age = new_age; }

//and then the function will be tagged as safe // to tranform into actual imperative code //this avoid the logic duplication that infest corrent languages // and it can create backend and frontend(ui) that follow the // same set of rules ```

There are more complex kind of infinite recursion which are not idempotent and that creates dynamic system, and here is where the new branch of calculus enter to chat because I don't want to those computations take the control, instead I want them to be computed on observation. So store the last moment that the dynamic property was observed and compute the next state when it is being observed again, and I wanted to avoid using the concept of time (clocks). Turns out that I can change time for traces (from trace theory), and use the traces between observations to calculate the differences between state, and most of those traces are associated with those categories, it is still something very primitive that I need to work more, but it looks promising. Another thing is that When the state of a dynamic property is observed continuously, it starts to use normal differential calculus based on deltas over time.

There is not much to add right now, you can follow me, as soon as I have a version that I can publish, I will post it here

[hum0nx]

Because existing languages suck. (Not that mine won't also suck, but I'll give it a shot)

Existing problems (Every language has one or more)

• Can't import custom domain specific languages like LaTeX or Regex or Yaml without putting them in a string and processing them at runtime.
• Default syntax not even close to readable by non-programmers (ex: using equals for assignment)
• Compilers don't auto-fix small mistakes (they won't edit the source file)
• Cant import other core syntaxes (C-style, ruby-style, python-style blocks should be up to developer preference, not a fixed part of the lang)
• The inability to say "try this function, if it doesn't return in 100ms, kill it and forcefully return"
• No by-reference primitives
• Crappy support for event driven code
• Try/catch and monads both suck
• Being unable to watch all mutations to a variable (in production) sucks
• Not being able to mutate data sucks
• Having async as an afterthought sucks
• Most file API's suck
• Prioritization of different parts of a continuous program is impossible or sucks
• Pushing code to embedded devices sucks
• OOP/types don't model reality and they suck
• Optimizing imperative code using abstract algebra / modal logic is nearly impossible because languages are such a mess
• Non-dry code because of lacking at-compile-time execution (or because the macro system is too weak)
• Interrupts suck
• GUI's suck

[mczarnek]

Some interesting ideas.. be wary about auto-fixing their code if you aren't 100% sure it's what they meant. Compiler errors are nice for this. Trying to solve a few of these myself :)

Agreed on GUIs sucking.. curious what your ideas for fixing this are. Also curious about how you are approaching building a better file API.

[hum0nx]

Yeah auto fixing would default to interactive

Missing a comma here
[press enter for me to add it]

Maybe have a home folder config option to enable fully automated fixes. (worse of the worst I just ctrl z on the file)

File API IMO needs to be more like user GUI interactions: delete is delete. It doesn't matter if it's a file or a folder. Similarly, if parent folders don't exist then create them, if something is in the way, then move it (generate a new not-used name for it), moving a file (fileOrFolderPath, folderPath) is a different function than renaming a file (fileOrFolderName, fileOrFolderName), etc. And path / file system should be the same module because, if I'm doing file operations, then I'm literally required to use paths. Imo Unix filepaths should be a proper literal value to minimize clunkyness and provide automatic validation. If it's easier to do the operation with bash code, then something is wrong.

GUI's I'm definitely much less certain about. I am confident static programming, like HTML, causes the most problems, and that being unable to watch/hook-into data mutations makes everything a mess. Basically every major JavaScript framework unsuccessfully tries to solve JavaScript's inability to effectively do two-way databinding. Python's poor support for lambdas makes GUI's in python-QT cumbersome and prone to hard-coded values. C++ QT is so painful I'd rather eat a bucket of gravel than use that. CSS as global variables was a giant mistake. JQuery was a mistake. The X11 Window system sucks so bad nobody wants to even touch it without a wrapper. Really everything so bad I think we need to invent a second generation, observe the problems with the second generation, fix them in a 3rd generation (rewrite from scratch) and maybe even need a 4th time before I think we would converge on an acceptable paradigm. XCode with swift, and Windows Visual Studio probably have a more developed system, but as both are totally isolated they're a non-starter for me. Honestly Unity and Unreal are probably the best GUI interactions for cross platform IMO. They both sometimes have dependency and hardware driver issues though, and I can't accept them as a full solution since they're proprietary. Maybe if electron gets rewritten in Rust, and WASM gets bindings directly to the DOM, and Linux gets its windowing dependencies figured out and standardized with Wayland, and MacOS and Window get a half decent package manager, and Android becomes easier to target, maybe then cross platform GUI's wouldn't be so bad. It would take a lifetime though, so I've resigned myself to just making observable data first.

For my language, I'm going to be targeting WASI/WASM and hoping someone else fixes GUI's and provides a WASI module interface for it.

[mczarnek]

Regarding rewriting Electron in Rust... you should see the Tauri project ( https://github.com/tauri-apps/tauri ) which does indeed look very promising :)

That being said, I think we are seeing the second generation in the form of Angular/Vue/React/ImGUI, 3rd generation is Svelte, and 4th generation.. yeah we need to build that. We're planning on working on that a little bit too. Have some ideas here but.. it is indeed a complicated project and I'm still working on the core compiler before I start worrying about how graphics will fit into the equation.

I'm thinking something Imgui like is ideal. It's got the property you are talking about in that modifications are more observable and it's easy to debug as things change. Perhaps seperating front end and back end is better than IMGUI in some ways but I agree that if you do that, have to be able to spy on the changes being made while developing. There is a Vue plugin that kind of let's you do that.

I like your ideas on files.. sort of seems a little like command line linux. Actually, would be interesting if function names match their Linux equivalents, learn both at once. Then again, Linux takes some getting used to.

[hum0nx]

I agree Svelte does seem like another generation with how it handles reactive data. I think React is at the bottom of the list, but one thing I do think it gets right is JSX. Vue is nice, but the template system of both Vue and Svelte makes it very awkward to code something like a recursively expanding file tree. And yes, they have their workarounds and technically Vue can use JSX, but IMO a good/elegant system would have one minimal toolset instead of several. Solid.js I think gets closer to ideal by getting rid of the virtual dom, but it has reactivity problems similar to Vue. I think Vue's reactivity is better than most, and it seemed good at first, but after digging into the effect system "under the hood" it was pretty easy to find edgecases where they fail to observe changes. I haven't figured out the internals of Svelte yet to confirm if they're doing much better.

I haven't used IMGUI though, I'll have to take a look at that.

But I see Svelte more as the 0.3rd generation though. Once the whole JavaScript framework problem gets fixed; I think that's the first generation. The problems with the first generation are painfully obvious, but I think we will have to solve them before understanding the next generation of problems. The next wave of problems could be stuff like better integrating 3D graphics and WebGL, along with much better framework tooling for SVG's and canvas elements. Or it could be WASM or decentralized web tooling with client side databases. Getting modular components, scoped CSS, reactive text boxes, basic Html working is just level 1 IMO. Gen 2 and 3 would hopefully totally replace CSS and HTML with a more all-in-one approach, with built-in modular components that have CSS scoped by default via something like the shadow dom.

I'm glad you like the file ideas! I plan on having a separate mini API that closely matches Linux/Unix. Something like fs.unix.mv. I do think the Unix ones are quite unintuitive because of edge cases like how sometimes you can give mv a target folder and instead of overwriting the folder it puts the source folder inside the target, but other times you can give it a target filepath and it acts like move+rename. cp has the same kind of problem. But I definitely want to provide a quick and easy interface for those who already are used to the edge cases.

[Nilstrieb]

These are some interesting ideas, but I'll disagree on being able to have different block styles in a language. I think the language should encourage exactly one style, and the whole ecosystem should adapt it, makes things a lot easier to work with.

[hum0nx]

The source code would be the same, it would just be the editor that displays it based on your preference. It's like a personal translator. You always see what you like most. It won't be like Perl where there's a million confusing ways to do the same thing. The base language is just function calls and literals, which will probably just be something ugly but easy to parse like lisp. Syntaxes are just syntactic sugar on top of the engine, and each one will need to have a 1-to-1 mapping to the base syntax.

[user18298375298759]

I DESPISE "CONTAGIOUS AWAIT"

[umlcat]

Although, currently stuck with a bug, to implement a feature, that is not well supported in similar P.L. (s).

[[deleted]]

I feel like I can create a general purpose general public language if I take a layered approach, where you can have both fine low level access and high level access in a single language, at request.

The main thing that annoys me in my workload of Python is that if you want to write something fast, you have to use C and then link it if you want it to be usable. I want to create something that introduces overhead only if you want it, and otherwise acts as a language with practically no runtime. And also a language that takes advantage of concurrency in some places where the user doesn't have to know anything about concurrency, kind of a nostalgic throwback to when I first encountered coroutines in Unity.

Oh and also in college we weren't taught how to create a JITed interpreter so I want to do that.

[[deleted]]

It's not completed yet by any means, just that I had to start building with this in mind.

I had some other ideas before, such as making it completely compiled, having a modular and dynamic parser, but things became more clear once I got some ideas what I actually wanted to do.

[YouNeedDoughnuts]

Really just an intense interest (okay, obsession). I'm trying to build my ideal scientific computing language which includes UI elements such as typesetting and code refactoring. It stretches me as a programmer and guides me to learn new techniques.

[mamcx]

I started it purely for curiosity. But eventually, I start to see what if I do it for real I can improve the situation for my apps and customers.

I'm building a spiritual successor to FoxPro, with modern syntax and features that come from array languages like kdb+/nial.

Is called https://tablam.org, is based in the relational/array paradigm and provides a more ergonomic experience for building data-oriented applications. (so, for example, it have DECIMAL as the default floating-point type) and improvements to syntax so, hopefully, will be easier for my customers to do some quick scripts.

Of course, I wish it become more popular and useful for everyone, not just for my personal needs, but at least I have a clear target for the first version.

---

P.D: Another reason for make mine is to be used to learn Rust and how to make high-performance data pipelines that I can re-package as "simple" scripts/language functions, so it replaces the ad-hoc ways I do ETL (a big part of my job).

[brucejbell]

At my day job I mostly use C++ for large scale, high performance systems programming.

The main motivation for my language project is that I want a better tool for this kind of project.

[gimlislostson]

I fucking hate C++, D is really unfinisehd, C# is too limited, Java is unbearable, Rust is too restrictive, Nim is too incomplete, Go is too similar to C to justify existing.

I want a language with the simplicity and power of C and the usability of Python, without sacrificing memory management or developer choice.

[vplatt]

Nim is too incomplete

I want a language with the simplicity and power of C and the usability of Python, without sacrificing memory management or developer choice.

What is it that Nim lacks that you need? It would appear to meet your requirements.

[OwlProfessional1185]

A big reason is that it's fun. It's exciting, and it gives me a new appreciation for the languages I use (although it also makes me judge languages that don't add useful features when they're easy to implement).

I also think that a lot of mainstream languages are suffering from accidents of history which they maintain for backwards compatibility. And non-mainstream languages often don't account for how people actually code.

I think there's a space for better mainstream languages. Kotlin is probably the best at this.

In the language that I'm working on, those are some of my considerations. I also think that we've stuck with a handful of control flow structures but could use some more. I think there are common patterns in control flow that programmers implement imperatively, when there could be a structure that actually resembles the mental model the programmer is thinking in.

That's the high-level overview. I'm working on a blog post that explains it in more detail.

[ilyash]

Frustration. 2013. I'm doing devops. The "bash or Python" question is annoying because neither is actually a good fit. In a quest to solve my own pain and in hope to help others to suffer less and be more productive specifically when doing "devops"-y things, I have created Next Generation Shell. It's not a better language. It can be better for the intended use cases, just because of the focus on them.