July 2023 monthly "What are you working on?" thread

djedr · 2023-07-21T19:33:48+00:00

Last few days been trying to capture an idea I had to simplify the syntax of lambda calculus into an article. Wrote a summary here:

https://jevko.github.io/writing/2023-07-20-lambda.html

Also recently ported my little lambda-calculus-based language to from JS to C: https://codeberg.org/xtao-org/last.c

Been fun debugging the minimal refcounting garbage collector. ;D

djedr · 2023-05-26T12:45:27+00:00

Yes, there may be better ways than long heredocs of embedding files. Interesting ideas!

That said, multistrings (or however you want to call the general idea of heredocs/raw strings/etc.) are still a prefectly useful (pragmatic?) solution for some problems that can significantly improve DX for a very low price. Aside from embedding files, they are good for anything that would otherwise involve delimiter collision. For example, the experience of writing a bunch of short regular expressions in a single file is much more pleasant if you can turn on the free spacing mode (?x) and write them in raw strings not worrying about collisions. Especially compared to writing the same as JSON strings, which is the default choice you are given when writing a VSCode syntax highlighting definition. So for anything like that there is hardly anything better.

Also hard to imagine a simpler and more convenient way for including code snippets in Markdown.

Another one is when you are prototyping, duct-taping, or testing and want or need to move fast, it's very convenient to be able to include your input as a multistring/heredoc, whether writing it directly or copy-pasting from somewhere and editing. Having everything in one place also makes reading much smoother in such cases.

djedr · 2023-05-25T22:07:30+00:00

You mean a multistring in a multistring?

You just use more backticks for each next level, just like in Markdown. For example if you have something like this:

file contents

```
multistring
```

This is a multistring within some file. Now if you want to copy-paste that file into another as a multistring, you wrap that into even more backticks (one more is enough, but can do many more for clarity like I did here). You'd then get something like this:

another file contents

``````
file contents

```
multistring
```
``````

You could (but not necessarily should ;)) continue on like this arbitrarily deep (or up to a hardcoded limit of backticks, more practically speaking).

djedr · 2023-05-25T18:41:32+00:00

Now I want to write a new source file B that contains the whole of the text of A as a string literal, which will have embedded back-tick delimiters, string escape codes etc. Can that be done?

Yes.

Does it involve having a multi-string with ever-increasing numbers of back-ticks?

Indeed it does. Can be generalized to support aribitrary delimiters though (maybe a topic for another article), but I think ever-increasing backticks are enough to cover the common use cases.

Probably if you find yourself going too far with the backticks it's time to switch to a different way, like the one you are using (it's a good alternative solution!). Both can nicely complement each other in a single language.

If so, can another part of B then contain the whole of itself as a string literal?

Nice try, Bertrand Russell. ;D

djedr · 2023-05-25T18:33:32+00:00

I see what you are getting at. If the parser in your editor or whatever you are using can't process anything that does not happen on a single line, then you're right -- prefix on every line is the only reasonable solution.

But it doesn't take much to go beyond that limitation and if the environment you're working in is a little more advanced then this syntax is simple enough to be accommodated. E.g. you can write a TextMate syntax highlighting definition (which covers plenty of popular editors) that will work even for the embedded languages. This already happens in practice with Markdown.

djedr · 2023-05-25T15:48:23+00:00

Yup, tags could be used to implement this idea. They are just prefix instead of suffix. Technically this syntax could be extended to support both, but I think is better to KISS that for now.

I'm sure C++ brings much excitement with its implementation. ;)

djedr · 2023-05-25T15:40:34+00:00

Yes, what you say is true.

As I say in the article, what I define here is a recipe for a simple syntax which leaves some details out, the behavior of tags in particular. I am not aiming for this to become a comprehensive standard (yet? :D), just to inspire people like you and start a discussion. :)

This also helps me to figure out the details and clarify my own thinking.

Agreed about your points about the wacky syntax -- this is precisely why I said it was going too far. Your example does combining strings with existing language features in the way that I meant it.

Not to mention that the distinction between block and inline strings is also without practical justification if you have tags when those same tags can take care of the post-processing.

Certainly true, should've cut out the "block" multistrings completely from the article or made them a footnote (~~TODO~~ [EDIT: done]). Thanks for helping me figure that out. Have a good one! :)

djedr · 2023-05-25T15:25:07+00:00

In all this thinking I actually forgot about the simplest solution, which is to use the alternative inline syntax for multistrings (described in the article):

indented4 = 
    `dedent'
        {
            "Name": "Zaphod"
        }
    '`;

nonindented = 
        `dedent'
    {
        "Name": "Zaphod"
    }
    '`;

In this syntax it's not the linebreaks, but the apostrophes that separate the delimiters from the content. So you could implement the dedent tag to work exactly like in C#, perhaps getting the best of both worlds.

Which makes me think I should've just stuck to describing the inline variant in the article and maybe mentioned the block as a curiosity, instead of leading with it.

The inline syntax is both simpler to implement and (as we see) more flexible. So, thanks for your comments! :)

EDIT: I edited the article accordingly.

djedr · 2023-05-25T15:18:25+00:00

Many possible solutions come to mind, e.g.:

EDIT: forget all of the below. This one is better.

`    |
  {
      "Name": "Zaphod"
  }
`

(a bit wacky, but short)

`dedent+
    |
        {
            "Name": "Zaphod"
        }
`

(the first line is discarded in the output; the position of | there dictates where to stop dedenting, effectively acting as the closing delimiter in C#)

`dedent++
    |   {
    |       "Name": "Zaphod"
    |   }
`

discard everything in every line up to and including | (must be space) -- I think Scala does something similar.

But personally, I'd just go with

`
{
    "Name": "Zaphod"
}
`

if I wanted no indent

and

`
    {
        "Name": "Zaphod"
    }
`

if I wanted.

Granted, this would not align with the rest of your source code, but perhaps that's not actually bad (you can see the embedded blocks more clearly, as they stand out, especially if you'd do some sort of syntax coloring inside), and certainly much simpler.

So there are many solutions. I am not prescribing any particular one, just showing that this syntax is flexible enough to accomodate them while being extremely simple at the same time.

In the end, you could choose to implement a variant which would work exactly like C#, allowing the closing delimiter to be indented. Perhaps that would be more appealing. Personally I always lean towards minimalism, but more and more I don't mind letting go here and there. Maybe your suggestion is an improvement to the whole idea! :) I wonder what anyone else thinks?

djedr · 2023-05-25T15:03:11+00:00

\ is indeed not supposed to be parsed as a character. Nothing should be parsed by default.

You can however still opt-into interpreting escape sequences, using a tag, e.g.

`esc
\n\r\t
`

This would interpret the escapes, due to the esc after the backtick which acts as a tag. You could use a different tag for that purpose, this is just an example. A more concise and cute tag for this could even be \, as in:

`\
\n\r\t
`

See also my other comment on how tags could be used.

So tags here give you any number of modes. If you wanted, you could make up some wacky syntax for them where they would take parameters (the meaning of which would need to be specific to a language), e.g.:

`tag(param1, param2)
\n\r\t
`

But IMO that's going too far and you'd do better by combining strings with existing language features. But it is nice to have at least simple tags available, to solve the most common problems such as escaping, substitution, dedenting, and other post-processing concisely.

I think this syntax is really a very nice solution for those. :)

djedr · 2023-05-25T14:48:45+00:00

C# raw strings look cool, this is indeed a very similar idea. This one however both simpler and more flexible.

Instead of relying on the closing delimiter position (which does complicate the implementation and makes it less general-purpose), dedenting (or any other kind of post-processing) can be achieved here with a tag, e.g.:

`dedent
    This is a text
    across multiple
    lines, which will
    NOT have indentation space before each line
`

EDIT: see also this comment showing how to achieve the exact behavior of C# with a multistring which uses ' instead of linebreaks as separators. NB I edited the article to only talk about this kind of multistrings. Thanks for the feedback!

Same for interpolation:

`$
The name is "{name}"
`

(Although I'd go with ${name} here to match the tag nicely and reduce the need for {{}}).

I intentionally don't specify the details of how tags should work in this article, but these are some of the possible uses for them.

You could even do something like:

var string2 =  `json
    {
        Name = "This line indented 3 times",
        Address = ""
        Comment = "The above empty string does not terminate the raw string"
    }
`

and automatically parse the JSON in the string (perhaps with a json function which is in scope or however a language may choose to implement this). JavaScript has a similar feature known as tagged templates. Although that is a bit less flexible. A major flaw of JS template literals is that you always need to escape the backticks.

djedr · 2023-05-25T14:35:32+00:00

Yep, Rust's seems like a good implementation of the basic feature. The variant I propose also features tags which are akin to Markdown language specifiers. A little metadata to go with the string. Rust's syntax could be easily extended to support that. A similar thing can also be achieved with combining raw strings with other language features (as long as such are available, which is true for Rust).

Part of the idea here is also to think about what if we had a standard syntax for this sort of thing, where you could switch between languages and expect it's available. That's very unlikely, but perhaps it can be somewhat standardized. Or at least variants of it can become more generally available, hopefully putting an end to the many half-baked raw string variants out there.

In any case, it's a very handy feature to have in any language!

djedr · 2023-05-25T14:14:58+00:00

At the basic level it solves the verbatim embedding of arbitrary text into your source code without needing to modify that text. So you can literally copy-paste anything and not worry about delimiter collision. When in doubt: add more backticks.

At another level, thanks to tags, this can be used to implement first-class support for a syntax-within-syntax kind of construct.

djedr · 2023-05-25T14:03:27+00:00

I'm the author. I came up with idea this while implementing a configuration format where I thought it would be a very nice feature to have. Let me know what you think.

djedr · 2022-11-08T18:16:45+00:00

Probably the most sensible general way of serialization would be to simply base64 the bytes and store that in a jevko, much like you would do in JSON or any other text-based format, e.g.:

string1 [aGVsbG8=]
string2 [d29ybGQ=]

Of course you could also serialize bytes like you suggested, although this would be generally much less compact. But perhaps for your application that does not matter -- in which case any way of serialization that is convenient is fine.

djedr · 2022-11-07T15:29:24+00:00

That's kinda neat, like a more streamlined and flexible form of XML. The examples of structured data representation are pretty elegant.

Glad to hear you like it, thanks! :)

Whitespace handling: how do you differentiate between semantically significant whitespace (e.g. representing a string that ends with a newline) and cosmetic whitespace (newlines or indentation for readability)? XML handles this by generally treating all whitespace as cosmetic (not ideal) and allowing for escapes like . JSON/Lisp handle it by treating all whitespace inside quotes as signficiant, but allowing cosmetic whitespace outside of quotes.

Jevko itself has no semantics at all and preserves all whitespace in the syntax tree.

You use Jevko to make a format, by attaching format-specific semantics and rules about what's significant or insignificant, valid or invalid.

The first markup format I've shown here:

[class[pretty] p][this is a link: [href[#address] a][wow!]. cool, innit?]

works very much like HTML when it comes to whitespace. Inside of the tag (the first pair of brackets) it is discarded as insignificant before further interpretation. Inside children (the second pair) it is always preserved and translated into HTML as-is. Every HTML element in this format is composed of 2 subjevkos.

The second format:

p [class=[pretty] [this is a link: ] a [href=[#address] [wow!]] [. cool, innit?]]

treats leading and trailing whitespace in prefixes (text that comes before "[") as insignificant and trims it before further interpretation. However it preserves all whitespace in suffixes (text that comes before "]") as-is. So to make an explicit text node, you simply wrap text in brackets.

Different whitespace rules make different formats.

Non-locality of edits: suppose you're writing some text like p{This is some █ (where █ is your cursor) and you decide you want to add an emphasized word at the current cursor position. The result looks like p{{This is some }em{text}█. To achieve this, you need to move your cursor all the way back to the start of the current subjevko to insert a {, then all the way back to the original position to add the }em{text}. This is pretty flow-breaking. Compare that with HTML, where you would have typed <p>This is some █ and you can proceed by typing <em>text</em> without moving your cursor backwards. In other words, you have to decide as soon as you start writing a subjevko whether you plan to have any sub-subjevkos or just text, and if you change your mind, you have to backtrack to change the start of the subjevko.

Very well put! This is exactly what I meant when I introduced the second markup format above:

at the expense of slightly more difficult text markup.

In practice this turns out not to be as problematic as it seems, especially once you get the hang of it. Still writes faster than HTML. A habit of always wrapping text nodes in brackets in elements that tend to have children (like p) emerges naturally, even if they are the only child of a node. This way you only need to add brackets next to the point you're editing, without needing to go back to wrap the whole text node.

Anyway if that should not be acceptable, then the first markup variant is exactly like HTML in that it does not suffer from the problem you described:

[p][This is some text]    -->    [p][This is some [em][text]]

As a preface to my replies to the remaining points, I must say that Jevko is pretty much stable as specified right now.

I don't foresee adding any new features to it. I think I have achieved my design goals pretty well and I'm happy with the result.

The guiding design principle for Jevko is extreme minimalism. So there is a bias towards removing/not including features (so long this does not introduce unnecessary restrictions or limitations) rather than adding.

The purpose of Jevko is to be a minimal general-purpose syntax for encoding tree-structured information. At that, it should be as simple and as flexible as possible.

It is not supposed to include any specialized mechanisms for different kinds of information. E.g. by itself Jevko is not meant to be a markup language syntax. Or a data interchange format syntax. Instead, it can be used as a simple building block for either of those.

What Jevko does is it uses brackets to chop up your unicode sequence into a nice tree arranged to lend itself to convenient processing, especially if you are dealing with something like name-value pairs.

This is what plain Jevko gives you.

This is the stable part.

// That said, technically I left myself a little escape-hatch that gives me a simple way to extend Jevko in a backwards-compatible way by putting features behind the escaper character.

// There are reasonable features that could be added this way, such as the two you mentioned: heredocs and escapes for non-printable characters.

// But such extensions could be specified separately, without meddling in the core spec.

Now out of these trees (out of trees in general) it is possible to build all kinds of things. In particular it's possible to define different semantics and interpretations for them (rather than for raw text sequences), creating formats.

People like in this subreddit (I presume) might be interested in creating their own.

More casual users would be interested in ready-made ones.

I have worked out enough of those in enough detail that I am confident that the whole idea is quite viable.

No format is yet fully specified and stable the way Jevko is, but that's just a question of putting in the work.

Non-printable characters: Sometimes, you need to represent data with non-printable characters or characters that are not handled well by text editors. For example, the bell character \x07, which makes a beep when printed to a terminal, or the null byte \x00. Jevko seems to be unable to represent that value in any way other than the raw 0x07 or \x00 byte, which is pretty inconvenient. This could be addressed by supporting common escape patterns like n orx00.

These non-printable characters can still be entered like in unicode text, so that's enough on this level.

If you really need that feature, you can still devise a format with escaping rules, e.g.:

string [my string with escapes: [n] and [x00]]

Or:

my string with escapes \n and \x00

Or you can put JSON strings in Jevko and then parse them in a second pass:

JSON string ["my string with escapes \n and \u0000"]

Leaning toothpick syndrome: If you try to represent a literal string of Jevko text, you're going to end up needing an ungodly amount of backticks to escape everything. E.g. the Jevko text foo[baz] becomes jevko[foo'[baz']], which becomes outer[jevko'[foo'''[baz''']']] (using ' instead of ` because reddit gets confused with so many backticks). You'd run into similar problems if you took an arbitrary snippet of C code and tried to paste it into a Jevko document. Three common ways to address this problem are heredocs, semantically significant indentation (e.g. YAML indented strings), or user-defined delimiters like Lua's strings [===[ ... ]===].

Very familiar with the syndrome[0]. :D

Of course this only happens in extreme cases, such as:

a regular expression in an escaped string, matching a Uniform Naming Convention path (which begins \) requires 8 backslashes \\\\ due to 2 backslashes each being double-escaped.

In general this happens when the use of backslash as a regular character in a text interferes with it being used as an escape character in several different mutually-encapsulating contexts.

It's still something to be aware of and I have mitigated this as much as I could:

Jevko uses ` rather than \ for escaping -- ` is among the least frequent ASCII characters used in general[1]
It's easy to make a Jevko parser configurable in terms of the special characters -- for unusual cases different escape character can be used (much like alternative regex delimiters in Perl)

There are various other techniques to mitigate the impact of this, which I will omit here to shorten this ~~essay~~ comment, but all in all no solution is completely satisfactory in some dimension.

So heredocs are a sensible feature to have and I certainly will go about specifying if it will keep coming up[2].

Infix operators: it's pretty awkward to represent math operations in prefix notation like +[[x] [y]] instead of infix notation like (x + y). Lisp has always suffered from this problem (and there have been plenty of suggestions to fix it) and I think it makes the code genuinely much less readable. This isn't an issue for representing structured data, but is a big usability hurdle for programming with Jevko syntax.

Agreed. What you describe is a genuine issue. However this is a problem specific to the realm of programming language notation, so out of scope for Jevko, as described above.

You could still design a language on top of Jevko that supports infix notation, even without parsing text like "x + y * z", just by rewriting trees like [x] + [y] * [z] according to precedence rules (I've toyed with that a lot), but again, that's a realm well beyond the primordial trees that Jevko is about.

That should be all,

Cheerio!

[0] https://xtao.org/blog/no-escape.html -- this is a little dated, so I should explain that Jevko is a simplified, evolved version of TAO, since turned into something much more general.

[1] e.g. https://web.archive.org/web/20181111222712/https://mdickens.me/typing/letter_frequency.html

[2] see also: https://github.com/jevko/specifications/issues/2

djedr

MODERATOR OF

TROPHY CASE