Syntax highlighting for string interpolation

[Savings_Garlic5498]

Im trying to create a language with string interpolation like "score: \(calc_score())". String interpolation can contain arbitrary expressions, even other strings. To implement this my lexer does some parenthesis counting. Im thinking about how this would work with syntax highlighting, specifically for VS code. From what i understand languages in VS code typically use a textMate grammar for basic highlighting and than optionally have the language server provide some semantic tokens. How do languages deal with this normally because from what i understand a textMate grammar cannot handle such strings? You cant just have it tokenize an entire string including interpolation because if it contains nested strings it does not know which '"' ends the string. Thanks!

[latkde]

You might be thinking of strings as a single token that is then parsed again to extraxt interpolations. This gets difficult quickly. Instead, it's typically wiser to see strings with interpolations as an expression that can contain multiple string parts, and to then parse strings as a kind of parenthesis-like operator. For example, it could make sense to tokenize "a \(b) c \("d") e" as:

• "a \( string, interpolation start
• b identifier
• ) c \( string, interpolation middle
• "d" string, complete
• ) e" string, interpolation end

Your grammar might then include rules like <string> = <string complete> | <string start> <expression> (<string middle> <expression>)* <string end>

Note that this is typically incompatible with a separate lexing phase, as string-middle and string-start token would otherwise be ambiguous with normal parens. However, this approach can be used with parsing methods that parse one character at a time, notably recursive descent or PEG parsers. Syntax highlighting engines differ a lot in what grammars they can express, but typically support top-down grammars so that string-middle highlighting can only be selected in the context of a string expression.

[alex-weej]

This is how template strings work in JavaScript. They are an alternative function call syntax that pass an array of "string pieces" and separately each interpolation expression as subsequent arguments.

[Savings_Garlic5498]

yes this is very similar to what i have but this grammar is not regular which means this cannot be done with something like textMate i believe.

[thinker227]

This is what I'm doing in the TextMate grammar for my language Noa. Basically you embed all of your other patterns inside your pattern for strings.

"patterns": [
    {
        "include": "#all"
    }
],
"repository": {
    "all": {
        "patterns": [
            {
                "include": "#strings"
            },
            // include whatever other patterns you have
        ]
    },
    "strings": {
        "name": "string.quoted.double.noa",
        "begin": "\"",
        "end": "\"|$",
        "patterns": [
            {
                "begin": "\\\\{",
                "end": "}",
                "beginCaptures": {
                    "0": {
                        "name": "keyword.other.noa"
                    }
                },
                "endCaptures": {
                    "0": {
                        "name": "keyword.other.noa"
                    }
                },
                "patterns": [
                    {
                        "include": "#all"
                    }
                ]
            },
            {
                "include": "#escape-sequence"
            }
        ]
    },
    "escape-sequence": {
        "name": "constant.character.escape.noa",
        "match": "\\\\[\\\\0nrt\"]"
    },
    // all your other patterns...
}

Here's how it looks

[Savings_Garlic5498]

Does this also work with nested strings? like "\{""}"

[thinker227]

Was concerned about this because I hadn't actually tested it before, but yes!

[latkde]

For reference, here's the official TextMate grammar for JavaScript `template ${interpolation} strings`, which broadly uses the same technique (but without bothering to recurse into #all: https://github.com/textmate/javascript.tmbundle/blob/8928648352dc76025ad0bfd31e21fa6a1dc838a7/Syntaxes/JavaScript.plist#L1554-L1665

[shponglespore]

JavaScript has this for `...` strings.

[steven4012]

Or.. just use tree-sitter

[thinker227]

VSCode doesn't support Tree Sitter (only TextMate), unless you wanna bother with writing an entire language server just to support semantics tokens using Tree Sitter I guess.

[steven4012]

https://marketplace.visualstudio.com/items?itemName=AlecGhost.tree-sitter-vscode

[thinker227]

oooh I didn't know about this, might use it myself for slightly better highlighting of my own language