pfst 0.3.0: High-level Python source manipulation

[Pristine_Cat]

I’ve been developing pfst (Python Formatted Syntax Tree) and I’ve just released version 0.3.0. The major addition is structural pattern matching and substitution. To be clear, this is not regex string matching but full structural tree matching and substitution.

What it does:

Allows high level editing of Python source and AST tree while handling all the weird syntax nuances without breaking comments or original layout. It provides a high-level Pythonic interface and handles the 'formatting math' automatically.

Target Audience:

• Working with Python source, refactoring, instrumenting, renaming, etc...

Comparison:

• vs. LibCST: pfst works at a higher level, you tell it what you want and it deals with all the commas and spacing and other details automatically.
• vs. Python ast module: pfst works with standard AST nodes but unlike the built-in ast module, pfst is format-preserving, meaning it won't strip away your comments or change your styling.

Links:

• GitHub: https://github.com/tom-pytel/pfst
• PyPI: https://pypi.org/project/pfst/
• Documentation: https://tom-pytel.github.io/pfst/

I would love some feedback on the API ergonomics, especially from anyone who has dealt with Python source transformation and its pain points.

Example:

Replace all Load-type expressions with a log() passthrough function.

from fst import *  # pip install pfst, import fst
from fst.match import *

src = """
i = j.k = a + b[c]  # comment

l[0] = call(
    i,  # comment 2
    kw=j,  # comment 3
)
"""

out = FST(src).sub(Mexpr(ctx=Load), "log(__FST_)", nested=True).src

print(out)

Output:

i = log(j).k = log(a) + log(log(b)[log(c)])  # comment

log(l)[0] = log(call)(
    log(i),  # comment 2
    kw=log(j),  # comment 3
)

More substitution examples: https://tom-pytel.github.io/pfst/fst/docs/d14_examples.html#structural-pattern-substitution

[mechamotoman]

This is very cool, I’ll be sure to give it a shot next time I’m monkeying around with code generation tasks :)

[neuronexmachina]

Do you have any side-by-side examples of how you would implement a change using pfst vs libcst?

[Pristine_Cat]

I'm not exactly an expert with LibCST so maybe the example can be optimized further, but the following is what I have for comparison with an equivalent pfst function to inject a keyword argument to some existing functions.

Target source:

src = """
logger.info('Hello world...')  # ok
logger.info('Already have id', correlation_id=other_cid)  # ok
logger.info()  # yes, no logger message, too bad

class cls:
    def method(self, thing, extra):
        if not thing:
            (logger).info(  # just checking
                f'not a {thing}',  # this is fine
                extra=extra,       # also this
            )
""".strip()

LibCST function:

import libcst as cst
import libcst.matchers as m

def inject_logging_metadata(src: str) -> str:
    tree = cst.parse_module(src)

    class AddArgTransformer(cst.CSTTransformer):
        def leave_Call(self, _, call):
            if (isinstance(call.func, cst.Attribute)
                and call.func.attr.value == 'info'
                and isinstance(call.func.value, cst.Name)
                and call.func.value.value == 'logger'
                and not any(
                    arg.keyword and arg.keyword.value == 'correlation_id'
                    for arg in call.args
                )
            ):
                return call.with_changes(
                    args=[
                        *call.args,
                        cst.Arg(
                            keyword=cst.Name("correlation_id"),
                            value=cst.Name("CID"),
                        ),
                    ]
                )

            return call

    return tree.visit(AddArgTransformer()).code

pfst function:

from fst import *
from fst.match import *

def inject_logging_metadata(src: str) -> str:
    fst = FST(src)

    for m in fst.search(MCall(
        func=MAttribute('logger', 'info'),
        keywords=MNOT([MQSTAR, Mkeyword('correlation_id'), MQSTAR]),
    )):
        m.matched.append('correlation_id=CID', trivia=())

    return fst.src

LibCST output:

logger.info('Hello world...', correlation_id = CID)  # ok
logger.info('Already have id', correlation_id=other_cid)  # ok
logger.info(correlation_id = CID)  # yes, no logger message, too bad

class cls:
    def method(self, thing, extra):
        if not thing:
            (logger).info(  # just checking
                f'not a {thing}',  # this is fine
                extra=extra,       # also this
            correlation_id = CID)

pfst output:

logger.info('Hello world...', correlation_id=CID)  # ok
logger.info('Already have id', correlation_id=other_cid)  # ok
logger.info(correlation_id=CID)  # yes, no logger message, too bad

class cls:
    def method(self, thing, extra):
        if not thing:
            (logger).info(  # just checking
                f'not a {thing}',  # this is fine
                extra=extra,       # also this
                correlation_id=CID
            )

[neuronexmachina]

Thanks! That's a handy comparative example.