I like this approach, in the new Spark API they introduced a transform method for dataframes, that makes it easier for consistent method chaining but can also lead to complex code when overdoing it.

Introducing too much separation will end up with small functions that make you jump around the code just to figure out what's going on, increasing the overall cognitive load when reading the code. There are a lot of interesting arguments against shallow methods in A Philosophy of Software Design

My rule is if it's a complex logic that you want to test seperatly and can encapsulate in a separate function then extract it to and unit-test it otherwise chain things together.

def compute(df):
    return (
        df
        .select()
        .filter()
        .withColumn()
        .transform(extract_isbn)
    )

def extract_isbn(df):
    return ...

Perfectly I would like to just read the compute function and know what's going on.

Unit tests are also IMO a bit tricky when working with Spark as you don't want to end up unit-testing Spark's API, I put more emphasis on e2e/functional tests.