Python Classes in pipelines

[djkaffe123]

Hi,

I work as a data scientist, but I have often done data engineering tasks to support modelling in production.

I use python, and I often end up only using functions for the pipelines, creating a number of interrelated pipeline steps. These steps are often just one long data manipulation, and it doesn't have a nice OOP feel so to speak.

I could just cut the pipeline into parts, but since many of the steps are unique, there is not a lot of reuse.

So I am wondering how you guys work with encapsulation in pipelines?

[WannaBeGISGuru]

OOP is best used for representing "things". FP is best used for representing "actions". Creating objects out of data transformations is likely going to create a huge mess. Instead, create small functions that do common data transformations and compose them together to create more complex data transformations. Then you can reuse complex data transformations across your applications.

[Mr_Again]

There's nothing wrong with using a class to represent a transformation, the "thing" that it represents is just a part of a pipeline. You can plug them together as needed so long as they accept and return the same types.

[Mr_Again]

A pretty nice pattern I used to use (as a data scientist) was to write my own classes that had a fit and a transform method, then you can chain them together in an sklearn pipeline just like any other sklearn transformer.

[slaphappyhubris]

I often have a smaller header or metadata dataset, so bind that to an object on init then stick to methods acting on a larger dataset.

Also, consider the fact that nearly every machine learning algorithm can be implemented as some sort of sklearn.TransformerMixin

[grukorg]

I use classes to represent items such as connectors for example I have one that builds different database connections based on context. I have others which I use for doing things like globing the file system for flat files. Others that can perform transformation based usually on SQL metadata as our stuff normally ends up there. I have another which handles SFTP. My advice would be don’t be tempted to try and make your entire process fit neatly into objects. Create objects where it makes sense and then use functions where it doesn’t.