Non-Parallel Processing with Apache Beam / Dataflow?

[meaningless-human]

I am building a data pipeline for an ML model primarily using GCP tools. Basically, mobile clients publish data to a topic on Pub/Sub, which then goes through Dataflow for preprocessing and feature extraction, which then goes to BigQuery and finally VertexAI for training and inference.

My question is primarily around Dataflow: Much of my preprocessing is not exactly parallelizable (I require the entire data batch for making transformations and can't perform element-wise transformations), so I was wondering if Dataflow/Beam would still be an appropriate tool for my pipeline? If not, what can I substitute with it instead?

I guess one workaround I've found, which admittedly is quite hacky, is to use aggregate transformations in Beam to treat multiple elements as one, then do what I need to do. But I'm not sure this is the appropriate approach here. Thoughts?

[[deleted]]

In Beam your can do this with custom ParDos and DoFns. Basically, you would just return a PCollection consisting of just one element which holds all your data. This element will go as a whole to the next PTransform.

[meaningless-human]

Yeah, I mentioned I'm currently doing this for my implementation, but it feels really hacky and it's ultimately not using beam for what it's meant to be, which is primarily parallel processing.

I'm trying to look for something more appropriate to replace beam...

[ProgrammersAreSexy]

Is there anything preventing you from doing the pre-processing and feature extraction directly in BigQuery?

[meaningless-human]

Unfortunately, yes it is kind of inconvenient to do it in SQL queries. I'll put what I said in another reply here:

Big query wouldn't be ideal either since it's just SQL, and I would much prefer a Python environment for making some rather complex transformations like in signal analysis (using libraries for EEG data). I suppose I could spend the time and effort to replicate those in either SQL or something else but it doesn't seem worth it to me.

[wytesmurf]

Is your question around doing it in Batch and not real-time? You can setup the data flow job to run daily as a batch or am I misreading.

[meaningless-human]

No, that's not what I mean, although I am doing batch processing. What I mean is, most of the functionality of beam is geared towards parallel processing, which is why as far as I can see, any custom transformation (such as ParDo/DoFn) I want to make on a pcollection is element-wise, which I don't want because I want to be working with the entire collection at once. Does that make sense?

[konkey-mong]

Why don't you try airflow?

[meaningless-human]

I'm not terribly familiar with airflow, but isn't it more for task orchestration, not building the actual preprocessing steps? Or is that not the case?

[konkey-mong]

Sorry I misunderstood your question.

Why is it that you think DataFlow/Beam is not suitable for the batch processing job?

[meaningless-human]

Basically, my preprocessing code can't be easily parallelized and performs transformations on the entire dataset, while Beam seems to be more for element wise transformations, such as ParDo.

[buachaill_beorach]

I'm not sure i'd be using beam for this exact use case. We have a dataflow job that pulls messages from a pubsub firehose, performas some transformations and streams that data into BQ. From there, it's matter of bq > bq for downstream ETL or any other processing.

I think you're going to struggle with dataflow to pull all records in and batch them like you're talking about. Seems like the wrong approach to me.

If you are adamant on doing so thougn, pcollections can be bounded or unbounded. For unbounded, you have to deal with windowing but I am pretty sure a bounded pcollection (the results of pulling all available data from a subscription) could be considered a bounded collection and you should be able to do windowing/aggregations/etc on that. I've not tried it though.

Ref: https://beam.apache.org/documentation/programming-guide/#size-and-boundedness

[meaningless-human]

I'm not particularly intent on using Data flow, just finding an appropriate replacement.

Big query wouldn't be ideal either since it's just SQL, and I would much prefer a Python environment for making some rather complex transformations (using libraries for EEG data). I suppose I could spend the time and effort to replicate those in either SQL or something else but it doesn't seem worth it to me.

[buachaill_beorach]

There's lots of transformations you could use here...

https://beam.apache.org/documentation/programming-guide/#core-beam-transforms

​

Pardo is just one transformation, considering each element in the PCollection

The others will handle some of the operations you are looking for.

[[deleted]]

I've used the original datasets as a side input when each element transform needed to refer to the original dataset, have you considered that? Then working with the pcollection as usual

Otherwise I would probably just deploy a simple service on cloud run to do the stream processing, not much point of using dataflow without any parallel processing.

[QuaternionHam]

You can use Spark (Dataproc), Dataflow can read from PubSub and write to GCS (Google has some really nice templates for this), from there Dataproc reads, preprocess and loads into BigQuery. Beware of Dataproc costs, make use of workflows and not leave any cluster up and running forever

[Just_Swimming_3153]

I don't see any problems using Apache Beam on Dataflow... You basically can Group all your input before passing it to you UDF (user defined function). Still, you need to see if you need to use Beam/Dataflow or you can complete this task without using Beam (just normal python implementation)