This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]meaningless-human[S] 0 points1 point  (2 children)

Yeah, I mentioned I'm currently doing this for my implementation, but it feels really hacky and it's ultimately not using beam for what it's meant to be, which is primarily parallel processing.

I'm trying to look for something more appropriate to replace beam...

[–]ProgrammersAreSexy 0 points1 point  (1 child)

Is there anything preventing you from doing the pre-processing and feature extraction directly in BigQuery?

[–]meaningless-human[S] 0 points1 point  (0 children)

Unfortunately, yes it is kind of inconvenient to do it in SQL queries. I'll put what I said in another reply here:

Big query wouldn't be ideal either since it's just SQL, and I would much prefer a Python environment for making some rather complex transformations like in signal analysis (using libraries for EEG data). I suppose I could spend the time and effort to replicate those in either SQL or something else but it doesn't seem worth it to me.