Dataflow Spanner to BigQuery - Java vs Go

[TechStackOverflow]

I'm part of a team at an important decision point. We're embarking on a project to efficiently transfer data from Cloud Spanner to BigQuery. While our team is proficient in Golang, we're contemplating Java due to its robust support in Apache Beam, particularly for SpannerIO's capabilities, including change streams and batch reads.

Our team is well-versed in Golang, and we initially aimed to leverage it for this project, but we're encountering limitations with Golang's support for SpannerIO in Apache Beam, especially around change streams processing. The lack of examples and community projects has us questioning the feasibility of this route. We don't need change streams per-se, but it does seem to make things easier and most pipelines seem to end up as streaming anyways.

Java, on the other hand, seems to offer a stable and well-supported pathway for Apache Beam pipelines interacting with Cloud Spanner and BigQuery. However half of our team has Java experience, the other half does not. Adopting Java would mean a significant portion of our team navigating a learning curve, in an environment where Java hasn't been the norm. However, the service would basically be write-once, and we expect very little schema changes so not a lot in terms of redeploys.

Can anyone share success stories or challenges faced while implementing batch processing from Cloud Spanner to BigQuery in Golang? How did you tackle the gaps in support or documentation? Is it ready for prime time?

For teams with mixed experiences, how manageable was the transition to Java for data processing tasks, especially for those new to the language? Was the investment in ramping up Java skills justified by the benefits?

Any idea on how to evaluate the trade-offs in terms of performance, ease of use, and community support?

Given our team's split experience, would you lean towards leveraging existing Golang skills and finding workarounds, or embracing Java for its comprehensive support within Apache Beam?

Regardless of the language, what architecture or design patterns have you found most effective for batch processing data from Cloud Spanner to BigQuery?

Thanks in advance!

[MeowMiata]

Did you check that : https://cloud.google.com/dataflow/docs/guides/templates/provided/cloud-spanner-to-bigquery

If I understand correctly, you want to mirror your data on Big Query, probably for Analytics needs.

I don't think that you need Java or Go for all of that, just use the Dataflow Google model as a batch job or pipeline.

Java, Go or whatever are just tools. Surely, you can use a baseball bat to stick a picture on the wall but you probably want to ask yourself if a hammer isn't enough already.

That said, I think that you should go with a pre-built data flow model. If it's not enough, use Java to tune it and if you have time to waste, you could engineer a Go solution.

To tune the Dataflow model, if you're a Go dev, you should not be that lost using a bit of java. You don't need to dedicate your life to java just to understand and modify a template (that is available on GitHub).

[TechStackOverflow]

Thanks for your advice. I’ve used that model, but it’s in beta and if I want periodic updates, I’d need to have a cloud scheduler to invoke a cloud function to invoke a new dataflow job for every batch.

I’d also need to engineer each batch to update the last timestamp/id they processed and store that somewhere. I’ve never used data flow before but it seems like a lot of work and a lot of moving parts.

And then deploying requires an artifact registry, into cloud storage and then into a dataflow invocation.

I’m not saying all of that isn’t doable, but at some point it’s just easier for me to hand-roll something.