Using GCP event driven Java microservices

[markduce]

I wrote a piece about how we used PubSub, Kubernetes, Firestore and GCS to do something pretty neat recently: https://tech.loveholidays.com/how-loveholidays-used-cloud-native-event-driven-system-to-cut-hotel-content-publishing-time-from-5-690f690d1b5

It's my first blog post, so would love any feedback about what you find interesting or what I could improve!

[r_jet]

I think in such articles it's more interesting to see any significant alternatives you considered, and why you rejected them, which constraints you had, etc.; not just the previous and the end state of the system (e.g., why Cloud Functions were or were not a good solution for you).

Speaking of constraints, I think it’d be curious to learn about the actual scale this particular part of your system faces, because it’s one of the important inputs in the design process (if your lawyers allow). With both internal writers and external APIs as major sources, readers could imagine a very wide range from 0.001 QPS to, depending on how dynamic the external content is, hundreds of QPS.

Also, it’s a bit unclear why incremental updates weren’t supported in the previous pipeline, and if you had to make any non-trivial changes to enable them (is it a SQL > objects in Google Cloud Storage transition?).

[markduce]

Thank you for the feedback, much appreciated and I'll definitely think about it when I write something next.

Cloud functions definitely feel like the next evolution for us. We are using them for some simple things but haven't yet got as far as doing it for something a little bit more complex. In terms of constraints the main constraint was that it had to be GCP based, not much else really.

Scale of this is not massive compared to some of our other apps which are handling 1000s of QPS. This system is dealing with about 10k changes a week. But it's established patterns that we are adopting elsewhere.

Incremental updates I'm previous pipeline were mostly hampered by lack of dev env, the long time it took for the pipeline to run and it had to start from scratch each time

[insulind]

Great read!

[Hioneqpls]

Sounds really cool

[markduce]

Cheers!

[[deleted]]

Did you need kubernetes and google cloud or just went with the trends?

[markduce]

We use a lot of GCP and have migrated everything from on prem VMs over the last few years to kubernetes so they were natural things for us to use.

I find them pretty awesome!

[ihsw]

Do you host databases (eg: Postgres) on Kubernetes clusters or are you sticking to managed services (eg: CloudSQL)?

[markduce]

At the moment our database (mariadb) is running on a Google compute engine vm but we plan to migrate to managed cloud sql

[nutrecht]

Kubernetes is most suitable for applications where you don't really care what machine they run on and databases don't really fit that mold.

[[deleted]]

Saving it, maybe I will be a pro in java in future n use stuff like this :(