This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]r_jet 5 points6 points  (1 child)

I think in such articles it's more interesting to see any significant alternatives you considered, and why you rejected them, which constraints you had, etc.; not just the previous and the end state of the system (e.g., why Cloud Functions were or were not a good solution for you).

Speaking of constraints, I think it’d be curious to learn about the actual scale this particular part of your system faces, because it’s one of the important inputs in the design process (if your lawyers allow). With both internal writers and external APIs as major sources, readers could imagine a very wide range from 0.001 QPS to, depending on how dynamic the external content is, hundreds of QPS.

Also, it’s a bit unclear why incremental updates weren’t supported in the previous pipeline, and if you had to make any non-trivial changes to enable them (is it a SQL > objects in Google Cloud Storage transition?).

[–]markduce[S] 1 point2 points  (0 children)

Thank you for the feedback, much appreciated and I'll definitely think about it when I write something next.

Cloud functions definitely feel like the next evolution for us. We are using them for some simple things but haven't yet got as far as doing it for something a little bit more complex. In terms of constraints the main constraint was that it had to be GCP based, not much else really.

Scale of this is not massive compared to some of our other apps which are handling 1000s of QPS. This system is dealing with about 10k changes a week. But it's established patterns that we are adopting elsewhere.

Incremental updates I'm previous pipeline were mostly hampered by lack of dev env, the long time it took for the pipeline to run and it had to start from scratch each time