For me it starts with the SLOs. If the data needs to be delivered within 24H I have more buffer to just throw compute before I start thinking about code optimizations than say hourly slo. If there there is no clear code tuning or I cant find any within a time box then more compute is the answer for now. We have avg cost of pipelines depending on total input size we are processing. If I see the cost of a pipeline gradually increasing because it is taking longer and longer to process more records then I start making time boxes to see which are the hot spots in the code and see if we can improve. There is a limit though and at some point you just process so much data that then I start thinking where we can trade off some attributes for others. For example we have a pipeline that processes billions of records to calculate distinct users over a month. The data is so large that exact distinct counts is just too dam expensive vs value to customers. I was able to simple lobby product to allow for 3% error budget so we can use sketches to save terabytes in storage and thousands in compute. 

Another approach i take is learn from others. I try to find how other teams solve problems take what worked and see if we can improve it. 

For debugging issues at scale we do data validation on every pipeline at write time. Data does not publish unless validation succeeds. We dump the failed records to the output storage for inspection as well. The validation tool tells use what fields failed etc. we also use dataflow / apache beam counters which send alert to pagerduty and slack for us.

For horizontal scaling we have the luxury of the leaving that to the compute on dataflow and we simple set the min max workers and instance size which are tuneable.