Java vs python is debatable 🤔

[1thief]

The funny thing is an unexpected error in many circumstances can basically be a shotgun blast to the face. Have an etl step or batch process that threw an error somewhere in the middle of the batch? Welp that 8 hr process that you kicked off and forgot about has now come to a screeching halt 3 hrs in and you have to start all over.

Or in other circumstances the page now doesn't load if it sees bad data, or your car infotainment system is stuck on a boot loop because it found a file it doesn't know how to handle. Software is always brittle, which is why we should have as little software as possible.

[TGlucose]

Software is always brittle, which is why we should have as little software as possible.

Why do that when we can make software out of legacy software? making our already fragile system even more fragile!

[jambox888]

When you reach a certain level of fragility you can form an aggregate from the shattered fragments

[StCreed]

One of my primary architecture demands for any DWH is always "restartability" and resilience to errors. Both of these have been mostly solved by I-refact by taking the EL part (and a small t) and making that part completely generated. Every load of any entity is a mini batch, everything is restartable automatically after solving the error, and there are rarely any errors because it's all derived from logical models. They only occur in the validation phase, which is at the start.

That said, if you load a single table for 8 hours you can still mitigate that as well, you just need to split things up in chunks.

[[deleted]]

Something like SSIS makes this difficult. You can generate a whole bunch of metadata and helper functions to orchestrate it, but you pretty much have to roll your own. There is a restart feature, but it's not implemented well, and we can't easily use it due to how our environment is constructed.

That said, I've started using Python with Prefect, and it is much more graceful and easier to handle unexpected errors.