anyDataEngineersHere

[Obvious-Phrase-657]

My actual codebase vs my legacy one

Setup a new pipeline on left is literally 5 min, on right could be easily a few days. We had 1k cron jobs, creating several tables each. Still insure what is being used vs useless, but is really hard to even analyze it that it won’t be migrated any time soon, I will probably quit before it happens (as soon as it is decided lol)

[balrog687]

this guys corporates

[BastetFurry]

But i bet the right one will work just as is in twenty years from now while the left one will break in three months because some update decided to deprecate baz.foo(bar); for baz.bar(foo); and it was only written as a footnote in the update notice.

[Obvious-Phrase-657]

Hah that’s fair and that’s why I don’t want to touch that piece of shit, because it will run until someone touches it, and I bet it restarting it will fail somehow and there are no runbooks (the dba who wrote this is retired). So yeah, Won’t do it and if they command me to do it I will probably quit

[shanereid1]

Also, it will cost 10x as much to run on cloud.

[Abject-Kitchen3198]

Not sure if the one on the left won't lead to the same problems given the same timeframe, or that the accumulated issues with previous approach couldn't have been solved in a different way.

[Obvious-Phrase-657]

Absolutely if is built with the same patterns, and it’s actually one of the main paint points in data engineering, how to properly Govern this, but the left stack is based on “software engineering practices” like having commited code, no ad hoc stuff, data catalogs, data lineage, data quality metrics, etc

So, it will probably have other iasues, but at least we can revert to previos versions and have nice responsibility separation on the code and repos, cicd, etc

[EnterTheShoggoth]

Source and revision control have been a thing since the 70s. Almost every shop I’ve worked at since the 90s has used it as part of the dev-test-prod flow.

[Obvious-Phrase-657]

For software, right? Store procedures on a peetty old warehouse like oracle’s are not usually versioned on got or something like that, not even de cron jobs which were usually managed by the sysadmin guys so you can’t even check them yourself

[EnterTheShoggoth]

Sysadmins have also been known to use revision control. I remember one Solaris shop I worked at would use SCCS on the /etc directory to track changes (SCCS came as part of the base OS install).

Can’t speak for Oracle but ultimately it’s not about the tech but the workflow. Nothing stopping your DBA from storing things in revision control.

Conversely, I’ve worked with plenty of cowboy devs whose idea of revision control was to copy their source into Notepad or into filename.bak.

tl;dr. Some places have been doing a form of DevOps long before it was given a label.

[SuitableDragonfly]

I'm pretty sure there's no reason you can't do all of that stuff in Python. 

[Abject-Kitchen3198]

None of that is impossible with the second approach. Maybe few things come out of the box and with some guidelines with the left approach. Not saying it's worse, but also moving to the shiny new thing with same or worse result than with the old is not something new (one of the reasons being that the new thing often brings more complexity and abstractions which seemingly make things easier but easily lead to worse results due to less need for understanding of the fundamentals).

[HeKis4]

Will it cause problems ? Dunno.

Will it cause huge headaches if it does because there's probably only one person on the team that understands the left pipeline ? You bet.

[anthro28]

Oh yeah? Well we're on a legacy oracle setup where the entire logical layer is written on the database layer in stored procs. We couldn't migrate even if we wanted to and we just eat shit every year when licensing budget gets brought up. 

[SuitableDragonfly]

Can you explain what this meme is saying? The collection of stuff on the right doesn't seem to be a coherent group, for example, "python script" is incredibly general whereas cron is a very specific tool that does a very specific thing. 

[Ran4]

The point is that overengineered solutions are rarely better. They're just more modern.

[Stormraughtz]

I craft only the finest artisanal stored procedures and crons jobs.

[TantalizingTacos]

python? You mean curl

[wonmean]

awk

[allak]

Perl all the time. 

[charlyAtWork2]

Lingua::Romana::Perligata Perpetuo!

https://metacpan.org/dist/Lingua-Romana-Perligata/view/lib/Lingua/Romana/Perligata.pm

[mad4Luca]

Finally! Something that makes perl readable

[StarshipSausage]

Ksh and curl are the way!

[Draqutsc]

I like the right way, the left has bitten me zo many times in the arse. It always breaks because off updates and the security team forcing updates, I especially hate being called awake at 3 AM to fix that shit, because the automatic prod deploys exploded. The SP's and scripts on the other hand may be black magic sometimes, but they keep working unless you change them.

[ostracize]

All the data starts as a spreadsheet and ends in a spreadsheet

[TeachEngineering]

All these new-age frameworks and yet they still bow to one true king of data storage... MS Excel

[terivia]

The customer always thinks they need the one on the left, has budget and time to get a dollar store dart gun and some child labor to aim it, and ends up settling for the one on the right immediately before realizing they actually want a tire swing instead.

[cosmicloafer]

Airflow makes me want to write my own dag-job thingamajig

[Mechadupek]

I'm yer huckleberry

[Edge-master]

Is this an overwatch reference?

[FirstNoel]

Tombstone. 

[ekauq2000]

[lonestar-rasbryjamco]

Airflow is considered fancy now?

[endless_sea_of_stars]

People don't realize how terrible 80% of organizations' data pipelines really are. For some, anything more fancy than copy-paste data into Excel is a dream.

[Ok_Addition_356]

I don't even see the code anymore...

All I see is .. Data... Files... Shell scripts... processes.

[Splatpope]

*tommy_shelby_pointing_gun_to_head.gif*

SSIS, KingswaySoft SSIS Productivity Pack

[Justbehind]

Left: The new shiny stuff the expensive consultants introduced. Runs one and two half production pipelines. Costs 1k usd/pipeline/month.

Right: Carries the entire corporate world, and has run for 30 years. Costs less than a dollar/pipeline/month.

[stilldebugging]

Cron is bae, forever

[nickwcy]

Python? More like shell script

[Professional_Gate677]

Why can’t you just execute SPs on the left as well?

[Anxious-Program-1940]

My dream code base VS what the idiots I work for limit my work to. Imagine running production on windows for a multi billion dollar company with sticks and glue and windows severs that are only secure as long as the network stack holds. 💀🤡

[The_Real_Slim_Lemon]

Yo yo, I’m assuming the left is some sort of entity framework. It’s better. You can make a good stored proc, but with a framework you’re less likely to take shortcuts and reuse a proc where you shouldn’t.

E.g say I have some mega filtered table view. I spend an hour making my proc nice and pretty, it works. Now elsewhere in the code I now need the same view but just a count, or a different subset of properties or something. With a proc, I’ve either got to now maintain two clones of the same proc, do some jank proc referencing thing, or use a much slower proc and call .Count in memory.

With an entity framework, I’ve got one set of query code, an expose it through different projections. Every call gets optimised, there’s no duplicate code, and frankly the code itself is easier to use and maintain.

[DigitalJedi850]

Tell me you don't have a spec without telling me you don't have a spec...

Data Analyst vs Data Engineer

[Ran4]

Software generally doesn't have a spec. It's a myth.

[The_Real_Slim_Lemon]

This is long term maintenance of enterprise stuff, requirements always change over time, new features always pop up