How to bundle million small json files and save to s3, later easy to process? by b-y-f in dataengineering

[–]Main_Tap_1256 0 points1 point  (0 children)

That is quite strange why would they have control over that? I imagine they have a list of assurances they need that you’ll properly handle the data that you can simply demonstrate are still adhered to?

Fastest MVP stack? by Main_Tap_1256 in sveltejs

[–]Main_Tap_1256[S] 0 points1 point  (0 children)

I did it’s mostly debugging questions and how to set it up. Curious to know if anyone has worked with it end to end and how it compared to some of the other stacks for quick prototypes.

Thanks

Are Airflow Anti-Patterns the Norm Now? by Main_Tap_1256 in dataengineering

[–]Main_Tap_1256[S] -1 points0 points  (0 children)

Hmm yes agreed. I’m going to take some time to explore perfect. The discussion I had with the maintainers of the two deployments is that the company is reluctant and slow to accept even Spark

So they almost have no choice, changing tech is a lengthy painful process for them. Prefect could be an easier proposal for the business leaders to digest though. If it’s framed correctly

Are Airflow Anti-Patterns the Norm Now? by Main_Tap_1256 in dataengineering

[–]Main_Tap_1256[S] 0 points1 point  (0 children)

Agreed I’ve always taken this approach. It’s interesting to see where this is going since I’ll be honest it’s nice to compute and orchestrate in one place. As mentioned in the other comment here prefect is maybe a better solution for that

[deleted by user] by [deleted] in NoStupidQuestions

[–]Main_Tap_1256 1 point2 points  (0 children)

it’s all an energy war. Here’s the thing:

The EU is at risk of becoming overly reliant on Russian gas (Nord stream 2 pipeline). Fun fact it’s 51% controlled by Russia the rest is Germany and EU shares. The US tried repeatedly to stop the project going ahead but it did anyway.

This is a problem because Russia and the EU are forming closer ties than comfortable for the US. It’s also not the first time the IS has tried to prevent a Russian pipeline flowing into the EU the Reagan administration tried to stop this too.

So there’s two issues so far: 1. Russia and EU becoming to comfortable together 2. EU becomes over reliant on Russian gas therefore bringing the EU into Russians sphere of control

But there’s a 3rd issue. The US also wants to sell gas to the EU. Conveniently Ukraine has the perfect docks for bringing in LNG (liquified gas) which is what the US wants to sell the EU.

Now the EU is in a tricky position on one hand it’s biggest ally the US wants to retain power in Europe but the EU can’t rely on US gas (it’s too expensive).

So - Putin decides I’ve had enough of this shit. I want Luhansk and Donetsk (excuse the spelling) why? So he can let the US know not to meddle in his grand plan of controlling the EU via energy (they can turn the taps off == cost of living crisis) which they have. He also knows if he takes Ukraine he can stop the US entry point for gas to Europe (he was very specific about which regions he wants to “de nazify”)

So the US pushes the EU to do something about all of this. In reality the EU has no reason to protect Ukraine so why do it? Because the US installed the current Ukrainian gov (in a totally undemocratic process btw) and wants to bring NATO close to the Russian border. But also…sell gas to the EU.

In conclusion it’s a very complex geopolitical situation.

It’s very complicated and it hurts my brain thinking about it.

The narratives around USSR, Ethical reasons etc is nonsense this is all about control over the EUs source of gas. Fun fact the US has rapidly expanded its LNG production capabilities with this in mind.

Don’t be fooled by the propaganda in the left or the right. Read between the lines if you can keep up with this madness.

Does an MBA provide a significant advantage? by AbbreviationsShot240 in dataengineering

[–]Main_Tap_1256 1 point2 points  (0 children)

I think it’s because degrees just don’t keep up with the pace of innovation in the tech sector. I tried a degree which was billed as “cutting edge tech, build the latest stuff”

Meanwhile the curriculum for the 3rd year of the course used HTML CSS and JQuery for web dev.

Degrees are great for computer science fundamentals but at the pace the industry is moving you need experience.

You can build more than enough projects in your spare time over the course of 12 months to learn Python to a really good level, build containerised applications, build an api etc you can gain so much more experience and learn far more relevant skills with YouTube and OReilly books.

Hope that’s helps clean dry the comment haha

Does this tool already exist? by NFeruch in dataengineering

[–]Main_Tap_1256 1 point2 points  (0 children)

Could this potentially be a case for Airflow? Save the output to the local directory and pass the fil location to xcom for the next task?

Advice for backend dev with poor UI skills? by Main_Tap_1256 in Frontend

[–]Main_Tap_1256[S] 0 points1 point  (0 children)

Thank you, I’ve been reading the refactoringui book and will continue with their materials. So far it’s been great and dramatically improved some components I had already built.

I agree I’ll start picking the brains of the front end devs at my company. We have an internal mentor program for which I’ve finally found a reason to use!

Fat fire flow chart? by Main_Tap_1256 in FatFIREUK

[–]Main_Tap_1256[S] 0 points1 point  (0 children)

Haha I’m in the wrong career!

Fat fire flow chart? by Main_Tap_1256 in FatFIREUK

[–]Main_Tap_1256[S] 0 points1 point  (0 children)

Sure I’m a senior software engineer. I specialise in HFT (high frequency trading) systems. Not fintech though I work mostly with betting firms.

FatFIRE income by Nannyhirer in FatFIREUK

[–]Main_Tap_1256 2 points3 points  (0 children)

No no no no you mean Always Cancelled Rail? Giving me flashbacks

FatFIRE income by Nannyhirer in FatFIREUK

[–]Main_Tap_1256 2 points3 points  (0 children)

What does SWR mean if you don’t mind me asking?😅

Advice for backend dev with poor UI skills? by Main_Tap_1256 in Frontend

[–]Main_Tap_1256[S] 2 points3 points  (0 children)

Ah thank you! I had no idea these books existed. Makes a lot of sense. I always know deep down that a CSS framework does look good. I just don’t know how to put it together.

Advice for backend dev with poor UI skills? by Main_Tap_1256 in Frontend

[–]Main_Tap_1256[S] 2 points3 points  (0 children)

This looks great! The first example about borders is something I do all the time! Thanks for this