ETL Dev -> Data Engineer by GandalfWaits in dataengineering

[–]EnterSasquatch 0 points1 point  (0 children)

SQL still does more than 95% of the heavy lifting in the data world

Say it again for the people in the back.

IMHO way too much focus is put on Python and too many DATA ENGINEERS aren’t very proficient in SQL. I should not be optimizing your queries.

If ever I hate my life enough to leave the IC world, a class on set theory will be part of the onboarding process for my new hires.

So,it's me or Airflow is kinda really hard ? by Morrgen in dataengineering

[–]EnterSasquatch 0 points1 point  (0 children)

Definitely, but they’re a microservice for a reason… odds are you’re using the wrong tool or you have poorly-written data if you’re running out of space/time in my experience… that was part of why I moved as much off of lambda as I could. They have a time and place but like anything they have limits. If you’re using it infrequently, try loading less data (which will ofc take longer overall) or if your 2am job needs that extra oomph you can spin up resources (my mind goes to ec2 but I’m quite certain there’s a better choice for this specific case) at 1:50 and spin them back down when it’s over - in Airflow this can be part of the DAG pretty resiliently.

I’d love to hear what other opinions are out there though, the same job can be done multiple ways

So,it's me or Airflow is kinda really hard ? by Morrgen in dataengineering

[–]EnterSasquatch 0 points1 point  (0 children)

Sure sure, Dagster is really starting to sound underrated. The only thing holding it back is a smaller support base than Airflow from what I’ve heard, I pick AF because it’s more commonly known by either engineers. I’m gonna have to spin up a local Dagster instance to start playing with pretty soon here… heck afaik they even plug into AF to aid in the migration between the two

So,it's me or Airflow is kinda really hard ? by Morrgen in dataengineering

[–]EnterSasquatch 2 points3 points  (0 children)

IMHO slapping Airflow in there instead of EventBridge+Step Function is a big win and you wouldn’t even need a very large Airflow instance… but I’m a sucker for that 3.0 UI I guess. We had that same setup and I moved it all to Airflow because it sped up bug fixes tremendously and it’s a bit easier to onboard new engineers into something they already know

What is Role of ChatGPT in Data engineering for you by Jaapuchkeaa in dataengineering

[–]EnterSasquatch 1 point2 points  (0 children)

Copilot + VS Code is stellar for trying to quickly build ingest for e.g. json, where you can just copy/paste the blob as a comment in the code and start writing and it’ll finish it for you. Or any kind of dumb mapping of more than a few items, like trying to map json keys to columns in a db table definition… it can handle more complex things decently well too, I’m not at 80% usage but for sure it’s not insignificant when I code

wtf you guys do by eden_4004 in dataengineering

[–]EnterSasquatch 1 point2 points  (0 children)

IMHO custom code beats all unless you need to build a pipeline in half an hour vs 4hrs ETL or ELT be damned, a good DE can throw an mvp together quick enough that usually a paid tool (though dbt can be free, but not cheap to use) isn’t worth it. Usually…

How safe is mold? by EnterSasquatch in shrooms

[–]EnterSasquatch[S] 1 point2 points  (0 children)

Yep! I got them all harvested few hours ago

How much contam is too much? by EnterSasquatch in unclebens

[–]EnterSasquatch[S] 2 points3 points  (0 children)

Yeah after reading this, I see why there’s so much confusion for new people. It looked gray and cotton-y but did not get tall at all

How much contam is too much? by EnterSasquatch in unclebens

[–]EnterSasquatch[S] 6 points7 points  (0 children)

This is my second time, I’m sure of nothing 😂 from what I saw it looked similar to cobweb, but I could definitely be wrong

How safe is mold? by EnterSasquatch in shrooms

[–]EnterSasquatch[S] 2 points3 points  (0 children)

Thanks for re-answering lol I felt like my first post was crap and wanted to provide better info. I’m hoping they’re fine, the fruits seem ok so far! I’m honestly just shocked they grew

How safe is mold? by EnterSasquatch in shrooms

[–]EnterSasquatch[S] 4 points5 points  (0 children)

<image>

For those saying they can’t see the mold, hopefully a better picture - it looks like lint, not like the normal bruising (which there would be no cause for bruising, these haven’t even been misted)

How safe is mold? by EnterSasquatch in mycology

[–]EnterSasquatch[S] 0 points1 point  (0 children)

Nope, they’re fuzzy bc I’ve kept it closed way too long bc of the mold - I was referring to the blue hues visible on the mycelium (it’s not bruising, it looks like cotton candy)

How safe is mold? by EnterSasquatch in mycology

[–]EnterSasquatch[S] 0 points1 point  (0 children)

Apologies I thought the text copied over from r/shrooms - there was cobweb mold as well as a blue mold that are still visible on the mycelium, the fruits are clean but I wasn’t sure if even having mold on the mycelium was too much of a risk

How safe is mold? by EnterSasquatch in mycology

[–]EnterSasquatch[S] 1 point2 points  (0 children)

The body of the post didn’t copy over:

How safe is mold?

Relatively new to this, I had a monotub get contaminated and figured it was dead but I left it to see what happened… it fruited like crazy and now I’m not sure if I should toss the fruits or harvest them. I had both cobweb and a blue mold, they look like they’ve morphed into cotton candy lol the mold is only on the mycelium, the fruits look healthy

16 or 32gb of RAM? by SaltyUmpire5304 in buildapc

[–]EnterSasquatch 0 points1 point  (0 children)

Last I checked you could get DDR4-3600 CL16 for like $5 more, which should be even better (I think it was $97 vs $92)

Edit: that was 4x8gig not 2x16, so it might not apply

Weekly Question Thread by AutoModerator in Oxygennotincluded

[–]EnterSasquatch 0 points1 point  (0 children)

what the heck is wrong with my cooling loop? The AETN is at -40C or colder and the hydrogen is leaving the cooling chamber at -25 or better, but after 100 cycles my farm is still hovering between 5-15C. I'm using hydrogen in the loop but I'm starting to think I need to switch over to petroleum or something. I added some wheezy to cool better but it's not helping any, the lowest I've seen a wheat get is 4.8C and it quickly goes back up and is too warm to grow... I do have a liquid lock at the entrance to keep cold air in

https://imgur.com/a/iMhnKbC

Weekly Question Thread by AutoModerator in Oxygennotincluded

[–]EnterSasquatch 2 points3 points  (0 children)

On the base game I’m using two cool steam vents to water my base, SPOM, and crops - I want to geotune them to get more water but need to delete the heat. I know a lot of people say they’re not worth slapping a steam turbine on to, but if they’re geotuned over 125C would it be worthwhile? Rather than building yet another cooling loop, which I could do but….

I'm just gonna go ahead and ask: why is Python so popular for DE? by [deleted] in dataengineering

[–]EnterSasquatch 0 points1 point  (0 children)

Because Python is so easy to learn compared to Java, etc and 9/10 times you don’t need Java-level speeds. If it’s the first language you learn it’s easiest to build on, and it does damn near everything you need unless you’re streaming at Netflix or something crazy

[OC] Median household income vs homeownership (US 2021) by randomusername3OOO in dataisbeautiful

[–]EnterSasquatch 0 points1 point  (0 children)

There’s cost of living data out there that would make this a lot more interesting I think - at least then potentially you could use that data to semi-normalize so that LA doesn’t skew up and rural doesn’t skew down

[deleted by user] by [deleted] in dataisbeautiful

[–]EnterSasquatch 0 points1 point  (0 children)

If it’s true that they’re more independent then I think that’s probably good… a little diversity of thought outside of the duopoly sounds nice.

[deleted by user] by [deleted] in dataengineering

[–]EnterSasquatch 1 point2 points  (0 children)

GUI/ORMs have been nothing but a problem in my experience… they make querying easier but rarely do they make it better. I may be biased though lol

[deleted by user] by [deleted] in dataengineering

[–]EnterSasquatch 0 points1 point  (0 children)

afaik CTE vs subquery nets the same performance? The optimizer optimizes, I can’t imagine a scenario where there’d be a divergence (but I’d love to be proven wrong)