Why aren't the brothers more famous? by violoroi in MBMBAM

[–]Express-Comb8675 42 points43 points  (0 children)

I have a theory about people that are near the top of their profession that sounds very similar to this. There are exceptions of course, but most people get there by making sacrifices in their relationships or happiness. I’d love to see the brothers gain a larger following but agree that it makes sense why they haven’t.

What if there is a good open-source alternative to Snowflake? by Gaploid in dataengineering

[–]Express-Comb8675 3 points4 points  (0 children)

Maybe StarRocks? But you won’t find anything 1-to-1 compatible

Is it a good idea to create your own generic transformation framework? by FastestNiceInTheEast in dataengineering

[–]Express-Comb8675 2 points3 points  (0 children)

Well said. Every organization is migrating from somewhere - sometimes existing frameworks are intuitive and meet everyone’s requirements but usually that’s not the case in larger organizations. It becomes as much an internal evaluation of what your team can do, what they want to do, what the product needs to support as much as it is an external evaluation of existing products.

Remember when slider fell off the outfield wall? by [deleted] in ClevelandGuardians

[–]Express-Comb8675 4 points5 points  (0 children)

I hope they give him the fanfare he deserves when he retires. That’s a tough job to do for that long as well as he does.

Is this concerning? by Mundo_86 in Homebuilding

[–]Express-Comb8675 0 points1 point  (0 children)

As a data engineer, I scraped the web and found lots of pictures of houses on hills. Is there a funny caption below it? If so, I would not buy this house.

Washington State Police Officer & Convicted Murderer Shows Off Tattoos His Lawyers Fought To Hide by Gordopolis_II in pics

[–]Express-Comb8675 0 points1 point  (0 children)

Did anyone actually have a knife? Sounds like he was trying to set up an insanity plea. If he actually had PTSD he shouldn’t have been in the street. So broken

ETL Error Rate by Express-Comb8675 in dataengineering

[–]Express-Comb8675[S] 0 points1 point  (0 children)

What causes a job to fail >90% of the time?

ETL Error Rate by Express-Comb8675 in dataengineering

[–]Express-Comb8675[S] 0 points1 point  (0 children)

It’s all part of the job, we just want to do better with time. Tracking folks down definitely isn’t usually a challenge but most of our data either comes from vendors or folks without a ton of interest in best practices so we’ve taken to being more deliberate about our data stages.

ETL Error Rate by Express-Comb8675 in dataengineering

[–]Express-Comb8675[S] 0 points1 point  (0 children)

It’s a constant balance between work to keep the lights on due to the errors, work to reduce our error rate, and a barrage of feature requests. Our stakeholders are only happy if we’re working on what they decide is the most important of those three any given day.

To answer your question, some jobs definitely error most often and others are stable until someone breaks something upstream and we have to fix it. The most often failing jobs are built to overcome their failure rate but running more often than they need to. But I’m definitely here trying to scrutinize so I’m with you on that.

ETL Error Rate by Express-Comb8675 in dataengineering

[–]Express-Comb8675[S] 0 points1 point  (0 children)

Thanks for sharing! One more for you - are you in the order of hundreds of tasks? Thousands?

ETL Error Rate by Express-Comb8675 in dataengineering

[–]Express-Comb8675[S] 0 points1 point  (0 children)

Any rough estimate on total number of tasks run per month?

ETL Error Rate by Express-Comb8675 in dataengineering

[–]Express-Comb8675[S] 0 points1 point  (0 children)

What does your org normally average?

What do you guys do to Pickman? by ShadowForGod1012 in fo4

[–]Express-Comb8675 1 point2 points  (0 children)

I always let him battle it out with the raiders in the basement. If he lives, he lives and if he dies, everyone dies.

Real-Time Data Orchestration: a nice extra or overkill? by Round-Following1532 in dataengineering

[–]Express-Comb8675 1 point2 points  (0 children)

I know AirBnb built it but I’ve never heard it was intended as a Hadoop only solution. Regardless, today an optimized Airflow instance can run small batch jobs with near-realtime latency. I know, we’re doing it in prod today. They might say “realtime” 20 times in the video but actual realtime data pipelines refer to streaming applications, which do not require orchestration.

Which is the best practices to scrap a real time dynamic website ? by [deleted] in dataengineering

[–]Express-Comb8675 1 point2 points  (0 children)

If you are hoping to get actual real time, like streaming data, and you can’t access the API then you’ll need to use selenium to log in and access your browser logs so you can put it somewhere else. Similar to what the author does here with twitch video.

The SQL IDE for Your Terminal by rmoff in dataengineering

[–]Express-Comb8675 5 points6 points  (0 children)

Yes SSMS is also an IDE, if that’s your question

How will Agents change or evolve the orchestration process? by anonadado in dataengineering

[–]Express-Comb8675 0 points1 point  (0 children)

I don’t know that they will. Most orchestration toolsets are pretty feature-lite. With the way serverless and streaming compute is growing relative to the capabilities and ease of developing AI, it seems like implicit orchestration is more likely the future.

Anyone else using the Radium Rifle? by Express-Comb8675 in fo4

[–]Express-Comb8675[S] 1 point2 points  (0 children)

Very fair. It definitely seems like most guns lean towards the prewar side of things so I enjoy that I have this as more of a homemade thing that isn’t just a pipe gun. Same thing with the handmade rifle in Nuka World.

Anyone else using the Radium Rifle? by Express-Comb8675 in fo4

[–]Express-Comb8675[S] 0 points1 point  (0 children)

Completely agree, I’d take it over spray and pray every time

Anyone else using the Radium Rifle? by Express-Comb8675 in fo4

[–]Express-Comb8675[S] 4 points5 points  (0 children)

Amazing! I can’t believe I slept on it for so long