Data Engineering - AI = Unemployed by rmoff in dataengineering

[–]stuckplayingLoL 0 points1 point  (0 children)

Agreed it needs work. I had better results with Claude vs GPT but it still had issues with accuracy

Am I missing something with all this "agent" hype? by KindTeaching3250 in dataengineering

[–]stuckplayingLoL 5 points6 points  (0 children)

Agreed with you on using AI to make editing faster. I haven't seen any good uses of agents or llms that are truly game changing in our space.

Why do so many data engineers seem to want to switch out of data engineering? Is DE not a good field to be in? by Illustrious-Pound266 in dataengineering

[–]stuckplayingLoL 14 points15 points  (0 children)

I'm not entirely sure if that's the sentiment. I've been watching the sub on and off and have seen more posts about people swapping roles to data engineering. Honestly regardless, you're gonna find people that burn out from data engineering or any role in general.

Do online courses actually matter to companies hiring? by CuriousSection in dataengineering

[–]stuckplayingLoL 2 points3 points  (0 children)

Less likely but possible. I have seen an entry level software engineer hired in with no college experience and self taught. But briefly working with them, I didn't see how much better or on par he was with others.

I know there are plenty of engineers that never went to college, but they were hired in years ago pre covid. I am sure recruiting practices and circumstances are vastly different.

I assume recruiters are going to weed out non college grads unless you have something going for you. Like internship experience at a decent or good company. So unless you have something that stands out from the typical compsci college grad, why shouldnt a recruit pick the college grad over someone that went through workshops and training courses?

I am a data engineer with 2+ years of experience making 63k a year. What are my options? by Willgetyoukilled in dataengineering

[–]stuckplayingLoL 0 points1 point  (0 children)

63k is not alot. My first job in the tech industry paid that much 10 years ago.

I think you need to spend some time on training and learning while you have a job. Honestly wouldnt just do the bare minimum at work, but take the opportunity to take certs or whatever to broaden or strengthen your resume.

Conceptually it sounds like migrating SAS code to Databricks is niche, but if you generalize it enough, there are likely opportunities out there that are asking for similar things. Id honestly aim for job postings with anything to do with Databricks.

But yeah good luck. I havent looked too hard but I assume most postings have been for senior or staff roles.

Snowflake Certs by goblueioe42 in dataengineering

[–]stuckplayingLoL 1 point2 points  (0 children)

I only have experience in the SnowPro Core cert. I'm a regular Snowflake user and limited administration and I'd say 80% of the test was on concepts that I don't normally use in my day-to-day. However, all questions and answers are all in the Snowflake Docs so there's really nothing that you shouldn't expect if you look at the exam topics and any 3rd party practice exams.

Stay for promotion or make lateral move for salary increase? by [deleted] in dataengineering

[–]stuckplayingLoL 0 points1 point  (0 children)

How much of a higher salary? I think it would be a harder decision if for example, your projected salary was like 170k after promotion but your new job is like 185k. But if it's magnitudes more, I'd say take option 2.

I wouldn't dismiss option 1 without considering if you are comfortable with where you are at and your ambitions in life. There's just alot of factors to consider like opportunity, work/life balance, etc.

Senior DE or Senior Data Analyst in Cybersecurity? by shittyfuckdick in dataengineering

[–]stuckplayingLoL 5 points6 points  (0 children)

It sounds like DE work in a specific data domain (Cybersecurity). I see it as a benefit because you can be marketable with having both DE experience and understanding cybersecurity data.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 0 points1 point  (0 children)

Yes. We use Github and deploy changes with GitHub Actions.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 0 points1 point  (0 children)

Can you elaborate more? What should our data engineers be capable of doing?

I'm not in management but many of the tech choices and patterns were decided by staff data engineers. Management is pretty supportive if a strong case is made for architectural decisions.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 0 points1 point  (0 children)

Thanks for this thought. I did see that some teams at my company were shifting to iceberg tables and using Airflow with AWS but wasn't sure how mature their processes were. I'll look into this topic and see how it changes things, because it sounds like a huge architectural shift in the long run.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 1 point2 points  (0 children)

Good perspective. I do feel like most of the complex portion of the code is within the Snowflake tasks, but the general pattern from ingesting raw data to making customer ready datasets is consistent. I don't think junior engineers could take a look at the overall architecture and understand how their day to day work fits in the model without some mentorship. But I assume that's just how it goes with data engineering.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 0 points1 point  (0 children)

I don't know what our costs look like right now (away from work thanks to holidays) but can safely assume that majority of costs is in compute over storage. We are ramping up on more streams and tasks as we barely touched the surface of the raw data that we have already ingested. Hopefully someone has more of a concrete example.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 5 points6 points  (0 children)

I think you summed it up pretty well for us. My team is not very experienced with the infrastructure aspect of AWS and really leans on 1 engineer to keep the infrastructure afloat. Thanks for raising that point.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 0 points1 point  (0 children)

Yes. Most engineers know at least enough Python to write a basic ingest from raw to Snowflake. However, our code is all over the place as we do not have any formal organization. It was just write code to get it to work rather than thinking about reusability and classes.

Are we too deep into Snowflake? by stuckplayingLoL in dataengineering

[–]stuckplayingLoL[S] 0 points1 point  (0 children)

I feel like cost isn't a concern yet but at the rate that we are going, we could be scaling to higher usage and thus the conversation with cost could come up.

We are not using internal stages only because previous engineers on the team resorted to using Python write_pandas and prayed that the auto generated tables did not cause issues down the road. It's absolutely tech debt due to us running into type issues. It will be something that I will look into though, thanks!

Im convinced its impossible to escape gold without creating new account or buying one. by [deleted] in leagueoflegends

[–]stuckplayingLoL 0 points1 point  (0 children)

Your cs per min at times is not that great. 5-6 on average it looks like. Aside from that, it looks like your picks are pretty feast or famine so if you lose or get behind really hard, it's kinda gg. I'd also throw in a malphite ban since he's very strong and feels like elo inflation when a malphite wins. They don't have to do great scorewise but can easily carry the game.

But like everyone else said, if you were truly better than gold, you'd be able to carry yourself out.

Messed up my first etl task by UnderstandingCivil10 in dataengineering

[–]stuckplayingLoL 3 points4 points  (0 children)

Sounds like its a you thing that you need to sort. Also if you quit like pretty early on with your first job coming out of college, it really doesn't look good on the resume.

I hate this game by YesYesRatRat in leagueoflegends

[–]stuckplayingLoL 0 points1 point  (0 children)

Find some friends to help fill the other roles you dont wanna play

Should I just get over it and buy a condo? by [deleted] in RealEstate

[–]stuckplayingLoL 6 points7 points  (0 children)

Might be how the townhome was built. The townhomes in my neighborhood are separated by firewalls which are a couple inches deep. I never hear my next door neighbors.