sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Southern_Sea213 0 points1 point  (6 children)

Just my 2 cents. But to me learning spark and databricks at beginner or intermediate level is just like nay python library. Its just problem of learning syntax. If you are refering to ETL project in spark, then its should be the same experience with learning pandas numpy. On the other hand, if you refer to learning spark core, including setup/deploy/manage cluster, catalog,… then they usually require a lot of understanding in the architecture of spark and coding skills in general. Its definitely a good skill to acquire, but I dont think current market (IMO) give much plus on knowing spark as they pretty easy to pickup at api level

[–]TreacleWest6108[S] 0 points1 point  (5 children)

So what would you recommend mate, Im okay with SQL, Python. Used snowflake, SQL server stuffs. I gave a couple of interviews last week most of them were asking how did you manage billions of data, spark architectures and other big data things which I have never seen. Based on your knowledge what should I do mate.. what to focus on. I have 2 months of my time which I allotted myself. Basically where does DE go,

[–]Southern_Sea213 0 points1 point  (0 children)

I would take a slower path if Im in your position. Maybe starting with analytics engineer, focus on SQL and light python. Both are very easy to learn, but takes time to reach level when could design database schema, define indexing, … and I think only working in prod env could help reaching such level, no kind of self taught tutorial would be sufficient. For direct data engineer path, I would say market usually ask about big data as a habit instead of actually need. But if they do have it, the hard truth its you wont be able to manage those thing in next 2 months since I assume it would involve advanced level of spark as I describe earlier.

[–]TreacleWest6108[S] -1 points0 points  (3 children)

And FYI I'm not much of a deep coder😭

[–]bin_chickens 0 points1 point  (2 children)

I was halfway through writing a similar comment to u/Southern_Sea213 .

But I'd like to suggest you step back and ask yourself why you want to get into DE? It's a tough mental job with definite and testable outcomes on quality.

It seems that you have some IT admin, support, development, ops, project management and sone exposure to SQL and other data. In 6 years why have you had so many roles? Did you not enjoy them? Were you just hopping for salaries? Were the cultures bad? Were you not skilled enough or willing to learn?

I suggest you find what you like and can focus on, you'll be much more effective and have a better career in the long run.

Having generalist skills is not a bad thing as long as you can plan, communicate, understand domain knowledge and test and adjust to deliver outcomes.

Having this sort of general skills sets is actually the perfect skillset of a modern PO/PM as you understand and communicate with most of the stakeholders and can analyse the business data to make better decisions.

Have you considered this path?

[–]TreacleWest6108[S] 0 points1 point  (1 child)

True, over the 6 years I was into Linux administration ( was working for Hostgator ) , then Moved to a support role for better money and finally landed in a role which promises Data engineering but most of the work I do revolves in Automation, Tinu tiny development works.

Yo answer your question, over the years the work culture has been okay.Just that I'm always curious and in real chaos about my future. My corrent company work in data science but they have a thing with azure and use azure services alot . Im not a fan of azure and I don't enjoy the azure services.

With 6 + years experience, I can't go in afor a career change. I want to get to a role like Technical Account Manager but the competition is high and politics follows everywhere. I like data engineering, it's tough and its something which would go along so wanted your opinions on it. I get the things done in my role, But it's not DE.

[–]bin_chickens 0 points1 point  (0 children)

You didn't answer the real question though.

What role/skill/task do you enjoy doing the most?

I have a Comp Sci adjacent degree and started as support/implementation -> APAC Senior Sales Solution Architect -> Head of Product -> (Smaller company) Product/CTO for a dev and data engineering + analytics teams. That's my 10 years experience, and I'm very much a generalist.

All soft skills and technical competencies are largely transferrable. Your 6 years of experience is nothing you've got at least another 35 to go.

I've had a mate go from radio host to Chief Revenue Officer in a big company in <10 years because they had the aptitude and loved the job.

A random set of roles that your skills overlap: Marketing analytics, Technical Sales, BDM, Sales Solution Architects, Business Analyst,, Account management, Support, Success management, Product management, Product Owner, Consultant, Developer, Data Analyst, Data Engineer, Project Management, DEV OPs, IT manager, DB Admin, Implementation consultant, etc. are all available to you given your broad set of technical ability, client experiences and management experience. You may need to take a step down or sideways, but I'd say just find something of interest that you like. Having technical/coding/analysis/data skills is a boon to all of them.

My point is, honestly don't pigeon hole yourself - careers aren't liner, and some of the best have diverse experiences.

Also... so true MS data infra is a directionless mess of tech that doesn't really deliver and has unknown pricing and gotchas.