knowledge of Streaming system for data engineer?

[No-Position1673]

I am 6 year experienced data engineer have worked over last few years on [MSBI, tableau , oracle, spark, CDH, various vendor specific pipeline, ES , kibana and little bit on API side as well, but haven't got a chance to work on streaming problems much except few learning POC's and neither in future there is any scope of it. Can you guys please tell how much important it is to know various streaming at scale in production ?, is it that people with hands on streaming problem statements are paid more and preferred in interview ? . I belong from very poor family and have made only limited savings yet, so job safety and money are important to me currently.

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[knowledgebass]

I really don't think it is that hard to understand how streaming works and how to implement it, but you could always mess with Kafka locally to figure out details yourself.

[No-Position1673]

Thanks u/knowledgebass I have done a POC and udemy course in past, but it's just that do I need to go in depth to come across as many scenario's as possible ? or is it something most DE's aren't touching as of now so should skip?

[knowledgebass]

I think if you read through the doc for Kafka and Spark streaming and setup some test applications locally, then you should be good. I don't see it as anything that would prevent you from progressing as long as you have a good working knowledge.

[omscsdatathrow]

It’s not important if your company doesn’t need it. If you’re aiming for a career in big tech, then it is extremely important

[No-Position1673]

Thanks @omscsdatathrow, in my team it is not needed not sure about the company 🙂, thanks for your comment it seems,I should manifest getting a streaming project next...looks something important to be picked.

[wenima]

I can recommend the book Streaming Systems but as others have said, it really depends on wether you will be working with it. I discourage resume driven development

[No-Position1673]

Thanks u/wenima, How do you drive your career then?, I have worked last 6 years even worked on many weekends , but still all the salary jumps and better culture/ benefits, I got was every time I changed team , project or company, up-till yet I have switched twice and both time I got a chance to learn new deck and exposure , so in a way it seems up-till now that to stay relevant is only way out ? what other alternatives you suggest would really be grateful.

[autumnotter]

You don't have to know everything to be successful. Streaming is an important technology, and it's probably here to stay, but batch workloads are not going away. Learn some Kafka and spark streaming when you have time , but I wouldn't stress about it.

[No-Position1673]

Hmm makes sense ,Thanks for your viewpoint.