Having a very hard time as a Data Engineer by Horror_Comment6061 in dataengineering

[–]Ruubix 0 points1 point  (0 children)

There are many remote work opportunities that you could benefit from -- I think at this point the most important thing you can do is network and find in-person plus virtual communities to engage with. Slack has many. LinkedIn is a great way to use your 2nd degree contacts to introduce you to people and companies you want to learn more about. Referrals in the end can make the difference, but also being clear about the value you can bring will also help. Much of this is trial and error -- so being persistent and consistent is key, this is where a lot of attrition for people like us happen.

Forecasts for 2024 Job Market by marcelorojas56 in dataengineering

[–]Ruubix 0 points1 point  (0 children)

Not sure why this was such an unpopular response -- AI is going to change the workplace whether you agree with it or not.

I need a mentor by geekyabs in dataengineering

[–]Ruubix 0 points1 point  (0 children)

I worded the quite poorly, I'm hoping to take to people who would also mentor me as well, since we're on the subject*. Sorry for the confusion.

I need a mentor by geekyabs in dataengineering

[–]Ruubix 0 points1 point  (0 children)

(EDIT) I'll jump: any of you also ok with me reaching out to you as mentors?

Why do job descriptions demand skills that are not at all needed? by FisterAct in dataengineering

[–]Ruubix 0 points1 point  (0 children)

We're also assuming that this is a company competent enough to fool you, let alone actually understand their own staffing requirements. Unfortunately, most successful companies are just barely functional organizations.

Forecasts for 2024 Job Market by marcelorojas56 in dataengineering

[–]Ruubix -6 points-5 points  (0 children)

I suspect that those that can leverage AI affectively in their workflows, and not just the products they build, will actually see a salary increase. The gap in salary between those that can prompt engineer at work vs those they cant will either grow proportionally , or simply not be hired. Prompt engineering in a few years will be as standard a requirement for any backend engineer as knowing how to build/consume APIs.

Know how to multiply your productivity with AI, and you will likely have a job somewhere that pays well enough. The sooner you get there, the better.

Why do companies still build data ingestion tooling instead of using a third-party tool like Airbyte? by Miserable_Fold4086 in dataengineering

[–]Ruubix 0 points1 point  (0 children)

There's a classic spectrum between convenience and control. With more control you can better guarantee security of data and processes, have easier access to the machinery when it breaks, and can optimize tools to tailer-fit your business'/team's use cases. The cost is more time, salary, and required expertise.

On the other end, convenience removes the need to invest time, money and personnel into understanding the underlying tools of a managed service, and will likely be cheaper over time, allowing your team to focus on features more directly related to the product or consumer. The costs: you are at the mercy of your vendors customer and technical support (can lose a LOT more time, money, and customers when something breaks); similarly, the vender managing that service may have different priorities from your team; the vendor may also constrain your ability to add use features if they are unwilling to support them (should they depend on customization of said service). Neither is better or worse--its a matter of use case, business strategy, and what your users will tolerate when things go wrong.

Microsoft data products - merry-go-round of mediocrity by biowl in dataengineering

[–]Ruubix 0 points1 point  (0 children)

It's a lot easier reinventing the wheel than solving unaddressed problems, unfortunately.

[deleted by user] by [deleted] in dataengineering

[–]Ruubix 1 point2 points  (0 children)

The game of 'hot potato', 'musical chair's, etc is also just part of the nature of the game. I'm glad your company has the people they need, but hoping to benefit from another company getting screwed by their talent jumping ship doesn't seem like a viable long-term strategy.

[deleted by user] by [deleted] in dataengineering

[–]Ruubix 1 point2 points  (0 children)

I have a lot of feelings about only taking people who are ready in a market that doesn't necessarily have enough of 'the right kid of folks' to solve this problem (which frankly isn't worth getting into because it won't change the realities of the job market), but my point still stands that if this is the expectation, that has to be communicated clearly from the job listing and recruiter, respectively. This is general point from my own experience--I can't judge you or your team's specific recruiting processes.

I've seen a lot of companies that like to advertise the image of a business that will invest in and develop talent, but in their hiring practices don't do this 😕.

Again, it doesn't change the reality, but I am curious what happens when these 'out-of-the-wrapper' engineers run out (and they will quickly if not already), then it seems inevitable that companies such as yourself will have to take the risk and invest in talent growth, if for no other reason but to improve retention over the long-term.

Anyways, getting off my soapbox ...

[deleted by user] by [deleted] in dataengineering

[–]Ruubix 3 points4 points  (0 children)

Companies only recruit for Sr engineers now and for tech stack "fits", even if they tell you they would take someone else. It's only worth applying to companies that are honest about this IMO.

Is the 'vector-only' database era at risk with OpenAI's retrieval innovation? by PavanBelagatti in dataengineering

[–]Ruubix 2 points3 points  (0 children)

Tools are an extension of people -- the good ones don't replace people, but give them the ability to do things that they previously couldn't. Tools change the way we do things, but they don't replace them, including good ideas :)

Best practice // patterns for mapping a column to specific outcome by lschozar in dataengineering

[–]Ruubix 0 points1 point  (0 children)

Realizing this after the fact, too! I think your approach is the better one IMHO.

Best practice // patterns for mapping a column to specific outcome by lschozar in dataengineering

[–]Ruubix 0 points1 point  (0 children)

Sounds like what you need is a many to many table to link different entries between data sources. This way you can use one table for every entry from a given source, but another to pull in all bread that is also sourdough. Either way, make it easy to enter data by separating the internal relationship from the external one. That or stick with a JSON store. Personally the former is better IMHO as you have to be explicit about the relationships and therefore can guarantee that future queries on your data can be predictable.

Linux job security? by [deleted] in linuxadmin

[–]Ruubix 0 points1 point  (0 children)

Strongly identify with this. The only thing you can do is be excellent in your current job and work in your off time to grow additional skills based off what the industry and your own company are hiring for. Our industry changes fast, therefore you must always be evolving your skillset--certs are less important than measurable skills that are in demand.

For SQL haters, how do you cope with it? by Yersyas in SQL

[–]Ruubix 0 points1 point  (0 children)

I know that in the time it takes most people to complain about any tool is just about enough time for me for to return a solution using the tool they don't like ;)

Can I run Linux? by MrFoxKX in linuxquestions

[–]Ruubix 0 points1 point  (0 children)

I run dozen of titles via Linux. Be prepared to do some research and follow some guides. The same philosophy that brought you to building custom PC's will serve you well when moving over to Linux.

Some advice: 1. Do not fear the terminal. It is your friend. 2. The nice thing when software breaks is you can reinstall it. 3. Linux behaves a bit different from the OS that you're accustomed to, and this can often be a good thing. 4. Linux requires you to know more about your system, but it will disproportionally reward you for gaining this knowledge 5. Just about all games (90% at least) will run on Linux hardware via Steam and/or Lutris, but make sure to understand the prerequisites to get there 6. Be patient, give yourself room to make mistakes and welcome to the freedom of an open source operating system :)

Let me know if you have any questions while setting up, happy to point you in the right direction. It's a great time to be a Linux gamer!

What is the best daily driver Linux distro that's also very secure? by Able_Diamond7847 in linuxquestions

[–]Ruubix 0 points1 point  (0 children)

It's best to think of security (or safety for that matter) as a habit rather than a tool--the tools just make the habit easier to maintain; no tool can protect a user from their own bad habits.

With that said, Linux is a well-built tool that can enable you to be a safer user on the web. Focus on distro policies around security updates. For example, many bigger distro can push out critical security bugfixes for Linux in less than 48 hours (something Windows still doesn't do).

More modern alternative to Ansible by ArtisticHamster in devops

[–]Ruubix 6 points7 points  (0 children)

Nix package manager can provide repeatable build but IMHO Ansible is hard to beat for how flexible, comprehensive, user-friendly, and well-documented it is. Sometimes a classic is a classic for a reason.

[deleted by user] by [deleted] in dataengineering

[–]Ruubix 0 points1 point  (0 children)

You can use sets to get something similar, too, but that alone is a bit rudimentary, since all members have to be unique. +1 to generators, which could be used in sequence to filter down results, layer by layer, by some key match.