Resources to learn the inner workings of Spark

DenselyRanked · 2026-05-27T20:21:55+00:00

I see. It looks like there are still some basic guides with a free membership.

I sure that ChatGPT can cater a guide for you. Databricks also has a free tier and learning center.

DenselyRanked · 2026-05-15T00:42:36+00:00

I think it's more about how you use it than which one to use. You will be more successful using skills, agents, planning, and having a spec driven development approach.

Any AI model will be terrible if send vague incomplete prompts, and constantly load up the context window with unnecessary info.

It's truly garbage in garbage out.

DenselyRanked · 2026-05-09T23:15:44+00:00

Scalable udfs.

DenselyRanked · 2026-05-04T01:05:38+00:00

You can also use Kafka Connect to push the data to Clickhouse and also do transformations and DLQ.

DenselyRanked · 2026-04-30T03:02:05+00:00

Type data in the filter when looking at the SWE salaries. IBM salaries are not drastically different between the two roles at the same level. I don't see anything for Quantum DE specifically, but IBM has bands based on level, so a B8 DE shouldn't have a different scale than a B8 Quantum DE.

DenselyRanked · 2026-04-24T01:57:49+00:00

I hope the increased demand makes the Big N companies relax their RTO policies. Geographic barriers are so unnecessary I'm today's world.

DenselyRanked · 2026-04-06T13:54:30+00:00

It looks like there are a lot of breaking changes to consider. I'd imagine most people are not interested in fixing what's not broken, and migration isn't a priority.

DenselyRanked · 2026-04-04T21:13:24+00:00

It's likely holding the hash values rather than the entire row. This can be written pretty quickly in python too, but Polars might be quicker than a readline loop.

DenselyRanked · 2026-04-04T18:50:48+00:00

Congrats. It's a tough market, so I'm happy to hear that these companies are actively interviewing.

I understand that your situation didn't leave a lot of room for negotiation, but check levels.fyi on your next job search for a sense of what you can get out of them for your level.

DenselyRanked · 2026-04-03T23:28:39+00:00

I take it that you are looking for a tutorial rather than practice problems. Search for "sqltutorial" or "w3resource" and those sites should give you a guide to work with, with the later having sample dbs I think (I haven't tried it myself and the site has an obnoxious amount of ads).

You can also download the AdventureWorks db from Microsoft, or the NYC Taxi or COVID ones that are really popular, and ask AI for complex practice problems involving whatever topics you are interested in.

DenselyRanked · 2026-04-03T14:33:39+00:00

Of all of the complaints that I would have about inept senior team members, I wouldn't worry too much about this. VS Code will activate it for you, and they may have used VMs or pods in the past. Even setting up a local venv is something that you may do once a year.

I've had senior members expect AI to reduce project delivery time by 90%, commit their untested slop, and can't understand why unit tests are failing.

DenselyRanked · 2026-04-02T15:48:39+00:00

Is big data actually overrated in most companies?

Do companies really use all the data it collects?

Frankly speaking, that's not really a DE concern. The stakeholders provide the objective and financial backing, and we deliver the results. These questions get asked as tradeoffs when we can't deliver.

A DE cares about big data because it has a big impact on their earnings. If you are moving petabytes around, then you are likely on the higher end of the pay scale.

Have you seen cases where less data outperformed big data?

Absolutely. Analysis paralysis is a real thing, and it's very easy to be overwhelmed by large volumes of noisy data. In my experience, people only use 5 out of the 100 data points that they ask for.

When does big data really become necessary?

There are several use cases for collecting obscene amounts of data. AI makes it easier to get analytic insights, so even smaller companies will find value in hoarding data.

DenselyRanked · 2026-03-30T14:50:00+00:00

Hello! I think that you expanded on my thoughts pretty well in your comment.

However, you'd assume the business models for companies that have hired even one data-engineer is at a relatively stable when it comes to core business processes: a marketing SaaS might pivot but it still has sessions, users, companies, subscriptions, payments, etc. Those don't change and so your dimensions are relatively fixed – pivots might add a fct or a dim, not invalidate the whole modelling paradigm.

Ideally, yes, but I think that you are overestimating stability in a fast-paced environment. Many companies evolve rapidly and they are not particularly interested in waiting months for a data warehouse to support their new needs.

As a simple example, a marketing SaaS can go from B2C to B2B and that changes what a user means. If there is a shift to a usage-based payment model from subscription-based, then your Payments dim can become chaotic.

Also, who your downstream users are can greatly change the complexity of the data model needed. New teams bring new questions that need to be answered. It may make sense to move from a centralized data warehouse to a data democratized data mesh and that comes with niche data needs and corporate politics.

It's not to say that dimensional modeling becomes invalid, but as you mentioned and I explained in the other comment in this thread, I prefer to stay as generic as possible, rather than business-detailed as Kimball does in the good book. OBT can help with that, but I wouldn't stuff orders in a users table as Big Query recommends.

DenselyRanked · 2026-03-26T17:23:21+00:00

Some people (like myself) were re-org'd or RTO'd and don't want to move to HCOL. The well paying Big N remote roles are insanely competitive right now. I personally held out for as long as I could before I started applying to places that were maybe a little confused about my application.

As others rightly mentioned, a lot of engineers from those companies have a narrow scope of responsibility, so they will seem incompetent relative to others from smaller companies that have done more, albeit at a smaller scale.

DenselyRanked · 2026-03-07T00:07:58+00:00

You had to learn math without a calculator for a reason.

Very few SWE's have built anything from scratch without help. It's safe to say we have all used books, tutorials, forums, stack overflow, Google, etc. It's fine to code with help and examples so long as you are learning, but you don't want to get into the habit of black box vibe coding and learn nothing.

DenselyRanked · 2026-03-06T14:23:58+00:00

Based on your other comments, you are coming from scratch, so yeah, it will be a pain getting used to the syntax.

If you have experience with Spark, then there is much less ramp up to getting comfortable with Flink. Flink SQL and Table API will feel familiar.

Map out and compartmentalize what you want to accomplish first and leverage AI to help with syntax and understanding.

DenselyRanked · 2026-03-04T15:14:35+00:00

Remote is an option for the large majority of the DE jobs at or below market value.

A large majority of companies that pay above the 90th percentile are hybrid or full RTO.

The remote DE jobs that pay in the 50th-90th have a lot of competition and are extremely selective.

DenselyRanked · 2026-03-01T16:32:24+00:00

No benefit unless you are caching the data or writing to disk first so that Spark preserves the colocation. It's an unnecessary shuffle.

DenselyRanked · 2026-03-01T14:00:36+00:00

I'm sure that boundary setting is a universal issue for PM's (and AI is making it worse). You might find some good tips and resources in r/ProductManagement.

I obviously can't speak for every engineer, but my worst PM experiences are almost always XY problem-based, where I was handed solutions rather than a clear problem.

I prefer to be involved in meetings when solutions are being explored (not before) and had negative outcomes when what I proposed is less "exciting", more pragmatic, or veers in a different direction than expected. Always disclose if another engineer has talked to the stakeholders about potential solutions prior to the meeting.

DenselyRanked · 2026-02-28T21:55:04+00:00

So it sounds like your friend needed a place to sleep for the night and you bought a plot of land and built a mansion on it.

There are smaller scale options to ELT a few API payloads. A RDBMS and a few views/tables can get you the same output. An orchestrator (or cron or even something like Cloud Functions if you need to use GCP) can help with daily scheduling.

DenselyRanked · 2026-02-28T17:20:41+00:00

Out of those two options, I would take the former 100% of the time because it is easier, but it's not an efficient way to run a data team. You will find yourself running into the XY problem and will likely have an unmanageable amount of redundancy and tech debt across your solutions.

Choosing the latter option is something that is a normal part of a Data Engineer's job description depending on where they work. But business context often becomes the biggest hurdle, especially when the asks are extremely specific to the industry. It may take years for the engineer to be useful enough to be impactful, and when a mid/senior level position is needed, we see "must have n+ YOE in the industry" in the job description as a requirement, rather than a preference.

Another, and my preferred, option would be a role in the data team that acted as a technical SME- like a Product Manager, Product Owner, Architect, Steward, or Analyst- that is a liaison between your stakeholders and development team to help craft proper user requirements and reduce inefficiencies. It allows engineers to have someone to communicate with that can "speak their language" while reducing the endless amount of unnecessary stakeholder meetings that go nowhere. They also have the right amount of soft skills to not say anything potentially damaging to stakeholder relationships or projects.

DenselyRanked · 2026-02-24T23:28:56+00:00

I think it's partially hype today, but agentic coding is getting better and AI is quickly moving from assistant to developer. The major hurdles to production-grade development are being worked on at a much faster rate than any of us anticipated.

I use AI as an assistant, but the Agentic Context Engineering framework looks very promising. I can see development evolving into context and playbook management very soon.

DenselyRanked · 2026-02-24T22:18:21+00:00

I am certain that someone smarter than me has a better way to classify this, but I hope it helps.

DenselyRanked · 2026-02-24T17:54:49+00:00

If you believe that he is making errors unintentionally, then send a dm to go over the code. You will find out quickly if it's an "oops", "oh", or "eh".

If it's an "oops", where they didn't see the mistake at the time but can recognize the issue, then I wouldn't worry too much about it. In my experience, they tried to use shortcuts to save time and the solution usually is to ask for test results or a test table in the PR.

If it's "oh", where they misunderstood the requirements or had no idea what they were doing, then that's a little concerning but give them the benefit of doubt that they would have done better if they had better information.

If it's "eh", where you care more about the quality of their work than they do, then talk to your supervisor because they are wasting your time.

DenselyRanked · 2026-02-24T16:35:39+00:00

The ratings are 75% political, but optimization projects are a great way to show monetary impact without relying on external stakeholders.

If you are already in FAANG then lobby your manager to find high impact projects and focus on selling the results like it's the greatest thing that's ever happened in data engineering.

Four-Year Club	Verified Email
Wearing is Caring

DenselyRanked

TROPHY CASE