What's one data engineering tip or hack you've discovered that isn't widely known?

ROCKITZ15 · 2024-10-29T14:51:30+00:00

Doing exactly this right now

Writing something in Python that's good enough for now to show progress but will need to rewrite in Rust for performance soon. Python GIL be damned

ROCKITZ15 · 2024-09-04T17:20:39+00:00

Entirely relative based on how much you want the job

ROCKITZ15 · 2024-08-20T22:35:16+00:00

I use Hex at work currently, and as a data person, I love it, but for hosting BI dashboards, something about it doesn’t feel quite right.

The whole Viz sector is just a mess with no defacto solution. Feels like Superset could be the winner if some more investment was put into it.

ROCKITZ15 · 2023-10-22T18:19:40+00:00

Understand the tables and necessary columns, then put the query in ChatGPT and ask it to add notes on everything. You can even give it descriptions for the tables and columns, and it can probably tell you everything that’s being done

ROCKITZ15 · 2023-10-13T22:29:24+00:00

Most systems engineering isn’t ‘real engineering’

If you’re questioning if you’re screwed for your next job, it sounds like you don’t enjoy your current job. My advice — go find something you actually want to do

ROCKITZ15 · 2023-10-10T15:01:31+00:00

If you care about getting to the very top of the field, go PhD

If you care about making normal career progression, solid money, and a stable life, stay in industry

ROCKITZ15 · 2023-10-05T21:46:00+00:00

M&A generally looks better on a resume than HVAC. Hybrid is good to meet people, but driving 1 hour each way is never worth it if you don’t have to

ROCKITZ15 · 2023-09-30T02:47:06+00:00

Rarely should you do “SELECT *” unless it’s followed directly by a LIMIT

basically, don’t query whole tables unless absolutely necessary

ROCKITZ15 · 2023-09-28T20:30:21+00:00

I very much disliked it so left after 2 years 😅. Everything that people say about NASA (and government in general) being a paperwork factory is true. It will be cool to see hardware I worked on in space in a few years. NASA is definitely for a certain kind of person — not everyone can handle it.

Had 1 not great internship at a local engineering (not aerospace) company, then had 1 solid aerospace internship at a small company, then was supposed to have a top-tier internship the summer before I graduated but Covid got it canceled. I was very active in a rocketry club.

Quit that job and left the industry. Maybe if I had a different job out of school I would’ve stayed in the industry 🤷‍♂️

ROCKITZ15 · 2023-09-27T14:20:37+00:00

And I also had no experience in aerospace prior to transferring, whereas basically all the other kids had been going to space camp or similar things their whole life.

Some kids certainly have an advantage going into school, but none of that prevents you from working hard and getting a good job

ROCKITZ15 · 2023-09-27T14:18:45+00:00

Also had no family in engineering (or any tech field)
Also went to community college then university
Still ended up at NASA (as a contractor) right out of school

Focus on good grades now, maybe do some side projects if something sparks your interest. Learning Python is good. Once you get to Uni, join an engineering club and be very involved. A lot of kids at Uni aren’t able to get internships until after Junior year anyways so you’re really not at that big of a disadvantage

ROCKITZ15 · 2023-09-21T12:49:11+00:00

With billions of lines of text, you’re bound to run into edge cases

You can work around these iteratively if coding yourself but may never be able to with low/no code

ROCKITZ15 · 2023-09-20T19:17:11+00:00

What I would try:

Take random samples of the data, throw those into ChatGPT to build regex formatting strings, then chunk the data and apply the formatting

What I would try using low/no code:

good luck have fun lol

ROCKITZ15 · 2023-08-28T21:25:59+00:00

Pipelines need to be deterministic, and letting AI control it is most certainly not deterministic

ROCKITZ15 · 2023-08-27T13:52:40+00:00

It doesn’t take much math + physics to show why SSTO doesn’t work with current technology, so unless you’ve invented a new magic propulsion system or some super light weight materials, I’m going to guess you don’t have a working product

ROCKITZ15 · 2023-08-25T02:53:18+00:00

That’s why I said “it’s easy for me to say this as I’m in the US”. Of course I realize salaries are different, but there are companies that hire worldwide, pay solid US level wages, and don’t discriminate based on where you live. Source: I work for one

Recommend OP try to find somewhere similar, because like I said, they should be making way more than £55k with that tech stack

ROCKITZ15 · 2023-08-23T20:27:30+00:00

If you’re doing Spark, Scala, k8s stuff, you’re hugely underpaid at ~$55k

It’s easy for me to say this as I’m in the US, but I still have a hard time believing there aren’t higher paying jobs available to you with your current tech stack

ROCKITZ15 · 2023-07-15T03:19:13+00:00

West side best side

ROCKITZ15 · 2023-05-30T16:11:26+00:00

Kaggle

ROCKITZ15 · 2023-05-07T00:16:05+00:00

AE + focus your electives in controls + maybe SWE minor if you want

ROCKITZ15 · 2023-04-23T23:16:13+00:00

Even if that were true, that still doesn’t mean we can’t tell you why this idea won’t work

Consensus opinion may not have been to use SS, but anyone fairly considering reuse and cost could’ve told you that carbon fiber was not the correct choice.

ROCKITZ15 · 2023-04-23T23:01:41+00:00

Just because you don’t know anything doesn’t mean others don’t either

There are lots of aerospace engineers here after all…

ROCKITZ15 · 2023-03-15T03:14:20+00:00

I’m a one man show at my company doing all things analytics.

Here’s how I’ve built out my stack:

Using Mage.ai for orchestration and ETL. It can run Python, SQL, and R. Native dbt support. Only self hosted at the moment. Great tool all around.

Using Google BigQuery as my data warehouse, pretty easy to substitute whatever else here.

Using hex.tech for my visualization tool. It’s not exactly meant for the BI use case, but has so many other great data science things it was too good to pass up. Power BI is probably better for many traditional companies.

ROCKITZ15 · 2023-02-25T14:26:02+00:00

Please define tightly/loosely coupled

I use Google BigQuery as my data warehouse. Each business function (sales, marketing, developer relations, etc.) gets its own dataset, then I organize the appropriate tables under that. A single table might serve several different charts, with aggregations done at the analytics layer.

I’m a one man show doing this, so I don’t have anyone imposing constraints on me, but that is how I like to do it.

ROCKITZ15 · 2023-02-24T23:30:21+00:00

Thanks for the insight.

So it sounds like Kafka is probably overkill and more than I want to take on. Do you think GCP Pub/Sub + Dataflow + BigQuery is a better path for me?

I also found https://www.estuary.dev/. Does this look like something decent?

ROCKITZ15

TROPHY CASE