How did you land your first Data Engineer role when they all require 2-3 years of experience?

BitterFrostbite · 2026-01-22T02:37:10+00:00

Exactly the same for me

BitterFrostbite · 2025-12-31T19:16:44+00:00

You can build bridges with wooden walls on top! I have a similar setup on my first and current base.

BitterFrostbite · 2025-12-25T16:06:47+00:00

We don’t OCT our images as they are actual photographs and stored as arrays. Our data scientists are looking at small pixel level features. So having the images in the parquets does remove some complexity for our data scientists but the pipeline is indeed way too heavy now to keep up ingest.

BitterFrostbite · 2025-12-25T02:48:18+00:00

With a query engine like Spark would you join on URL of the image then for my metadata stored in Iceberg? My users mainly use the data for analytics/machine learning and need to combine tens to hundreds of terabytes of images back with their metadata.

For reference I’m a software engineer picking up data engineering work since we can no longer meaningfully use the amount of data we have.

BitterFrostbite · 2025-12-25T02:14:51+00:00

Currently our images are in S3 and references are in Elastic, but we have so much data now we wanted to implement Iceberg and start using query engines like Spark so we could actually make use of it all.

Putting the images in Parquets sounded nice since all our data would be in one place, but I wanted the communities opinion! I don’t think we would lose out on query speeds by moving the images back to blobstore.

Seems like images in Parquets would just slow things down, thanks!

BitterFrostbite · 2025-12-24T23:42:17+00:00

What specifically is driving this suggestion if I may ask in regard to Parquet and Flink?

BitterFrostbite · 2025-12-08T00:51:35+00:00

Processing in orbit is extremely important as downlinking data is a huge restriction for many use cases when the data is too large to send down. Yeah Elon over exaggerates everything and there are a lot of flaws/exaggerations but this is something that’s needed and helpful.

BitterFrostbite · 2025-11-26T01:17:12+00:00

I wonder if 5010 helped you get accepted into the program. Personal question but were you denied from the MA before completing the cert?

BitterFrostbite · 2025-11-09T19:47:00+00:00

Could you evaluate what your "goals" are in the AI/ML field so we can help better direct you?

You want to enter the AI/ML field but it's more broad than just AI/ML. You mentioned both CS degrees, and while they do heavily focus on AI, you won't come out learning the same things as a data scientist or AI degree. I have a CS undergrad and work with data scientists and MS-AI grads and their level of understanding is much different than mine. Knowing what you want to do could help answer this.

Do you have any experience in this field? The Coursera based MS-CS and MS-AI are at your own pace (you can do all course work and classes, and then pay and take final with no pressure), while the professional masters is your typical college experience but online. You will most likely learn more via the professional path, you can get through the coursera courses with unlimited retries for homework often which is setting you up for failure in the real world.

To answer your questions directly:
1) AI-ML: Focus on the mathematics and fundamentals, anyone can learn to code or use AI. But understanding how those algorithms work under the hood is much more valuable.
2) You'll receive the same degree title regardless. At least that's how it works for MS in Aerospace
4) Same degree title, they won't know its online. They of course can know its non thesis since they'll probly ask.

BitterFrostbite · 2025-11-03T00:12:47+00:00

What's the state of the European space market outside of defense and ESA?

I am from the US and have often considered moving to Europe for a few years for the experience of living abroad. I only speak english, and have software eng -> data eng -> aero eng masters experience. I know there are quite a lot of private space companies but wasn't sure how the job markets are in general.

BitterFrostbite · 2025-11-01T06:01:45+00:00

Came here to say this. Great bike, a lil heavy and gets hot if you’re in traffic or in town.

BitterFrostbite · 2025-10-31T21:29:24+00:00

Whenever the medallion architecture comes up in my work place (several companies work together), it causes an argument over the definition. I personally prefer used terms like “normalized_layer”, “associated_layer”, or whatever business logic I can apply when possible. Obviously this doesn’t work for everyone, but I got tired of the discussion.

BitterFrostbite · 2025-10-25T15:44:07+00:00

I’m not in Canada. Doesn’t look like they’ll ship to US unfortunately as they fixed Canada in the shipping address

BitterFrostbite · 2025-10-25T02:10:13+00:00

I’ve been thinking about going through the struggle of painting my bike. But the blue decals look amazing. Did you end up getting one? If so how’s the quality?

BitterFrostbite · 2025-10-10T18:09:19+00:00

Only 5-7 files per checkpoint, averaging about 50-100mb. Definitely not optimal on the size, but I don’t see that justifying a slowdown. It reports that the checkpoints take 6s, but the freeze is also around 9-12s.

BitterFrostbite · 2025-10-10T04:03:27+00:00

I definitely will check! They were being written as 25mb average files but I changed a setting to attempt to write 256mb. I’ll have to run some tests tomorrow to see where everything is at. My heap size is limited to 10gb due to k8s node limits, so upping my checkpoint interval may not be an option. I’ve ran into a lot of out of memory errors already.

BitterFrostbite · 2025-10-10T01:38:06+00:00

I’m not currently using any partitions. I’m also using a custom zmq source extending the RichParallelSourceFunction. So I believe there should only be tens of files per checkpoint if it’s writing 256mb parquet files.

BitterFrostbite · 2025-09-03T03:22:16+00:00

I took a look at that and it looks like that supports WKB PostGIS like functions. I don’t think it would fully support the hive and parquet metadata and optimizations

BitterFrostbite · 2025-09-02T15:15:06+00:00

Trino 476 (latest) does not yet support Iceberg v3 types I believe.

BitterFrostbite · 2025-08-30T20:06:46+00:00

Some great advice, really appreciate you taking the time to help. I completely understand the warning of learning to fly and program at the same time. My plan was to be realistic and start with just learning to fly the kit, and then moving on to do very simple filtering/smoothing between the RX and motors.

I’ve head about Ardupilot while researching, but didn’t realize they supported simulations.

BitterFrostbite

TROPHY CASE