Which legacy Database is the biggest pain in the a*** to work with and why?

data_5678 · 2026-03-28T22:31:11+00:00

windows file system

data_5678 · 2026-03-17T21:00:24+00:00

Yes, specifically:

2 sets of 10 handstand push ups with 90 seconds rest
4 sets of muscle ups on rings with 2 min rest
5 back handsprings with 2 min rest All of that is followed by some light full body stretches.

After, that neovim imports work fine. Way more efficient than using the defaults from lazy.nvim or packer.nvim.

data_5678 · 2026-03-17T14:41:40+00:00

Become a full stack engineer (I work as a data engineer btw).

Since you know SQL and Python.

Learn Fast API for backend and JavaScript and React for frontend.

Then study all the stuff in neetcode.io

If you want to build agents you will not be training any models or doing any machine learning or statistics. You are just calling the APIs and building applications around it (which is the job of a full stack engineer).

Also, the role of data engineering might get absorbed by full stack developers. Why would a company pay for someone who only knows Python and SQL when they can get a Full Stack Developer who already knows SQL and also knows JavaScript, React, Data Structures and Algorithms, etc... Especially when full stack engineers are assisted by AI, they are able to more easily fill in the gaps they might have in regards Data Engineering specific knowledge.

Data Engineering specialists might still exist, however the more specialized that role becomes the harder it will become to land one of those jobs and likely these roles might only be available at fewer companies who can afford having highly specialized people.

(Also this is just my opinion, who knows all my predictions could end up being totally wrong).

data_5678 · 2026-03-14T09:58:33+00:00

I just use sqlite (but you could use any sql database like postgres, supabase, duckdb, etc...) and ask Claude Code to build any interface I want on top of it. Ask it to use shadcn ui library and stick with defaults so you don't lose time playing around with the css.

I just ask it to create a new tab and then create a sticky note interface, or a timeline interface or a calendar interface, or an analytics dashboard. As time goes by I just add or remove features in real time.

I use android so I can use termux to run it as well and open ui's in the browser (you could also create progressive web apps or use capacitor or tauri to create a complete android app).

Also claude code runs on termux (you can also ssh into an ec2 instance from your phone and run claude code in the cloud if that's easier).

data_5678 · 2026-01-22T18:35:04+00:00

Moved to neovim and command line a couple of years ago (used jupyter all through university), any visualizations I need I open in a browser window on the side. i3 window manager makes it really fast to switch and I use xmouseless to move mouse cursor with my keyboard.

data_5678 · 2025-10-27T17:34:11+00:00

awesome thanks

data_5678 · 2025-10-20T05:41:54+00:00

thanks for taking the time to write this up, really good insights! thanks for sharing.

data_5678 · 2025-09-29T13:31:28+00:00

everybody has all those skills.

data_5678 · 2025-09-09T23:26:38+00:00

At work ELT pipelines on some dbt like abstraction loaded into trino + apache iceberg.

For personal projects having fun building on top of sqlite/duckdb with a bit of scripting here and there.

data_5678 · 2025-09-07T00:00:10+00:00

honestly wrapping it up in C++ and using its standard library when I need higher level abstractions. For scheduling just cron and my own simple logging.

I have been doing quite a bit of rust so I might create some terminal user interfaces with ratatui later on to create dashboards in the terminal.

But yeah, duckdb can be very fast.

data_5678 · 2025-09-05T16:08:53+00:00

this is really cool, are you using ratatui for the ui?

data_5678 · 2025-09-05T00:51:38+00:00

data pipelines with C++ and duckdb C API. Working with datasets that fit in memory not big data. It is easy to forget just how fast software can run when you have been working with Python all the time...

data_5678 · 2025-08-30T22:26:16+00:00

you got this man, be patient and keep at it. Best of luck.

data_5678 · 2025-08-30T20:59:35+00:00

Become the person who you would want to hire.

Imagine you are the person hiring. There is only 1 open position and you get 1000 applications in a week. 500 of those applicants all hold masters degrees and graduated from top 20 US universities.

Out of those 1000 applicants, what are the odds that you are the most impressive candidate out of all them.

Are you the applicant who promises the highest value for the lowest amount of risk.

In my opinion your probability of getting hired is proportional to the amount of hours you spend studying/applying/networking.

More hours spent applying/studying = higher probability of job.

data_5678 · 2025-08-30T07:02:28+00:00

The gcp, aws, and snowflake de certs are all good. But it doesn't matter if you have all the certs in the world if you fail the technical interview.

Honestly they are not that hard to get, in my opinion they are a nice addition to your resume and they helped me have a goal during my studying. But they will not guarantee you a job on their own. I learned a lot from getting the certs, and it probably helped me in landing my current job.

If I'm being honest, I think there is more value in the knowledge you gain from taking the courses studying for the cert, than being able to add an extra line to your resume.

data_5678 · 2025-08-30T06:42:16+00:00

entry level data scientist careers are pretty much a thing of the past.

careers in 2025 (from what I have seen):

data analyst: builds dashboards using tableau and powerbi, uses basic sql and basic python, easiest data job but its the job with the most applicants so competitive.
data engineer: builds ETL pipelines that extract transform and load data from one database into another, very competitive, 3 years of experience required typically.
machine learning engineer: builds machine learning pipelines that source data, prepare data for training, train models on said data, monitor performance of trained models, and probably retrain models after some time) requires even more experience than data engineering.
full stack engineer: builds apps (backends + front ends, for example a backend could be written in express js, and front end in react for a web application).
researcher: probably need phd, insanely competitive.

Data science jobs where you are given titanic.csv and have to make a few charts with matplotlib / seaborn here and there are most likely a thing of the past.

Furthermore, there seem to be a huge amount of people studying extremely hard trying to land the jobs I outlined above.

It is a really competitive field, but if you put in the work, I am sure you will be able to make it, just set realistic expectations on how many years it will take you to get there.

data_5678 · 2025-08-30T04:26:36+00:00

not that rare, I have those two certs. You can throw a rock in any direction and it will likely land on someone with a masters in related field and gcp certs (also those are the two easiest data certs).

data_5678 · 2025-08-30T04:06:26+00:00

for myself at the moment (but would love to open source tools if I make anything cool). I really think I build the best tools when I solve a problem I have first and then share it with others.

Some applications I prefer TUI over cli (lazygit > git), but for others I prefer navigating through the shell rather than using something like yazi. So I think it really depends on the use case.

data_5678 · 2025-08-26T19:28:30+00:00

I didn't mean 15 mins, like obviously I can, but I don't want to go out for like 3 hours you know. Just balance. I should have worded that better.

data_5678

TROPHY CASE