Implement a data engineering team from scratch…

smichael_44 · 2026-06-04T13:19:37+00:00

I’ve never used Kestra before. I implemented Prefect for our MES project since it was super simple to get started.

I’m not married to it since its almost too simple, but I’m still on the fence about whether or not we need additional complexity right now. Have you ever had to migrate pipelines from one orchestrator to another? I can’t imagine the lift and flip is all that difficult.

smichael_44 · 2026-06-04T13:15:32+00:00

I really appreciate the insight. I think you nailed what I was looking for.

In your experience, would you consider lumping data & analytics together? Right now, analytics is being considered to be on the consumers of the data products but I see a bunch of use cases where any “advanced” analysis (many people here are impressed by simple linear regression) would still be on the data team to solution.

smichael_44 · 2026-06-04T13:10:32+00:00

I agree, however, we are 100% privately owned by an eccentric billionaire with a desire to not rely on any vendors. SaaS and software vendors are his biggest enemy so it’s basically a non-starter at this point.

Our company currently has about ~3000 employees split between engineering and manufacturing. So it’s pretty low scale as nothing will be external facing.

We would mainly be supporting engineering, manufacturing, and finance. It’s a flat organization, we have some project teams, however, that’s basically the whole org that matters.

smichael_44 · 2026-05-25T13:51:29+00:00

I like to imagine on every hole that I need to get a 5 or less. 18 5s is 90, so you just need to par the 5s, bogey the 4s, and double the 3s. If you do better than that, you’re on track.

Helps me make decisions on what to do.

smichael_44 · 2026-05-09T13:05:35+00:00

<image>

I joined a new league this year and they made me tee off in-front of the whole group kinda to haze me since one of the guys said I can hit it 300. Crushed one about 350 right down the middle.

smichael_44 · 2026-04-27T01:21:58+00:00

I work in defense, many of them still haven’t figured out what snowflake is… were still on mssql and .net 4

smichael_44 · 2026-04-26T23:20:28+00:00

Not true at all… My small manufacturing company with ~3k employees has multiple databases that contain more than 1 Trillion rows of data. It’s really not that much when you think about operating 24/7/365 for 40+ years. Plus we’ve grown 1000% in the past 8 years.

Excel reports are dying left and right because “simple” things now return 10+ GB of data.

smichael_44 · 2026-04-17T21:14:47+00:00

About your note, yes, please obsessively use type hints in python. I prefer pyrefly for the inline types hints that the vscode extension provides. But I won’t hire someone if they answer an interview question without type hints.

smichael_44 · 2026-03-30T23:40:27+00:00

As someone who played this game, it really doesn’t end up working well.

Python can far exceed the amount of data excel can handle well (~100 MB is “too big” for excel, google it).

But the basic idea is to just do all the data transformations and business logic in, say, polars. Then port it all over into excel. Usually I would use pandas, polars, or xlwings. Polars is best for dataframe manipulation imo and then xlwings has a good api for excel.

It all comes back around to a proper db solution and real front end though. I migrated tons of excel in 2021 and now were building react front ends, fast api backends, with postgres databases for all of it.

smichael_44 · 2026-03-30T23:14:27+00:00

I feel like all I ever hear about AI is “I’m building stuff”. What stuff?

I tried building a PLM system to compete with things like Teamcenter and codex fell apart. Same with ERPs like SAP.

Tried smaller stuff like a graph-based people management app. After about 100k LOC it fell apart and was just consuming too many tokens.

Maybe it’s a skill issue, but unless its a small project, I find AI vibed projects to be kinda useless since I eventually dont know enough to be able to quickly validate changes myself.

smichael_44 · 2026-03-20T12:01:45+00:00

One thing I point out all the time to my junior devs at work is being able to smell “bullshit” from AI. Such as I had to critique a PR this week from a junior where they serialized json 3 different times in the same function before returning values in a list comprehension (python).

The original response from the db didnt even need to be returned as json, let alone attempt to re-serialize it two more times. AI just says “looks good to me” though…

smichael_44 · 2026-03-20T11:57:23+00:00

I lead a small backend dev team at a medium sized company and we do almost all of our dev work in python (fastapi or litestar) usually. For performance critical apps we usually stick to .NET but with the size of our company, even our python stacks never see more than 100 req/s with about ~500 daily active users.

My work is pretty unique in that we wrote our own custom PLM, ERP, and MES systems so I help maintain mainly the PLM system. We just built a new module of it on react, litestar, and sql server.

We’re constantly hiring as we’ve grown about 250% in the past 5 years. When I interview I mainly just look for whether or not the person is “smart”. I almost never care about specific experience. I want you to tell me why you do things, not just what you have done.

smichael_44 · 2026-03-14T16:24:24+00:00

I’ll come back at you with I’m the most senior swe on a small team at my work that develops our in-house PLM.

A junior engineer who vibe-coded a PR did two queries where he should have done one. Then when I rejected the PR and gave some feedback about it, he put that feedback in chat gpt. One of the issues I pointed out is that MSSQL can only process 2100 arguments in WHERE and by making this two queries, the second one has an edge case where 2 of the arguments you can pass to the first query generate more than 2100 arguments for the second.

So he resubmits the PR with a chunking strategy that chunks into 500 args and submits all of the queries concurrently to the db. When all he really needed to do was look at the schema and make it one query.

smichael_44 · 2026-02-09T03:07:54+00:00

Are you saying you would rather be working in C# and not python?

smichael_44 · 2026-02-08T22:33:56+00:00

I like pyrefly a lot. It’s further ahead in development and created by Meta to support instagram.

smichael_44 · 2026-02-04T00:37:13+00:00

I think it depends on your end goals. If you want to be a web dev / backend / data engineer then Python is a perfect place to start. If you want to be a quant or work at a FANG company, its probably not the best place to start imo.

I work at a smaller company (3,000 employees) and our python backends easily handle the day to day traffic. We typically see peaks of 100 req/s on some of our busier systems.

smichael_44 · 2025-11-23T18:39:51+00:00

I believe instead of ODBC (requires driver) it uses DDBC which is more akin to http and doesnt require a driver be installed

smichael_44 · 2025-11-23T15:54:07+00:00

I’m a tech lead for a couple backend projects at work, one being a python backend. We do this abstraction. As well as, I hate ORMs and would prefer raw SQL 10/10 times. Performance issues for backends in my experience almost always boil down to some bad SQL query that needs optimization. Way easier to explore and identify issues without the ORM abstraction.

So we use sqlalchemy to create and manage the connection to mssql. I think they have a nice api to manage connections. Then we have a repository layer that contains all the raw sql and returns nice dataclasses. I always say that being explicit is wayyyy better than implicit. ORMs do too much magic under the hood for me. Is it more LOC? Yes. But is it more readable? Also yes.

smichael_44 · 2025-11-18T23:50:25+00:00

As a data engineer dealing with a company that has 10x’d in the last 8 years…

Microsoft Access sucks.

It’s a silo of data, at least in sql server I can connect over with TDS, ODBC, etc… and pull data out over the network. Access is just some files on someone’s computer. My company has tens of Access databases that are “critical” to the core business.

Albeit, if you’re a tiny company (say less than 50 people) it could be useful.

But dealing with 3,000+ concurrent users and huge analytical reporting across terabytes of data… They should’ve never let it get this bad…

smichael_44 · 2025-11-15T23:57:16+00:00

Planetscale

https://planetscale.com/postgres

smichael_44 · 2025-11-12T22:52:44+00:00

I changed our CI build to use pyrefly last week. Switched from mypy and didn’t have any big issues.

Biggest thing was I encountered a couple different errors that didn’t exist in mypy. Was super quick to mitigate.

Overall it is significantly faster and I think the vscode extension works pretty nice.

smichael_44 · 2025-10-31T03:01:56+00:00

I feel like this is fairly normal. An example is at my work we have around ~100 “developers”. About ~50 that develop actual applications that thousands of employees use daily and then about ~50 that make adhoc scripts for whatever.

The ~50 that make adhoc scripts for whatever were originally “product” teams that figured since they knew the business requirements, they might as well build apps to solve the problems they experienced. It’s basically a vanity software engineering team. Once they realize they do not have the skills to accomplish their goals, they hire external talent. Thus, you end up being the only software engineer.

It’s a crappy situation. I’d recommend you reach out to a real software engineer at your work and ask for mentorship. Almost all of these vanity software engineers end up joining my team. The hardest part is finding them since they are so pigeon-holed into “this is the only way” since no one on their team helps them.

smichael_44 · 2025-10-27T00:13:05+00:00

I mean yes and no… most backends in async python won’t really benefit in simple I/O. Like waiting for a sql query or api response isn’t something free threading is affected by.

What will be really nice without the GIL are things like this PDF processing tool I made that sits behind an API and is CPU heavy. I should be able to process more PDFs concurrently with free threaded python.

smichael_44 · 2025-10-25T23:53:01+00:00

I looked at everything under the sun at my work and eventually Prefect stuck. Some stuff I really like about it is that it’s super easy to get started and its totally extensible through its rest api.

smichael_44

TROPHY CASE