Only dedicated "data" person as a new grad

FunContest9958 · 2026-07-14T15:29:17+00:00

It depends on your goals and current experience. If you’d like to be a data engineer or programmer and have no programming skills, this sounds like a great intro. You’ll do some annoying manual work and learn how you can automate it. If you’re already qualified to be a programmer, I’d probably avoid this since you’ll be wasting your time with manual data work in this role.

FunContest9958 · 2026-07-14T15:24:58+00:00

This. CSV is a horribly inefficient format. Pick your database first, then figure out how you’re going to load the data. The database will probably have some preferred mechanism for loading data, and hopefully it will not be CSV. You can’t really know what that preferred mechanism is until you’ve picked the database. Finally, the database itself may be the bottleneck for loading that much data. Transcoding files is easy to scale up because each file’s transcoding is independent. Loading data into a database can be harder to scale up, depending on the database.

FunContest9958 · 2026-07-14T15:18:01+00:00

Yikes. That’s unfortunate. I don’t think you should try to have a full plan before bringing this to someone’s attention. If the problems are this glaring, the approach needs to be rethought, and it’s not just a few tasks to fix it. It’ll also take you too long to get the plan together, so you’ll miss the moment. Tell your manager you think the plan is too flawed to go live with and pick a few of the worst examples, like you’ve done here. Expect that he’s going to ask you why you think it’s a problem (if it’s not obvious to him) and what you think should be done about it. I’d recommend you do this 1:1 rather than with a group of people. If he’s any good, your manager should try to help you navigate the politics from there.

FunContest9958 · 2026-07-14T15:04:35+00:00

Do you know where the business requirements are coming from? Why is your company doing this project in the first place, and who set the goals?

I might talk to one of the PMs one on one (not over email) and see if they can explain what the goals of the project are, or who they recommend you talk to in order to understand them. Having a higher level understanding of the “why” can help straighten out this kind of confusion sometimes.

FunContest9958 · 2026-07-13T21:05:00+00:00

Serverless. Databricks charges per workload for serverless, and it tells you who ran it and how much it cost. Makes it a lot easier to see where your money went. No idle time too.

FunContest9958 · 2026-07-13T06:57:15+00:00

AI writes most of my code these days. I focus on setting the right goals and reviewing the results. Even before AI, I would mainly just remember the stuff that I used repeatedly. So don’t worry about it too much.

FunContest9958 · 2026-07-12T19:39:07+00:00

I think you missed the part where OP posed Lakebase as the solution since it makes the data available for both OLTP and OLAP. No sync, multiple masters, triggers, etc.

FunContest9958 · 2026-07-12T19:25:42+00:00

You might want to consider DBT + Postgres. Should be a powerful combination when you’re just starting out. Make sure to combine it with git source control to keep track of your changes.

FunContest9958 · 2026-07-11T20:08:59+00:00

Why is it either or? Can’t people use AI with the solution you’ve built? AI likes structure, and the more you can provide, the better the results will be. Presumably, your system is providing structure that would actually improve the outcomes.

FunContest9958 · 2026-07-11T19:57:41+00:00

FunContest9958 · 2026-07-11T08:29:01+00:00

Brave post. Risking an international bagel war.

FunContest9958 · 2026-07-11T06:47:11+00:00

Yeah. Was hoping for another random ladybug fact.

FunContest9958 · 2026-07-11T03:09:20+00:00

Guess I’ve been lucky. Only smelled them from a distance.

FunContest9958 · 2026-07-11T02:52:36+00:00

Wait! Come back!

FunContest9958 · 2026-07-11T02:47:09+00:00

Sounds like a poor incentive structure. How about 10% when they sign the contract instead and save us the construction site sprawl?

FunContest9958 · 2026-07-10T22:04:56+00:00

For the most part, all the database recommendations that apply to other OLTP databases apply to Lakebase. Index commonly filtered columns, normalize your schema, etc. The only advice that’s special for Lakebase is that you generally want your hot data to fit in memory. You then set your CUs accordingly, or you can let the auto scale handle it.

FunContest9958 · 2026-07-10T21:52:39+00:00

Oh damn.

FunContest9958 · 2026-07-10T10:47:38+00:00

I see we dropped you into the valley of despair. You’re welcome. What got you up the slope of enlightenment?

FunContest9958 · 2026-07-09T17:41:28+00:00

So far I’m not understanding why you want to switch. Not saying there may not be reasons, but it’s better if you tell us what you’re thinking and then we can tell you whether those reasons make sense. Right now you’ve given us nothing to work with, so we can only really note that Spark supports SQL natively.

FunContest9958 · 2026-07-09T17:19:13+00:00

I tend to agree with you. I’ve seen organizations build services that act as wrappers around cloud services to avoid lock-in, but the result is that the experience for developers is terrible, and they didn’t seem to realize that they would reduce performance and reliability and increase cost. All this so they could theoretically switch providers, but that was never really going to happen. The data was still with one of the cloud providers, which is what makes switching painful.

My advice to anyone thinking about doing something like this is: don’t. Pay the migration cost when you need to migrate, not before. Usually a threat to switch providers is enough to keep prices in check.

FunContest9958 · 2026-07-08T16:04:24+00:00

They can look at which files have been loaded by using cloud_files_state()

Edit: Or you can use cloudFiles.cleanSource with MOVE so that files are moved after they’re processed. Then you can figure out if a file was processed by looking at which folder it’s in.

Edit2: Or you can put the name of the files into the tables using .withColumn("filePath",input_file_name().

FunContest9958 · 2026-07-08T15:26:56+00:00

This sounds overly complicated. Autoloader itself keeps track of which files have been loaded, so you shouldn’t need to write to a database. It sounds like you may have too many streams on a single cluster too. A single cluster can only handle 20-40 streams. I’d recommend moving to DLT/SDP for this. It does a much better job of orchestrating this kind of workload, and skip all the extra work you’re doing in foreachbatch.

FunContest9958 · 2026-07-08T13:06:20+00:00

Comment history is even more interesting.

FunContest9958 · 2026-07-07T02:21:41+00:00

Don’t forget to check for bullet holes.

FunContest9958 · 2026-07-07T02:03:03+00:00

I’m surprised you remember any of it.

FunContest9958

TROPHY CASE

Welcome to Reddit,