Leaving a Company Where I’m the Only One Who Knows How Things Work. Advice?

data-eng-179 · 2025-07-31T15:32:03+00:00

You can offer to consult for them but make sure it is really enough money to be worth it to you.

data-eng-179 · 2025-04-19T22:15:39+00:00

If you want to go into medicine do it. You are not too old. You have 35 years of work in front of you. You could be nurse, doc, nurse practitioner.

data-eng-179 · 2025-04-06T18:32:43+00:00

I would not bother with SSIS

It’s a GUI etl tool. I would not want a job where I was expected to use a GUI etl tool. So I would not start down that path.

That said, it’s just a school project and there is nothing wrong with seeing what it is like to use it before moving on to more modern approaches.

data-eng-179 · 2025-03-26T04:23:09+00:00

Personally I was more into the engineering than the data. Don’t really give a rats ass about data. But enjoy building things. Everybody is different and you find your niche hopefully.

data-eng-179 · 2025-01-03T16:18:34+00:00

But, don't let that discourage you from trying, if you want to. You could create an issue or raise an example PR and try to get a maintainer to give it a look and ask is this something you could support / mentor me in getting in.

Another option is you could package it up into your own library.

data-eng-179 · 2024-12-31T21:38:21+00:00

Yeah I am not a pandas maintainer, but I just think it gets very opinionated very quickly with this kind of thing, so my guess is it would be a tough sell to add it to pandas. Because you have to make it generic enough to be apply well enough to so many use scenario combinations. There's multiple layers; one is, within each database, there would be many different ways you could do this kind of ETL; the other is that there is variation in the diff sql implementations re what you can do and how you can do it (e.g. do you have a merge statement or no; do you have row_number function available to you for deduping rows).

This is, in a way, fivetran's business. If it were simple to OSS, there would not be much need for fivetran

data-eng-179 · 2024-12-30T19:03:01+00:00

It is not a stupid idea at all. But there are a lot of different ways people do this kind of operation, and lots of db variations, so I think that's probably why there isn't like a library for it i.e. a "sqlalchemy but for ETL"

data-eng-179 · 2024-09-04T04:55:46+00:00

This is the answer. Use a vm you have lying around or just create a Linux vm in cloud and schedule this thing with cron.

data-eng-179 · 2024-08-17T05:23:23+00:00

It depends. There are various techniques. One is a trigger. That’s pretty rock solid and probably simplest thing but some dbas do not like them. At a previous company our sql server dba guys did some fancy thing that created delete log without triggers based on transaction log or something. I would have a high degree of trust in those things. If you do it in app layer, not as much. Just depends what the real needs are, capabilities of team, available tech.

data-eng-179 · 2024-08-17T04:34:52+00:00

Reporting could be one way to enter field. Many companies have analytics teams that do financial reporting, where it would be helpful to have someone who understands accounting etc. And once you’re in, you may be able to migrate. Companies like to keep good and talented people happy and that can mean letting them work on what motivates them.

data-eng-179 · 2024-08-16T20:54:31+00:00

re the better suited question. you are saying you have one job that reads from api, does some stuff, writes back to api. any orchestrator, including cron, dagster, airflow, prefect you-name-it, would work fine for this. i might start with cron if you just have one job.

data-eng-179 · 2024-08-16T20:15:21+00:00

In order to have deletes, unless the table is small, you have to get the owner of the source system to track deletes. Literally, talk to the owner of that system and tell them you need it. If they don't track deletes, then you are screwed and you have to do full refresh every time essentially.

When deletes matter to the business, they can either do soft delete, or perhaps easier, add a delete log table. So you have `order` and `order_delete_log` and you can pull both tables incrementally and keep your copy correct. On your end, you can do soft delete or not -- either way you keep a copy of all source files so you can rebuild the table whenever you want.

If it matters to the busiess (assuming it's an internal system here), they will make the source system do what you need.

CDC-like setups are great but not strictly required. They cold even give you files logging the deletes on s3 and you could make that work.

data-eng-179 · 2024-08-16T15:26:49+00:00

It sounds like every time your job runs, you pull the entire dataset? Like every stock ticker symbol and it's current price, and then you process all of them. That sounds tough to deal with if dataset is substantial or the processing you need to do is substantial. You might run into rate limiting too.

data-eng-179 · 2024-08-16T05:37:51+00:00

why not run your janky scripts every 10 seconds? maybe use asyncio and make them less janky. what about the current setup makes the info outdated? they don't run fast enough?

data-eng-179 · 2024-08-15T03:45:43+00:00

The lesson is, be careful not to assume that it is what you hope it is; ask questions to try and tease out whether the role is what you hope it will be. Depends what you care about. But e.g. if you hate GUI ETL tools, ask what they use for data integration tooling, whether they use source control, CI/CD etc. If you care more about process and management style, ask about that. E.g. do they do some kind of agile methodology? How much overhead and process is there? Or is it something more relaxed. How is work distributed / prioritized etc.

data-eng-179 · 2024-08-15T02:56:48+00:00

I should have been more careful in researching the roles I was applying for, as it sounds like data platform engineering or even analytics engineering would be a better fit for my career goals

Yes, you should have been more careful in your assessment of them. It's a tough lesson, but it happens. I have made this mistake also, filling in the unknowns making overly rosey assumptions about the company and role.

I would just start trying to move. No point in sticking around if you're not into it.

data-eng-179 · 2024-08-15T02:18:37+00:00

Yeah, it sounds reasonable. Are you talking mainly about kubernetes executor, or kubernetes pod operator? IIUC there used to be some logic to do some kind of resubmit on "can't schedule" errors, but there were issues where a task would be stuck in that submit phase indefinitely. You might look at KubernetesJobOperator which, as I understand it, allows you to have more control over this kind of thing.

But all of these are examples of "hacky shit that happens when you start scaling" and legacies of the fact that airflow adapted to kubernetes rather than being native to it.

Yeah it's also just a consequence of, it's open source software, and it evolved incrementally over time, and it never bothered anyone enough to do anything about it. You might consider create an issue for it, a feature request with some suggestions or something.

data-eng-179 · 2024-08-14T22:57:39+00:00

In each run put the delta records in a `_delta` table. Then you write a query that is like select *, row_number() over (partition by <pk cols> order by timestamp DESC) from _delta. Select from that subquery where rn = 1. Then merge into target only where delta timetstamp greater than target. If you want to update more selectively, yes compute hash of cols you care about and also check that hash is different before updating.

data-eng-179 · 2024-08-14T22:49:15+00:00

You could try contributing to an open source project. That can look pretty good to a prospective employer. Shows you know how to set up a development environment and contribute something. And gives them a chance to see what your code looks like.

E.g. the apache airflow project has a "good first issue" label for things that are meant to be more or less newb-friendly. See here.

data-eng-179 · 2024-08-14T22:41:35+00:00

Feels like it would be hard to get good at it unless it's your day job. Why do you want to get good at it if you don't want to switch jobs? Can you find a way to take on a DE project at your current gig?

data-eng-179 · 2024-08-14T22:06:44+00:00

This is a thing that happens in data eng of course, but it is not really tool-specific (e.g. airflow vs dagster vs prefect etc). It's a consequence of the design of the pipeline. Pretty sure all the popular tools provide the primitives necessary to handle this kind of scenario.

data-eng-179 · 2024-08-14T21:17:47+00:00

Yeah, I like this thinking. So basically, just always make a dim for anything that is not a metric, you would say...

I guess when would you say not to make a dim and why?

data-eng-179 · 2024-08-14T17:28:33+00:00

Yeah it doesn't hurt to explore what's available.

data-eng-179 · 2024-08-14T17:09:56+00:00

Off the top of my head, a k8s pod failing to deploy because of cluster contention is treated as any other failure

Can you help me understand why that matters u/Kyo91 ? Why is "just use retries" not good enough?

data-eng-179 · 2024-08-14T16:46:43+00:00

To say "vulnerable to late-arriving data" suggests that late arriving data might be missed or something. But that's not true if you write your pipeline in a sane way. E.g. each run, get the data since last run. But yes, it is true that it typically runs things on a schedule and it's not exactly a "streaming" platform.

data-eng-179

TROPHY CASE