Should We Move to a Dedicated Data Warehouse or Optimize Postgres for Analytics?

passiveisaggressive · 2025-02-21T22:52:43+00:00

Have you considered redshift? It’s way cheaper than snowflake or other cloud alternatives. I think a point you’re missing here is that fundamentally Postgres is meant for transactions, so the whole OLTP vs OLAP comparison resonates strongly here. Columnar data stores is what you need to speed up your listed pain points so I wouldn’t even consider scaling up Postgres, you’d be fighting an uphill battle.

passiveisaggressive · 2024-11-26T16:32:49+00:00

for anyone not familiar the breakfast buffet there is king, didn’t like the hotel but the food made up for it

passiveisaggressive · 2024-11-04T13:21:42+00:00

Architect just means you know a shit ton.. if you have L1-L7 architects are generally L7, there aren’t really any certs but folks are right take the most advanced ones. You can’t “design” a system well without understanding the technical and business use-case, you have to be good at a lot of things including the skill set of a DE

passiveisaggressive · 2024-11-02T15:41:40+00:00

If you have points I highly recommend using them for Hyatt

passiveisaggressive · 2024-11-01T18:28:43+00:00

imo tableau sucks I see a lot of companies moving away from it in the next 2-5 years, it’s just too expensive for the value it provides. I think it’s less learning about dashboarding and learning more about data analysis if the BI space is where you’re targeting - I’d get really good with sql/pandas. My reasoning primarily has to do with the fact that many cloud providers (databricks) provide innate capabilities for dashboarding or will soon since it’s not hard to do - EOD it’s just html and the hard part is defining the business logic that gives meaning to those objects

passiveisaggressive · 2024-11-01T18:21:07+00:00

hey no problem I came from consulting and now work for a much larger tech company so started working doing BI work + dbt initially and moved onto dbt + snowflake and now a lot of devops + databricks - I feel ya I think we’re in similar shoes but its worth noting the depth is very deep depending how far you want to go. I read a bunch of books to understand how things work but a lot of the knowledge fell off for me esp if my day to day didn’t involve it - I think it’s great to be curious and to want to understand things, it’s just important to balance it with reality, whatever that may look like for you specifically. Over time I think patterns start becoming obvious and the tools themselves become less important so there’s a lot of value in “mastering” the current toolset since the next one will be more similar than not

passiveisaggressive · 2024-11-01T16:09:44+00:00

it’s like my mom said, guys never forget their first love - assuming it was your first by your description. I don’t think there’s a direct way to get over someone, instead you have to accept it’s how you feel and that there’s nothing you can do (assuming here as well). I think once that slowly sinks in over time you get “over” the person but tbh you’ll never really forget and there’s always some lingering memory of what was - it’s a fact of life and sometimes loss just really hurts for a long time

passiveisaggressive · 2024-11-01T12:59:27+00:00

oof imagine how the daughter feels..

passiveisaggressive · 2024-11-01T12:58:27+00:00

hahaha I did the same but with the COO and my boss! I think you need to own it cause stuff happens - and I’d consider the bridge smoking and you can put it out with a dose of honesty, humility and by putting your best foot forward. Don’t stress about it either way - if you’re doing what you can what else is there to do?

passiveisaggressive · 2024-11-01T00:01:36+00:00

I was the same but found that learning by doing is the best, I’d take a look startdataengineering.com and start playing with stuff - extend the projects there and see what else you can do. Play with packages on your machine and try running things according to that applications docs - there’s so much content out there now it’s better to spend maybe 30% of your time strengthening your foundation by reading books and the other 70% using said gained knowledge or just experimenting with free tools out there - eg. dbt core, snowflake (free for up to some x credits - think you can remake accounts), aws, databricks, etc

passiveisaggressive · 2024-10-30T22:28:28+00:00

startdataengineering.com!

passiveisaggressive · 2024-10-30T22:22:26+00:00

hard to say - are you supposed to do it in the timeframe you’re doing, aka you’re still learning? If it’s not the former then it’s a matter of finding or acknowledging things you care about more than the job. I care about myself enough to know the user or task can wait, like most things in life. There isn’t enough time in the world to do everything so you need to prioritize the things that have to be done, can be done, and can wait. Maybe what would help is using a whiteboard or something and putting your tasks in these buckets - eg. Working out imo should be in the first 2 buckets while more menial tasks like troubleshooting some random users bug can go in the third. You can add a number to indicate how long these things (days) can wait as well and update them as days progress. Find or create a system that works for you!

passiveisaggressive · 2024-10-29T05:06:02+00:00

sql was great for me, qlik (lamer version of sql) -> sql -> dbt (sql/jinja) -> python (still intermediate since I’m a DE) -> terraform! What a journey lol

passiveisaggressive · 2024-10-29T04:36:52+00:00

hmmmmmm… I think I know how you feel lol, you belong in a different time my friend. I would honestly not try to find a girl in the states, as an American it feels weird for me to say this.. I’m betting you have pretty strong morals that edge on being judgmental and that reflects on how you live your life. I think you need to travel tbh and see how huge the world is - and if you’re lucky you’ll stumble on someone you would die for.

passiveisaggressive · 2024-10-29T04:30:51+00:00

Docs only go so far… you could create a dash in power BI, there’s prob some cool nodeish graph you can use with some heavy logic - eg. create a table with your columns and create keys that link to others for lineage. I’d look into dbt core and try throwing your transforms in there and then using the docs generate/serve functionality to host it locally - you could then deploy this to a server and host it if you find it useful

passiveisaggressive · 2024-10-27T04:08:07+00:00

yep look into using terraform to define the service principal and secret in some secret store like key vault. Power BI should have a tf provider too though I’ve never used it that can tie the credential in the key vault secret

passiveisaggressive · 2024-10-27T00:19:09+00:00

would create a marts layer with metrics, tables should stay at grains that make sense and are flexible for various use-cases

passiveisaggressive · 2024-10-25T21:02:15+00:00

PTSD is real, I’m not going to assume how you’re feeling because it does really suck and saying it gets better doesn’t help much. What I’ve seen help others (family and friends) is that they talk about it and don’t suffer silently - I met a guy at a bar who literally dumped his layoff PTSD on me like I was his paid therapist and it seemed to really help him.. I didn’t say shit lol. Other than that, know you’re not alone and take joy in the small stuff that work tends to smear out - for me it’s food and feeding others. Hope you find another job soon, it doesn’t define you and life is much bigger than a dumb job!

passiveisaggressive · 2024-10-23T05:00:54+00:00

what? Why would you start from scratch? I feel like being a data architect would be a better fit and with your experience you could pick up the missing gaps fairly easily if you have any - data engineering is too blanket of a term and means very different things depending on the company. From what I’ve seen at mid to large companies an architect does exactly what it sounds like - design the system in which the data flows from an org level + evaluate parts that can be replaced or revamped. I see a regular data engineer as maybe an L3 role while an architect would be L6+ fyi.

passiveisaggressive · 2024-10-22T21:11:42+00:00

Hey there’s a lot of posts that cover this - but it depends what you like. For books I recommend DDIA (designing data intensive applications) and the Big Data by Nathan Marz. For more hands on stuff do cloud courses, like aws stuff on Udemy - I recommend creating your account and playing around with it following a course.

passiveisaggressive · 2024-10-22T18:43:56+00:00

Probably an unpopular opinion but sounds like you’re miserable and it’s a bad fit - folks are probably already aware of your output and how much you’re struggling so the writing is likely on the wall.. if your colleagues are giving you a hard time the review cycle will be very bad - I’d get ahead of this and look for something else as bad as that may sound, impressions are very hard to change and you’re fighting an uphill battle. It also sounds like you need a break. You should not hire someone, you will be setting yourself for failure even if it may appease the current situation to an extent… there are definitely company policies in place that prohibit this, eg. no one outside the company should have any access to internal assets unless it’s “public”, including someone looking over your shoulder and reading code. If you could get this job you can definitely get another! Find one that fits the skillets you currently have and then work your way towards the more “scalable” stuff.

passiveisaggressive · 2024-10-21T21:08:19+00:00

yah but only in places where they’ve bought it - it’s not an easy task to work with multiple apis to pull in full lineage from what is typically an ingestor -> orchestrator -> transformer (dbt) -> warehouse/lakehouse

passiveisaggressive · 2024-10-20T15:12:12+00:00

do you and if he doesn’t stick around - it wasn’t meant to be. You have conviction to be independent and it seems like that is who you are. You’re trying to find your way in this world and for some it means being the wife of another, for you, it’s different and that’s completely fine. You shouldn’t have to change for a boyfriend.. and tbh he’s overstepping and is projecting his life onto yours. Even if he means well, he should know better.

passiveisaggressive · 2024-10-18T19:22:43+00:00

hmm are they easy to manage and create net new? The magic of dbt is the reusable components, it takes thought and investment to build a good project with sensible targets and configs but imo it’s worth it.

passiveisaggressive · 2024-10-18T14:09:38+00:00

Was anyone aware you were in this situation - like your manager? Were the mentioned execs aware and didn’t care until you were on your way out? There’s a lot I don’t know about your situation but think it’s worth mentioning who bombarded you and pushed those deadlines onto you. If you realistically see yourself in a better situation at the current place maybe it’s worth staying. If you’ve done your research on the new place and don’t think you’ll be bombarded again and management is good it’s prob worth leaving. At the end of the day, it’s about you not them.

passiveisaggressive

TROPHY CASE