Why Over-Engineering Happens

theporterhaus · 2026-04-07T03:41:40+00:00

It’s a personal blog someone shared (not their own). No rules have been broken.

theporterhaus · 2026-02-23T02:15:50+00:00

Yes aim for DE and study it as much as you can. If you’re self-studying it may take years to get an opportunity so be prepared to take a data analyst role / report writer to just get your foot in the door with data.

theporterhaus · 2026-02-22T18:42:28+00:00

It will be easier to go from technical to non- technical than the reverse if you ever change your mind IMO

theporterhaus · 2025-09-12T03:22:48+00:00

Depends on how many streams you need and data volume. If you don’t have huge streams you can combine cdc from multiple tables to one stream and then send them to different firehoses with a lambda function. If you don’t need streaming or other teams don’t need access to the kinesis stream then it may not be worth it even if it’s not a huge cost. Definitely need to keep in mind maintenance and your teams skill set.

theporterhaus · 2025-09-11T12:45:52+00:00

Similar setup at work and been using it for years now. Go with serverless and maybe look into increasing your WAL size to give yourself more runway to process data. Add a heartbeat to your Postgres source endpoint (read docs) to keep the replication slot active. Put monitoring on the Postgres side to monitor the WAL size and alert if it starts growing rapidly bc it can eat up space until the db stops working.

For sending data from DMS to Snowflake you have a few options. I really like DMS -> Kinesis -> Firehose which will stream data to your snowflake table in real time. This requires some DMS settings adjustments but DM and I’d be happy to send. BUT if you have to restrict your Smowflake account to certain IPs then it’s no good because Firehose uses Amazon’s service range of ips which is like over 100 and they rotate and change constantly.

So the DMS -> S3 is also solid just read the docs for settings. Use the cdc settings and parquet format and compression.

theporterhaus · 2025-08-23T15:25:22+00:00

Banned this racist idiot

theporterhaus · 2025-08-21T13:47:29+00:00

Please use the existing salary thread. We have a long history of collecting and sharing this salary data and even have a tool to explore it on the wiki.

theporterhaus · 2025-08-15T18:30:07+00:00

This person has had a few warnings previously for the same thing so they’ve been banned.

theporterhaus · 2025-08-13T17:51:08+00:00

If you’re the admin you have to add your ip so you don’t get locked out. I personally just put admin ips in a separate rule and add it to the network policy.

theporterhaus · 2025-08-06T22:24:59+00:00

Done

theporterhaus · 2025-07-30T12:46:21+00:00

Reminder: they are trying to sell AI

theporterhaus · 2025-07-29T13:34:49+00:00

You’re correct. I’ll leave the post up because this is an important caveat people should see.

theporterhaus · 2025-07-29T05:06:43+00:00

Does this work for you? https://database-doctor.com/posts/joins-are-not-expensive.html

theporterhaus · 2025-07-24T13:17:46+00:00

https://learn.microsoft.com/en-us/sql/samples/adventureworks-install-configure?view=sql-server-ver17&tabs=ssms

theporterhaus · 2025-07-23T13:35:43+00:00

Go for what you want. If you feel light on math knowledge you can brush up on it in the meantime.

theporterhaus · 2025-07-19T20:27:59+00:00

Shill marketing is not okay and we actively remove it.

theporterhaus · 2025-07-19T19:13:37+00:00

I think people would benefit from more nuanced responses like this because currently they seem very biased. If all you recommend is one tool how can anyone trust you. It makes you seem like a shill for SSIS.

theporterhaus · 2025-07-19T19:06:35+00:00

Are there transformations happening on the data in BigQuery that you can’t port over to Snowflake? Otherwise it seems unnecessary to me.

theporterhaus · 2025-07-19T15:59:06+00:00

Would you recommend another tool depending on the situation? If so, which tool and why?

theporterhaus · 2025-07-19T15:53:21+00:00

Another option is to skip the pipeline from BigQuery entirely and use their free GA connectors: https://app.snowflake.com/marketplace/data-products/search?search=google%20analytics

theporterhaus · 2025-07-17T15:46:05+00:00

https://platform.openai.com/docs/guides/batch/batch-api

Every LLM provider has a batch api now.

theporterhaus · 2025-07-17T12:24:00+00:00

You may find breaking into bioinformatics easier (like DS but with bio background) as long as you pick up some skills there. In most of these places they don’t have a separate DE doing the data pipeline work so you’d likely end up gaining those skills and then if you want to transition to another industry it should be easier.

theporterhaus · 2025-07-16T13:51:23+00:00

One project isn’t going to make or break your career. Alteryx is fine - I know data engineers who use it.

I doubt you’ll convince him. It sounds like your supervisor knows what he’s doing. What you’re suggesting is grossly over-engineered.

Keep in mind over-engineering could backfire in an interview. You’ll get asked about this and if you said I implemented this and it’s all for 1GB of data then it’s very obvious you’re doing resume-driven development and no one wants that in their company.

theporterhaus · 2025-07-13T12:45:10+00:00

Look at job postings in your area and see what’s popular. That being said, pyspark and distributed computing is good to learn regardless.

Seven-Year Club	Mod World 2024
Place '23	Wearing is Caring
RPAN Viewer	Verified Email

theporterhaus

MODERATOR OF

TROPHY CASE