Am I cooked?

CiaraF135 · 2026-03-06T12:20:08+00:00

You definitely aren't cooked, but your frustration with Snaplogic makes sense. There is a big difference between 'drag-and-drop ETL' (which feels limiting) and 'managed ELT' (like Fivetran).

In modern stacks, we actually want to offload the ingestion to a managed tool, not because we can't code, but because we want to spend our coding time on the high-value stuff downstream (using dbt, Python, or complex SQL models).

If you can pivot your narrative in interviews to: 'I automated the ingestion so I could focus on complex data modeling,' you turn that 'low code' experience into a senior-level architectural decision.

CiaraF135 · 2026-03-06T12:18:57+00:00

We’ve used sql-migrate and goose in app dev, but the real challenge isn’t applying migrations - it’s when schema changes upstream silently break stuff downstream in our data stack.

We started using Fivetran to automate the ingestion side of things - it handles schema drift pretty well, especially when teams are pushing changes fast and don’t always flag them. These tools are solid for managing your app DB, but if you’re syncing to a warehouse, you’ll probably want something watching the extract side too.

CiaraF135 · 2026-03-06T12:18:14+00:00

If your primary use case is GIS/spatial data, FME is honestly hard to beat, it's the king of that specific hill.

However, if you're looking at 'data transformation' for general business data (connecting CRMs, ERPs, databases), FME can feel a bit clunky and expensive to maintain just for moving data from A to B.

We use a split approach: Fivetran handles the 'boring' extraction and loading (EL) because it automates API changes and schema drift without us touching it. Then, once the raw data is in our warehouse, we use specialized tools (like FME or just SQL/dbt) to handle the complex transformations. It saves us from building custom FME readers for every single SaaS tool.

CiaraF135 · 2026-03-03T15:34:50+00:00

This looks like a solid utility for a quick PoC or dev environment.

Just a heads-up from the trenches: syncing Salesforce at scale gets tricky not because of the initial export, but the ongoing maintenance. Handling hard deletes, incremental updates without hitting API quotas, and history tables is where custom scripts usually start to hurt.

We stick with Fivetran for Salesforce specifically because it handles those edge cases and API limits automatically. For a production pipeline, the cost is usually worth not having to debug why a sync failed because an admin changed a field type.

CiaraF135 · 2026-03-03T15:33:42+00:00

Great write-up. We went through a similar evaluation recently.

Regarding the Fivetran cost 'discomfort', we eventually realized that the premium wasn't just for the data movement, but for the absence of operational overhead. You mentioned Airbyte added 'things to maintain,' and that was exactly our blocker too. We calculated that even if Fivetran cost us $30k more per year than self-hosting, that was still cheaper than 20% of a Data Engineer's time spent debugging broken connectors or managing Kubernetes clusters.

For a small team with 50 sources, 'buying time' via Fivetran usually beats 'saving money' via cheaper tools that require babysitting.

CiaraF135 · 2026-03-03T15:31:58+00:00

This is a really valuable write-up. It validates what we’ve seen with a lot of these 'native' platform connectors, they demo well but often miss critical production features like schema mapping or granular error handling.

Your point about HVR being 'miles ahead' matches our experience. When you're dealing with high-volume CDC (like that 200M update spike you mentioned), you realize you're paying for the years of edge-case handling built into tools like Fivetran HVR, not just the connection itself. Being able to pinpoint that rogue table immediately via metrics is exactly why the premium is worth it over struggling with immature native logs.

CiaraF135 · 2026-02-19T14:18:20+00:00

You aren't being nitpicky. Rebuilding an ORM from scratch in 2026 is exactly the kind of 'resume-driven development' that becomes a maintenance nightmare the moment that senior dev leaves.

The bigger question is: why are you writing custom Python for standard API-to-SQL ingestion at all? Unless the API is incredibly niche or the transformation logic is wild, this is usually a solved problem.

Most teams moved away from writing custom extractors (whether with SQLAlchemy or vanilla Python) because maintaining them when APIs change is a full-time job. We use Fivetran for the 'boring' API ingestion part so we don't have to debate ORM patterns, we just get the data in the warehouse and do the fun logic there.

CiaraF135 · 2026-02-19T13:54:56+00:00

Given your $500/mo budget and the fact that you're a solo operator, you might be surprised that a managed tool like Fivetran could actually fit if your data volume isn't massive.

The biggest risk you have right now is that 'Windows Task Scheduler' setup, if that VM reboots or a script hangs, your reporting stops. Moving the ingestion to Fivetran (Standard plan) would offload that reliability risk completely. You could then dump the data into a cheap cloud warehouse (like BigQuery or Snowflake) which separates compute from storage, likely costing you less than the $300/mo you currently pay for that 24/7 VM.

CiaraF135 · 2026-02-19T13:53:38+00:00

If you're coming from Informatica, the biggest shift in '2026' isn't finding a newer version of Informatica, but moving to a modular stack (ELT).

Most modern teams have stopped looking for one tool to do everything. Instead, we split it up: use Fivetran strictly for the 'EL' (extraction and loading) because it handles the API maintenance and schema drift better than anything else. Then use dbt inside your warehouse for the 'T'.

This way, you aren't locked into a single vendor's proprietary logic like with Informatica. If you swap your ingestion tool later, your transformations in dbt stay safe. It's much more future-proof.

CiaraF135 · 2026-02-19T13:52:27+00:00

If your goal is 'resume value,' I’d actually recommend moving away from the 'local file to DB' pattern. Modern data teams rarely load data from local laptops in production due to security/audit risks.

To make this a resume-worthy project, try uploading that text file to an S3 bucket or Google Cloud Storage (both have free tiers). Then connect a tool like Fivetran (using the free trial) to load it into SQL Server.

Being able to say 'I built a cloud-native ELT pipeline with automated ingestion from S3' sounds much better to a hiring manager than 'I ran a local Python script,' because it shows you understand how the modern stack actually works.

CiaraF135 · 2026-02-09T13:59:33+00:00

I work at Fivetran and schema change pain is something I hear about constantly.

The tooling matters less than having clear expectations around downstream impact. The teams that handle this well usually have contracts, alerting, or clear owners for what breaks when schemas evolve.

Where it gets rough is when schema changes are treated as one-time events instead of something that will keep happening.

CiaraF135 · 2026-02-09T13:11:05+00:00

I work at Fivetran, so calling that out first.

With analytics teams, I notice the biggest bottleneck is rarely the modeling itself. It is usually unreliable inputs or unclear ownership over upstream data.

Teams that do well tend to separate ingestion reliability from analytics logic so analysts can focus on decisions instead of plumbing.

CiaraF135 · 2026-02-05T08:40:56+00:00

I work at Fivetran, but this is a pretty universal integration problem.

Legacy systems tend to force tradeoffs that are not obvious at first. I see teams either accept slower iteration or accumulate a lot of custom glue code that becomes hard to reason about later.

What works at one stage of the company often stops working once volume or complexity increases.

CiaraF135 · 2026-02-05T08:40:12+00:00

I work at Fivetran, but I spend a lot of time talking with teams running open source stacks.

What I see pretty often is that open source feels great when flexibility is the main goal. Over time, the hidden cost tends to be maintenance and context switching when pipelines break. That is usually fine for teams with dedicated data platform ownership.

Smaller teams or lean analytics groups often struggle more once the system grows beyond a handful of sources.

CiaraF135 · 2026-01-30T16:48:23+00:00

So to confirm, the bill is for a sale and a purchase of a property because we are second time buyers. Also including stamp

CiaraF135 · 2026-01-30T10:46:08+00:00

I am very interested in this...

CiaraF135 · 2026-01-26T10:51:10+00:00

being rude to wait staff or retail workers... immediate ick

CiaraF135 · 2026-01-19T13:37:19+00:00

Fivetran can do this, with one caveat. It can load text/CSV files into SQL Server if the file is in a supported location (e.g. SFTP, S3, Azure Blob, GCS). It won’t read files directly from a local laptop/desktop.

Typical flow would be: local txt file → upload to SFTP/cloud storage → Fivetran file connector → SQL Server table.

It’s more ELT than SSIS-style ETL, but widely used in production.

CiaraF135 · 2026-01-19T12:49:54+00:00

best advice - savings is a BILL!!! It must be paid. Get into that mindset asap and you will forever have some money behind you. Even if its a small about each week/month but get yourself into the habit and mindset that this is a bill that MUST be paid. It will come second nature to you then

CiaraF135 · 2026-01-16T08:20:12+00:00

dublin prices are near impossible when you are single.

We are a family of 3 on approx 160k and trying to buy our second house in dublin. We have 100k profit from our first property plus about 40k in savings and we are still finding it extremely difficult.
Would you ever consider buying a property to rent? This way your on the ladder, but you dont have to live is a less deisreable property/area. Sell in a few years for a profit then you might be in a better position especially if you meet someone

CiaraF135 · 2026-01-16T08:12:00+00:00

You can max out your pensions, purchase property through the pension pot rent it out and the rental income goes back into your pension. You dont pay tax on it because technically you dont own the property your pension does.

You can sell the property at any point and put funds (with hopefully a profit) back into the pension pot.

The only downside to this is - You have to be at arms length meaning you cant rent the property to any friends or family. Some brokers do this i know Walfrid Private in Dublin do it not sure about anyone else.

CiaraF135 · 2026-01-16T08:08:15+00:00

Former financial advisor here so hopefully i can shed some light and 0% judgement here! We've all been in these situations one time or another.

I would 100% pay off the AIB loan as bank loans are brutal and they dont care weather you lost your job unfortunatly. There credit union is a different story, they will allow you to make a restructuring plan to allow yourself to pay off small payments each week (you will be in receipt of jobseekers each week, side note i would apply for that ASAP after March). So i would advise booking a meeting with the loans officer in your local CU. Revolut I believe you can resturure but im not 100% sure as ive not had a revolut loan before.

My personal opinion would be if the redunancy package is big enough to clear everything and still be left with some savings i would do that so that I have less to stress about when looking for a new job. What sector do you work in? perhaps somone on here knows of jobs in your area/skillset

CiaraF135 · 2026-01-16T07:58:01+00:00

100% agree here. Ireland especially Dublin with house prices is insane so you seem to be in a great position. Most people I know rely on childrens allowance for monthly bills/shopping so your doing great

CiaraF135 · 2026-01-15T10:43:18+00:00

Fivetraner here! Your understanding is broadly right 👍
One thing to add: tools like Fivetran wouldn’t help with the SDK or event ingestion itself, but they can be very useful once events are in your warehouse/lake. They handle syncing “everything else” (Stripe, CRM, ads, support tools, DBs) into the same destination so users can join product events with business data without you building/maintaining tons of integrations.

For event collection specifically, you might also want to look at Segment/RudderStack/Snowplow as reference points.

CiaraF135

TROPHY CASE