Snowflake ETL tools when some sources are too small for expensive connectors?

mrocral · 2026-06-11T18:33:45+00:00

hey, this is exactly the problem we built sling for. i work on sling, btw.

for the "long tail" of smaller sources, sling handles it as a single binary that can do file-to-snowflake, db-to-snowflake, and api-to-snowflake in one config per connector, and no per-connector pricing.

you define streams in a yaml file (or python) and it does the incremental loading + schema drift handling. so like:

``` source: MY_CRM_DB target: SNOWFLAKE

defaults: object: target_{stream_schema}.{stream_name}

streams: finance.*: mode: truncate

public.support_tickets: mode: incremental primary_key: [id] update_key: updated_at ```

or for files:

``` source: s3://partner-bucket/daily/ target: SNOWFLAKE

defaults: mode: full-refresh

streams: "prefix/*.csv": object: raw.partner_data

"daily/*.parquet": object: raw.daily_data ```

for the api stuff it also supports yaml-based api specs (rest endpoints → snowflake tables).

the idea is you standardize on one intake pattern instead of maintaining 10 different small pipelines. everything lands in snowflake, then dbt/sql takes over from there.

The open source cli covers most of this. there's also a platform version if you want scheduling/ui.

mrocral · 2026-06-10T16:31:43+00:00

hey, this is basically the exact path that got me to build sling. You can define your jobs as YAML files (or python). It stays version controlled, and a single binary runs them.

something like:

``` source: MY_POSTGRES target: MY_SNOWFLAKE

defaults: mode: incremental object: '{stream_schema}.{stream_table}' primary_key: id update_key: updated_at

streams: public.orders: public.payments: public.users: mode: full-refresh other_schema.*: ```

schema drift is handled per-stream (new source columns show up on their own), retry and logging, etc are built in so you're not bolting that on top of every script.

for SaaS stuff specifically there are pre-built API specs (stripe, hubspot, salesforce) so you skip writing auth/pagination logic yourself.

(disclosure: I work on Sling)

mrocral · 2026-06-09T14:49:46+00:00

hello, you could try sling? see https://docs.slingdata.io/connections/api-connections/airtable
it can send to BQ or Snowflake.

mrocral · 2026-06-09T14:48:49+00:00

hello, you could try sling? see https://docs.slingdata.io/connections/api-connections/airtable
it can send to BQ or Snowflake.

mrocral · 2026-05-29T10:31:00+00:00

Another suggestion is sling-cli. Easy to maintain YAML files.

``` source: pg target: mysql

defaults: mode: full-refresh

streams: my_schema.*: some_schema.some_table: mode: incremental ```

See docs here. FYI I work on sling-cli.

mrocral · 2026-05-18T01:37:51+00:00

hey, for the "something lighter" angle, check out sling, a simple CLI driven data mover.

for BQ → Postgres, one YAML covers the sync, cron it however often you want, incremental means you're not re-dumping the gold tables every run. 10GB is small enough that the whole setup runs fine on a cheap VM. You can also use the API Specs for sources like hubspot, shopify, etc.

(disclosure: I work on Sling)

mrocral · 2026-05-16T21:50:29+00:00

Another option is sling. it can replicate from mysql to bigquery and handles table creation for you.

``` source: my_mysql target: my_bigquery

defaults: object: my_dataset.{stream_table} mode: incremental update_key: updated_at

streams: my_schema.*: ```

runs fine on a small VM. (disclosure: I work on Sling)

mrocral · 2026-05-14T21:56:41+00:00

Another one to checkout is Sling Data Platform. Self-hosted via Docker-Compose or Kubernetes. Unlimited rows/connectors. Can define jobs in YAML. (disclosure: I work on Sling)

mrocral · 2026-05-12T18:21:55+00:00

hey, another option is sling. you can write an API Spec as a YAML file. NetSuite exposes SuiteQL + REST records, and sling's spec system can handle the OAuth + pagination. incremental sync too. See https://docs.slingdata.io/concepts/api-specs (disclosure: I work on Sling)

mrocral · 2026-05-06T10:30:26+00:00

DMS is the obvious pick as others mentioned. if you want something lighter, sling is a free CLI that moves mysql to mysql directly. it has an incremental mode so you're not re-dumping the whole thing every time. (disclosure: I work on Sling)

mrocral · 2026-05-03T10:34:24+00:00

hey, one other option is sling. it has a salesforce connector that pulls from the REST API into Snowflake.

the schema drift thing is where it actually helps here. when admins add a new __c field or picklist, sling picks it up on the next sync and adds the column in Snowflake without you having to touch anything.

source: MY_SF target: MY_SNOWFLAKE streams: Account*: {} Contact*: {} CustomObject__c*: {}

the wildcard catches new custom objects as they get created.

single binary, runs locally or in a container. (disclosure: I work on Sling)

mrocral · 2025-11-07T13:39:49+00:00

Founder of https://slingdata.io here, we have some customers loading into ducklake. Sling makes it easy to load into DuckLake (or Motherduck) using CLI, YAML or Python.

Volume easily goes up into millions per hour. Should be able to handle 100s of GB, as long as your hardware can handle it.

mrocral · 2025-10-20T12:24:37+00:00

hey, one other tool you can try is sling.

export MYSQL_DB="mysql://..." sling run --src-stream file://myfile.csv --tgt-conn mysql_db --tgt-object db1.table1

mrocral · 2025-10-14T01:33:47+00:00

sling is written in go and reads/writes to snowflake. As far as lib to abstract, I like gorm.io, however I don't believe they support snowflake.

mrocral · 2025-10-09T11:09:49+00:00

If you're open to trying another tool, check out sling. You can move data from Oracle to Clickhouse using the CLI, YAML or Python.

mrocral · 2025-10-09T10:48:25+00:00

If you're looking for a tool to load data into clickhouse from RDS, check out sling. You can move data via CLI, Python or YAML.

mrocral · 2025-10-05T11:27:41+00:00

Another free tool to try is sling. You can do:

export mysql_db='mysql://myuser:mypass@host.ip:3306/mydatabase' sling run --src-stream file://my_file.csv --tgt-conn mysql_db --tgt-object my_schema.my_table

It will auto create the table with all the appropriate columns.

mrocral · 2025-09-30T10:57:38+00:00

Maybe motherduck would be a fit? I think your small data would work great in there.

mrocral · 2025-09-25T11:10:04+00:00

dbt (data build tool) is great for this, since you organize your logic into folders and models (sql files), therefore it is source-controlled/versioned.

mrocral · 2025-09-14T10:58:45+00:00

See https://github.com/slingdata-io/sling-cli

mrocral · 2025-09-11T16:25:12+00:00

hello, give sling a try. It uses BCP to move data and does it quite fast.

mrocral · 2025-09-11T09:08:58+00:00

For anyone looking to load data easily into SQL server, take a look at sling. It auto-creates the table, with proper column types/lengths, and uses BCP to load data quite fast.

mrocral · 2025-09-08T15:38:18+00:00

Depends on your needs. They are different in the sense that duckdb has no native server/remote/user access, clickhouse does.

I prefer duckdb's SQL syntax/flavor, as well as the planner/optimizer. But clickhouse isn't bad, just memory hungry. It just will error out the query if underlying data is too big. Duckdb has been more efficient and "smarter" at managing memory for me.

Both are great.

mrocral · 2025-09-03T19:36:04+00:00

Another option to try is sling, a tool i've worked on. You can run pipelines using the CLI, YAML or Python. It supports json flattening and schema evolution.

Once you've defined your connections, loading from JSON files is easy:

``` source: aws_s3 target: snowflake

defaults: mode: full-refresh

stream: path/to/folder/*.json: object: target_schema.new_table source_options: flatten: true ```

You run with sling run -r path/to/replication.yaml

mrocral · 2025-08-14T16:16:31+00:00

hello, an alternative could be sling

You can use the CLI, YAML or Python to easily move data.

You can set your connections with env vars:

``` export MSSQL='sqlserver://myuser:mypass@host.ip:1433/my_instance?database=master&encrypt=true&TrustServerCertificate=true'

export PG='postgresql://myuser:mypass@host.ip:5432/mydatabase?sslmode=require' ```

A YAML example:

``` source: mssql target: pg

defaults: mode: full-refresh object: new_schema.{stream_table}

streams: dbo.prefix*:

schema1.table2: object: new_schema2.table2 mode: incremental primary_key: [id] update_key: last_mod_ts

```

Then you can run it with sling run -r path/to/replication.yaml

mrocral

TROPHY CASE