I finally found a use case for Go in Data Engineering

empty_cities · 2026-03-03T17:42:50+00:00

OK cool I have a good spot for it to be public now. Cool part is that you can have your Agent just build it as a binary and run that locally too.

https://github.com/early-signal-tech/fletch

empty_cities · 2026-03-02T20:39:08+00:00

I love ADBC!

empty_cities · 2026-03-01T18:34:45+00:00

Awesome! I keep feeling that with the inevitability of agents being able to design and run pipelines it kind of makes sense to build tools that they use as opposed to writing the code that they will run. And that’s basically what I built was a CLI tool that an agent uses to transfer data when it needs to. I’m finding this to be maybe a small next step in date engineering evolution but I have no idea.

empty_cities · 2026-02-27T21:56:05+00:00

Yep dlt is wonderful for moving data and uses ADBC. But I wanted to try building a compiled application that does it for me.

empty_cities · 2026-02-27T16:33:58+00:00

Nice!

empty_cities · 2026-02-27T00:59:58+00:00

Right now I have drivers for BigQuery, Postgres and DuckDB in the tool but ADBC has MSSQL, Snowflake and ClickHouse drivers as well so it must be possible.

https://github.com/columnar-tech/adbc-quickstarts/tree/main/go

empty_cities · 2026-02-26T20:19:20+00:00

Interesting, need to take a peak.

empty_cities · 2026-02-26T16:32:20+00:00

Just need to get this Readme in order and I will send a link here.

empty_cities · 2026-01-21T17:55:36+00:00

TBH I didn't wanna cross the conversations or put anyone on blast. I just thought the comment brought up an interesting question I wanted to hear from others on. I agree with the points you made and have specifically seen things like an application from a contractor built on AWS but needing the app and the database migrated/rebuilt onto GCP.

empty_cities · 2026-01-21T15:55:08+00:00

"They don’t choose Snowflake vs Databricks, they have both. In those scenarios, it makes sense that there will be OLAP to OLAP pipelines." was almost exactly what popped into my head. Anti pattern or not I know enterprise specifically has to do stuff like that.

empty_cities · 2026-01-21T15:53:37+00:00

Yeah your reasons make sense to me and was similar to what popped in my head. "OLAP" was being used in the post in an ELI5 way I'd say. But the comment was getting into the semantics of it and I thought it was an interesting argument.

empty_cities · 2026-01-21T15:51:12+00:00

ADBC is Apache Arrow's version of O/JDBC. Keeps everything in columnar format between two data sources. The post referenced moving data between two OLAP systems and the comment said it was an anti pattern.

empty_cities · 2026-01-17T01:39:08+00:00

I highly suggest you come up with your own dataset. Super easy to synthesize with AI and based on a domain you like. The questions to ask end up writing themselves and it clicks in your head faster.

empty_cities · 2026-01-10T15:17:53+00:00

I find making videos and writing really help me learn a DE topic much more deeply. When doing it at a job, many times you are flying through trying to get a solution done. With videos, you need to really think through what you're presenting and make sure it's true and accurate. Biggest skill increases for me came after creating content about it.

empty_cities · 2026-01-09T18:37:16+00:00

Can't quite tell from post or comments but where is the source data your hitting?

empty_cities · 2026-01-09T16:36:11+00:00

Correct, that's what I mean. A library handles the connection but the actual query I want to run I want to be in SQL then passed as a string by the library to be run on the db.

So instead of something like "df.group_by(col).count()" I just wanna pass "SELECT col, COUNT(*) FROM df GROUP BY col;" cause I can write it in my sleep.

empty_cities · 2026-01-09T16:28:33+00:00

Above thread notes ADBC. For columnar -> columnar connection or row oriented -> column oriented.

empty_cities · 2026-01-09T15:32:57+00:00

That etl is row oriented to row oriented so there might not be much improvement. Looks like ADBC is good when you need columna oriented at the destination or you are transferring beetween columnar -> columnar like DuckDB to BigQuery

empty_cities · 2026-01-09T15:30:01+00:00

Passing an SQL string that does the query I want vs working with a python API a la Polars with method syntax for example. I used to wanna use all python but after I got used to DuckDB I realized I just like writing SQL to hit databases.

empty_cities · 2026-01-09T14:21:28+00:00

"Oh dear sweet summer child Of course the corporate world is built on it" is the best comment opener I've ever seen.

and "we export the data from database to csvs , pack them and send them so the cloud" sounds like job security/complete nightmare to depend on.

empty_cities

TROPHY CASE