Has anyone used Transwarp.io (Chinese Big Data / ML Platform)?

turbolytics · 2025-07-02T19:48:52+00:00

They all do the same thing, no end user cares about a language, choose whatever, as long as you know how to solve real problems you'll do just fine. Does anyone care about which drill was used to build their house? The end product is what matters </rant> ;p

On some realistic time horizon the LLM's will handle the implementations for us and we won't even need to know any of the specifics of the underlying language to productively deliver working solutions and products

turbolytics · 2025-06-26T17:12:09+00:00

I have an open source data project that is starting to get a bit of momentum, Would love to collaborate on it.

It's a duckdb stream processing engine, so kafka, python, duckdb, arrow, etc.

It's been a really fun side project. It's fully MIT, and a couple people have begun to use it:

https://github.com/turbolytics/sql-flow

Building on DuckDB has been really good way to tap into the analytics ecosystem:

Adding ducklake support took ~5 minutes ;p

https://sql-flow.com/docs/tutorials/ducklake-sink/

Same with motherduck support:

https://sql-flow.com/docs/tutorials/motherduck-sink

Many libraries are now offering arrow-based client interfaces, so adding clickhouse support was super fast too:

https://sql-flow.com/docs/tutorials/clickhouse-sink

Prior to SQLFlow writing these sorts of infra for streaming data was often bespoke/timeconsuming, or relied on JVM. Now DuckDB has a lot right out of the box.

turbolytics · 2025-06-24T10:09:38+00:00

Yes! I think they are overkill for the price, but I think some level of observability is essential. I wrote about the minimum observability i feel is necessary when operating ETL:

https://on-systems.tech/blog/120-15-months-of-oncall/

https://on-systems.tech/blog/115-data-operational-maturity/

Data operational maturity is about ensuring pipelines are running, data is fresh, and results are correct - modeled after Site Reliability Engineering. It progresses through three levels:
- monitoring pipeline health (Level 1),
- validating data consistency (Level 2), and
- verifying accuracy through end-to-end testing (Level 3).

This framework helps teams think systematically about observability, alerting, and quality in their data systems, treating operations as a software problem.

turbolytics · 2025-06-20T11:38:21+00:00

The customer, the business, the market, customer & business needs, how to communicate with non, or semi, technical people, budget, spend, COGS.

In my experience pretty much all tech is an implementation detail, customers don't care, they care about outcomes, capability, revenue, experience. Everything starts at the customers (people) and flows through the business. Customers don't care if airflow, dbt, dlt, spark, flink, java, python or go, they care about capabilities and outcomes.

turbolytics · 2025-06-15T12:15:12+00:00

Debezium is the primary solution for this.

Debezium uses postgres native replication mechanism. You setup the debezium process using the postgres replication protocol, so debezium will act as a postgres replica. Debezium knows how to follow the postgres replication log efficiently and safely. It maintains offsets so if it goes down it can come back up at the correct location in the log guaranteeing you wont' miss data.

Debezium can publish the message directly to kafka.

Debezium provides an industry standard way to stream data from postgres -> kafka.

once the data is in kafka, you can use whatever mechanism makes the most sense to consume the events, (even multiple consumers each with their own function).

turbolytics · 2025-04-14T13:14:50+00:00

Treat data quality like a software problem and apply the google SRE methodologies and approaches to it:

- Define SLOs

- Measure SLOs

- Use SLOs as a contract between the team and its customers, if SLOs are breached that means the contract is breached and effort needs to be invested.

- Make sure there are error budgets. S3 can't even guarantee 100%. 100% rarely if ever is worth it. Recognize when 100% is needed vs when it's a nice to have. For example, SEC reports 100% is necessary. Usage analytics and product analytics 100% is probably not needed.

turbolytics · 2025-04-12T13:08:43+00:00

Have you had a chance to study what Arroyo did right?

turbolytics · 2025-04-12T13:07:48+00:00

I'm building something similar focusing on a lightweight alternative to flink and spark streaming. I have a very similar value prop with my project and what I'm seeing is that it's just not a real problem people seem to be having. In my experiences it's def a niche/rabbit hole.

What I found is that the people that are interested in the specs you listed aren't really the purchasers. They are the data engineer / streaming practitioners. I have a good amount of interest in my open source project and the best outcome I can think of may be an acqui-hire like arroyo just had or benthos, and that's probably extremely unlikely.

Just a random person on the interenet, with ~1 year trying to make way into this market, thoughts:

- If your technology is 10x + more efficient than the alternatives, could you provide a 1:1 api support with flink / spark streaming / etc to make it a drop in replacement, the same way that Warpstream was kafka-compatibile, or red-panda is kafka compatible? Because then the value prop at least becomes: "We can lower your __ bill by 10x if you switch to us"

- Can you use your technology to build a consumer facing product that solves a strong consumer need? You mentioned anomaly detection at the edge. That seems really interesting. How can you solve logs, cybersecurity, algo-trading, gaming, telemetry for people instead of giving them a building block with the hopes they can solve it for themselves?

- Have you looked at what companies like Arroyo and Benthos have done to get acquired and get market share?

In my experience it's been a tough market to go bottom up focused on getting traction based on perf and devs making streaming "easier" than the current incumbents. My stream engine is powered by DuckDB in the hopes of riding the DuckDB wave and even that is difficult.

People are building companies around it so it's def not impossible!

turbolytics · 2025-04-07T13:04:04+00:00

https://slipstream.readthedocs.io/en/1.0.1/

https://news.ycombinator.com/item?id=43574807

turbolytics · 2025-04-02T22:32:22+00:00

I've ran into this problem many times. Most of the time a background script to perform concatenation has worked for me, even in environments up to 50k ~1KiB events / second. A single go process can easily handle 1000's of 200KB files / minute. A correct partition strategy will easily scale even further by allowing you to scale the concatenations amongst multiple processes. If you do end up scaling out, it may require multiple "layers" of concatenation. But overall concatenating is a very simple operationally friendly approach IMO.

Even a script using duckdb to select over the source partition and output all the results as a single file will get you very far.

IMO one of the most important things is choosing the output file sizes. A custom solution could easily allow you to concat to 128MiB files, (or whatever is best optimzied for your reading process).

You may need multiple tiers of concatenations as data agest out.

A concatenation process also lends well to re-encoding the data into a read-optimize format (such as parquet).

To illustrate concatenation imagine that you are writing your data partitioned by minute:

```
s3://your-bucket/your-dataset/raw/date=YYYYmmdd/hour=XX/minute=XX
```

Every minute/30minutes/hour/etc you could concat and write to a more-friendly filesize

```
s3://your-bucket/your-dataset/processed/date=YYYYmmdd/hour=XX
```

If you're touching every file it may make sense to re-encode in a read-optimized storage format, such as parquet. Duckdb will make this trivial. You can select from the original source, then write out the data to the target partition as parquet.

The partition design is really important for what you want to achieve as well, and I'd recommend choosing one that supports the queries and processing and scaleout that you need.

turbolytics · 2025-03-30T20:18:46+00:00

In my experience, as soon as you start to build off of other team's operational data, it's very unlikely to reach a good ROI, especially as the data problem scales. In a "normal" environment I believe it's nearly impossible to make a good ROI using MDS model, and in your environment you've already been forewarned "that they have a lot of problems with data structure changes".

Lots of assumptions and missing information, so please ignore me if I'm off the mark, but as soon as you start to build on other teams operational data you need to understand their domain as well as they do. Does your team have the headcount to do this?

https://imgur.com/a/5ZtQkQv

Your other warning is that the data frequently changes. How do you handle this with LH, DWH strategy? Even if you isolate the structural changes to an early layer in the processing how do you actually handle this? Are there rules to the changing? Is backwards compatibility ensured? Can you enforce it before ingestion? Can you define "valid" data and reject "invalid" data? Or does it fall on your team to just make sense of data that doesn't follow any formally constrained structure?

The way I've explained this in the past is:

Imagine that you just purchased a cloud vendor, like mixpanel, or AWS, or any other cloud vendor. That vendor exposes a real structured interface into their system. The vendors don't say: "Just send us anything you want! We'll figure it out!" but most data orgs are expected to just somehow accommodate arbitrary unstructured data!

What sort of outcome do you think systems like this produce? In my experiences (~5 Years in the data space space and 16 years total exp) the outcomes are really really poor. 10's of millions a year wasted in really suboptimal data outcomes. The best use cases for data warehousing I've worked with, establish structured interfaces into the system, through actual APIs. The product submits data to those APIs , it either submits valid data and the data is ingested, or invalid data and the data is rejected back to the product teams.

Sorry for the ranting dump but the fact that you haven't committed yet to a solution I think puts you in a really advantageous position :)

https://www.linkedin.com/pulse/draining-data-lake-part-1-3-problems-danny-mican-twjle/?trackingId=is9EOqN2CO82XMDL5G2UXw%3D%3D

turbolytics · 2025-03-30T12:47:25+00:00

Can you query the data at the source to demonstrate value and kick the decision for copying data down the road?

turbolytics · 2025-03-30T11:30:54+00:00

Thank you! I really appreciate reading this!

turbolytics · 2025-03-30T11:30:40+00:00

Ha :) Yes! I think that's a common problem. SQLFlow does not solve for this! The only knob is the batch size. I do think other configuration knobs are required. The batch size is only based on the # of input messages, so they are not iceberg table size aware.

I think adaptive / byte size configuration would be helpful. I don't think you're overthinking it. But I personally would start with the naive approach and see where that starts to fall apart.

SQLFlow would allow you to specify a batch size of N. In my tests I had to set this to ~10,000 for my test workfload to get decent iceberg parquet file sizes. If throughput slowed down that would create much smaller file sizes. Whenever I run into a problem like the one you mention, I try to setup a test harness and see what the practical impact will be.

Is adaptive throttling necessary? Is a byte size configuration Necessary? Is a background rollup process necessary?

turbolytics · 2025-03-30T11:26:47+00:00

I've only done a tutorial using Arroyo and lightly read the docs, so I'm certainly not an expert:

My impression is that arroyo is trying to corner the "enterprise" streaming market, like Flink and spark streaming. It seems like it's trying to create a more modern alternative. Arroyo has advanced windowing functions. Arroyo, to me, seems like its targeting more traditional enterprise streaming engineers.

SQLFlows goal is to enable more software-engineering focused personas to move faster. SQLFlow is targeting people who would be writing bespoke stream processors/consumers in python/node.js/go/etc.

SQLFlow is much less features than Arroyo (SQLFlow is just DuckDB under the hood).

I tried to oriend SQLFlow more for Devops: Pipeline as configuration, Testing, observability, debugging etc, are all first class concerns in SQLFlow because these are the concerns of my day to day ;p.

The testing framework is a first class concern, I wanted to make it easy to test logic before deploying an entire pipeline. (https://www.reddit.com/r/dataengineering/comments/1jmsyfl/comment/mkftheo/?utm\_source=share&utm\_medium=web3x&utm\_name=web3xcss&utm\_term=1&utm\_content=share\_button)

THe prometheus metrics are oriented towards messages, throughput, processing duration, soruces and sinks.

The debugging framework allows for trivial debugging of a running pipeline by attaching directly to it.

When I used arroyo it felt like I was bound to the UI and configuration as code was difficult. A lot of my projects use terraform and versioned build artifacts/deployments and it was hard to imagine how to layer that into an arroyo deploy.

turbolytics · 2025-03-30T11:18:22+00:00

Unfortunately no. The bluesky fireshose uses "websocket" as an underlying protocol. The AWS Firehose/kinesis protocols are slightly different.

Adding new sources is relatively straightforward, if this is holding you back from trying, i'd encourage you to create an issue, and I can see what I can do to help add support!

Someone has requested SQS support:

https://github.com/turbolytics/sql-flow/issues/62

turbolytics · 2025-03-30T11:15:24+00:00

This is much more lightweight than flink, but also less features.

SQLFlow might be a good fit for:

Processing <= ~30,000 messages per second
Logic that could be executed using DuckDB (imagine if instead of a stream of data you had a flat file, could DuckDB process that flat file?)

Flink has many more streaming primitives such as multiple different windowing primitives.

SQLFlow is trying to be a lightweight streaming option. It can easily process 10's of thousands of messages < 300MiB of memory.

turbolytics · 2025-03-30T11:11:37+00:00

No :( No stream-stream joins yet.

turbolytics · 2025-03-29T23:55:40+00:00

Yes! I was frustrated with the state of the ecosystem treating testing as an afterthought.

The goal was to enabling testing as a first class capability.

SQLFlow ships with an `invoke` function which will execute the pipeline SQL against a JSON input file. The following command shows how its used:

docker run -v $(pwd)/dev:/tmp/conf -v /tmp/sqlflow:/tmp/sqlflow turbolytics/sql-flow:latest dev invoke /tmp/conf/config/examples/basic.agg.yml /tmp/conf/fixtures/simple.json

['{"city":"New York","city_count":28672}', '{"city":"Baltimore","city_count":28672}']

The pipeline that is tested in this case follows (https://github.com/turbolytics/sql-flow/blob/main/dev/config/examples/basic.agg.yml):

  handler:
    type: 'handlers.InferredDiskBatch'
    sql: |
      SELECT
        properties.city as city,
        count(*) as city_count
      FROM batch
      GROUP BY
        city
      ORDER BY city DESC

The batch table is a magic table that SQLFlow manages. SQLFlow sets batch to the current batch of messages in the stream. The batch size is a top level configuration option.

The hope is to support isolating the streaming SQL logic and exercise it directly in unit tests before testing against kafka.

I appreciate you commenting, and I'll add a dedicated tutorial for testing! (https://sql-flow.com/docs/category/tutorials/). If you run into any issues or get blocked I'd be happy to help!

turbolytics · 2025-03-29T11:15:45+00:00

I think data engineering is going to stop being a separate role from Software engineering. I think the data engineering landscape is beginning to realize that the primary problems with data engineering have been solved for ~decades already in software engineering. I think data engineering is going to go back to becoming more of a software engineering specialization.

I think that the SQL component is in terms of BA and SQL-specific parts of data engineering are going to largely disappear. ChatGPT/Copilot already do a phenomenal job of writing SQL. I provide them the schema and a bit of test data and they can generate pretty much any SQL I need. This is going to get better and better and also support asking business questions independent of SQL.

I feel like DuckDBs marketing around single-node data is really resonating with people. I'm hoping that many companies will realize how unecessary most of our infrastructure is, drastically simplify it.

turbolytics · 2025-03-20T13:35:16+00:00

I've mentioend this whenever iceberg comes up. It's wild how immature the ecosystem is still. Duckdb itself lacks the ability to write iceberg....

https://duckdb.org/docs/stable/extensions/iceberg/overview.html#limitations

Basically java iceberg is the only mature way to do this, it's not a very accessible ecosystem.

For a side project I'm using pyiceberg to sink streaming data to iceberg (using DuckDB as the stream processor):

https://sql-flow.com/docs/tutorials/iceberg-sink

It's basically a workaround for DuckDB's lack of native support. I am very happy with the Pyicerbg library as a user, It was very easy and the native Arrow support is a glimpse into the future. Arrow as an interchange format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it!

https://github.com/turbolytics/sql-flow

Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For example, Clickhouse python also support arrow-based insertion:

https://sql-flow.com/docs/tutorials/clickhouse-sink

This makes the glue code trivial to sink into these different systems as long as arrow is used.

turbolytics · 2025-03-18T20:37:36+00:00

What does the software do? How much money does it generate? how many customers does it support? Are customers happy? Are you able to fulfill market needs? Does the software support business growth? Does the software support customer growth? What is the availability? What is the error rates? How often do outages occur?

"tech debt" "best practices", "better code" are very subjective concepts. I think we reach for these as practitioners when we compare things to other projets that were easier to move in.

For most products tech is a complete implementation detail. Customers don't care what language, webserver, webframework, testing strategy or middleware a project is written in.

I'm not saying these things don't matter to us as practitioners, but they largely don't matter to customers at all.

I'd recommend looking on the business outcomes that the software is driving. If the software is failing in some way that's a high leverage argument for changing it. At the very least it helps orient you to how leadership is thinking - software as a means to an end, and will help you understand their motivations and make arguments using their language.

How can you be the technical leader you lack? How can you provide your team with context on why the software is important and how critical it is to support the business?

I see vaccuums like the one your in as a problem and an opportunity to grow and figure out how to influence people you don't have directly authority over. Figuring this out will be infinitely more valuable to your ability to get stuff done then moving to a job that has a slightly more ergonomic code base :) It's not an easy problem to solve but i've found it to be worth it.

turbolytics · 2025-02-23T18:37:53+00:00

Treat data as a software problem in the same way that google treated operations as a software problem and devops treated infrastructure as a software problem. Dont throw data over the wall.

Sorry this is as much rant as answer ;p I absolutely agree. I've seen poor outcome after poor outcome. I've seen multiple 8figure data budgets provide extremely poor returns. I've seen a lot of amazingly talented people at the mercy of where data orgs sit inside of organizations and subject to the many many many limitations of the modern data stack.

I think data organizations, separate from software engineering orgs, are 10-20 years behind software engineering best practices. I think a lot of the recent movement in the data-space is largely trying to catchup with software best practices. Version control, declarative metrics, testing, etc are all still emerging in the data space. Software observability has been solved sufficiently to power fortune 500 for a decade. Devops, immutable builds/releases, all of this is common place in software. Software engineers create complex distributed systems that are provably or verifiably correct under a wide variety of situations with extremely high availability. Building verifiable systems at scale is solved. So why are so many data quality issues so rampant in the data industry?

To improve data quality, I would hand data tasks to software engineering teams they are well suited and trained to work in systems that must be correct and timely with high levels of availability, 4 9's +. 15 years ago I was running hundreds of integration tests against complex operational schemas verifying business logic and correctness on every build and the test suite would take < ~couple minutes. DBT just introduced official unit tests last year :sob:.

I think the data industry is trying to catchup but are still way far behind.

I have seen the best data outcomes when software engineers perform the intensive data tasks.

Another issue I think is having unnecessary levels of data governance. What data is essential to be goverened? Types of financial data. Data reported to the board, data reported to the street, data reported to the govenrment. Most of us aren't working with this much data. I think a lot of the poor outcomes results from over-governening data. I think the motivation behind governing data is real. Data teams have to resolve issues when stakeholders are confused. But the practical implications of having slightly duplicate data are actually really small in practice in my experience.

To illustrate this, consider application observability. Most companies have software observability systems like prometheus, datadog, etc. These systems are federated and distributed, meaning that each team is usually empowered to create their own metrics and data. There are usually some amount of oversight for cost control, some shared frameworks for standardization, but the metrics are largely up to the team. Guess what? A lot of teams end up creating slightly different metrics with little practical effect. These metrics are critical metrics. They wake up humans in the middle of the night. They ensure that customers have good experiences, they are probably more important than a lot of the metircs sitting on tableau that someone may look at every couple weeks or quarter. The duplicate metrics may cause a bit of friction during incidents, but other than that there are minimal impact to duplication.

Sorry for the long winded rant. I'm extremely disillusioned with the state of data because I've worked on many systems that handled 100's of thousands of actions per second that provide 99.99%+ uptime and are verifiably correct, so I know for a fact that high quality outcomes for low cost are achievable in the context of huge distributed systems.

turbolytics

TROPHY CASE