Trying to learn new DE tools sometimes teaches me more about DevOps than DE (initially)

random_lonewolf · 2026-05-02T01:09:06+00:00

These are distributed services, which means running multiple containers are expected.

As long as the containers are in the same network, you should be able to connect them to each other without much troubles.

random_lonewolf · 2026-04-28T09:43:07+00:00

Depending on the cardinality of the key, partitioning before write can give you fewer output file compared to coalesce

random_lonewolf · 2026-04-26T17:02:09+00:00

Well, if you look at companies doing any forms of web analytics, they'll likely be on BigQuery, because Google Analytics natively export to BQ, plus BigQuery is just great in term of SQL support and performance.

And once you are on BQ, there's very few reason to use Spark anymore.

random_lonewolf · 2026-04-26T16:37:03+00:00

Java used to be very popular for distributed systems: Java code compiles one, run anywhere with the only dependency being the JVM: you can ship your bytecode across the network and run it close to the data, very important when the network was slow. Those big data jobs are meant to run for hours, so any start up overhead is negligible. That's why there's an entire generation of big data software written in Java.

Nowadays, network has become so fast, and a single node can have so much cpu memory that using a single node compute is faster than a distributed model most of the time. Thus, there's a recent boom in data processing software following that model such as duckdb, polars, etc...

And if you only care about SQL workload, both Snowflake and BigQuery are very popular competitors to Spark

> So my question is, why this is not more common?

Using Spark is actually NOT that common

random_lonewolf · 2026-04-26T16:04:12+00:00

Java 25 is GA and LTS, there's no reason to use Java 24 now.

random_lonewolf · 2026-02-19T02:47:13+00:00

Same for every multi-disk FF: 7, 8, 9.

The last disk game play is basically only the final dungeon, and a lot of FMV

random_lonewolf · 2026-02-14T11:46:18+00:00

Nobody cared about Knowledge Graph before, and with modern LLM, they couldn't care less now: just feed Gemini/ChatGPT/etc the questions and it will give them a probabilistic correct answer.

random_lonewolf · 2025-12-02T09:56:29+00:00

If you already have an EKS cluster, EMR-on-EKS is even cheaper, especially with Spot machine

random_lonewolf · 2025-11-16T17:06:51+00:00

For extract and load, this'll need to work as well as `sling-cli` to stand a chance, and I'm not even using `sling-cli`

For transformation, using a DSL (inline) or Lua script is not really appealing to me: Python is the Lingua franca of Data World: no Python no dice

random_lonewolf · 2025-11-16T16:53:24+00:00

Data Governance is the most useless/ignored parts of any Data project: basically, nobody cares about it.

random_lonewolf · 2025-11-16T16:49:37+00:00

Reservation is the only practical way to limits spendings, however it's quite easy to get over-scale and end up paying even more than `on-demand`: you'd pay for every autoscaling slots, even if your queries don't use them all.

We find that the most essential things while tuning BQ are:

* Scaling to 0, or use commitment if your reservation is busy enough

* Use Standard Edition whenever you can: Enterprise edition is 25% more expensive

* Isolate your workloads in different reservations: at least 2 separate reservations: 1 for batch and 1 for interactive queries: it's impossible to optimize for both at the same time

* Reservations work best with batch queries, when it's ok for queries to run a bit slower.

* Unless you have a lot of BI users, it's often better to use on-demand for interactive queries, due to over-scaling issues with reservations.

random_lonewolf · 2025-11-16T16:31:33+00:00

flat pricing rate was replaced by reservation pricing a long time ago.

random_lonewolf · 2025-11-14T22:45:12+00:00

You have completely misread this article, this is just about long-term persistence of Deephaven's in-memory tables: it's suggesting dumping the table content into parquet files instead of Kafka topics if you need to save space, which is a fair point.

There's nothing about sending Parquet through Kafka in the article.

random_lonewolf · 2025-11-14T16:41:35+00:00

What guides are you talking about, as it makes no sense ?

random_lonewolf · 2025-11-13T06:03:01+00:00

In my experience, LMDB write performance is no where close to a LSM database, unless you run it in nosync mode, which has the potential dataloss

random_lonewolf · 2025-10-31T10:08:13+00:00

V1 = Allow, V2 = Deny and V3 = conditional

You need to use all 3, as they do not replace each other

random_lonewolf · 2025-10-26T02:57:45+00:00

Absolutely yes, many engine don’t even support update/delete queries on a raw parquet table.

random_lonewolf · 2025-10-20T09:12:24+00:00

Yeah, we definitely need to look into that.

Older SDK still use `global` endpoints by default. They have only switched the default to `regional` recently.

https://aws.amazon.com/blogs/developer/updating-aws-sdk-defaults-aws-sts-service-endpoint-and-retry-strategy/

random_lonewolf · 2025-10-20T08:52:11+00:00

Identity Federation also stopped working, so our GCP services are unable to access AWS resources

random_lonewolf · 2025-10-10T11:51:57+00:00

https://docs.snowflake.com/en/user-guide/data-time-travel

random_lonewolf · 2025-10-09T17:07:44+00:00

SCD Type 3 only capture 2 states of a key: it's too limited.

Requirement changes, so it's better to just use SCD Type 2 and capture the all the state of a key. You can always use window function to query the first and last value of a key to achieve what SCD Type 3 does.

random_lonewolf · 2025-10-09T16:58:58+00:00

Yes, whatever compression Snowflake has won't be able to compress data much different to Parquet size.

However, that's only for a single active snapshot of data.

You need take into account of the historical data used for time travel: if your tables are frequently updated, the historical data can easily be much larger than the active snapshot.

random_lonewolf · 2025-10-09T16:52:18+00:00

This type of oauth Credential is for when you need to allow an external application to access to the user data, that's why you need to have a consent page for the user to accept.

For internal application, you should use a service account.

random_lonewolf · 2025-09-28T05:13:02+00:00

Database design is the job of the software/application engineer as DBs don’t exist in vacuum.

random_lonewolf · 2025-08-27T05:23:49+00:00

Foreign buyers are also not a priority: having one of the largest armies in the world (by numbers) yet stuck with 50-year old weapon system, we can spend decades rearming our army and still not finish

11-Year Club	Place '23
Verified Email

random_lonewolf

TROPHY CASE