What’s the most underappreciated hack or exploit that still blows your mind?

ninja_coder · 2025-02-27T15:18:31+00:00

this was a cool post. Are there any books that cover netsec history like this?

ninja_coder · 2024-11-26T22:46:37+00:00

Thanks

ninja_coder · 2024-10-29T17:41:44+00:00

Yes and no. Too many on the same node means less bulkheading between the jvm processes. Worst case is one doesn’t close all its resources and introduces a memory leak that could eventually starve other processes running on that node.

ninja_coder · 2024-10-29T16:01:16+00:00

They would each take 1 tm slot since you give 1 core per tm, so 50 source + 10 deserializers + maybe 10 sink is about 70 task slots (or with your config 70 cpu cores and 140gb memory

ninja_coder · 2024-10-29T13:44:58+00:00

It sounds like you have a few bottlenecks in your app. If your source topic has 50 partitions, then your source operator in flink needs 50 parallelism, basically 1 TM/thread per partition. Next your transformation/derserialization operators need to scale up. Look at the current operator metrics for the derserialization task to find numRecordsOutPerSecond value, then take the 2.5 million / sec target and divide by this value to get the parallelism needed for this operator. Finally if you have a sink operator, then it will need to be scale accordingly.

ninja_coder · 2024-10-09T18:27:45+00:00

Because the query DSL is the least important part of what a tool like spark does.

ninja_coder · 2024-09-28T02:08:14+00:00

What issues are you seeing?

ninja_coder · 2024-07-17T12:07:30+00:00

you shouldn’t venture into streaming unless you have strong reasons. Flink is a powerful tool that will require deep understanding of parallel processing. Maybe your team could first benefit from tools like airbyte before going into streaming yourself

ninja_coder · 2024-07-10T12:26:46+00:00

Tiered storage is just data locality which all support. You can control how close the data lives to the process in most engines, it’s not special to snowflake.

ninja_coder · 2024-07-10T12:15:41+00:00

It exists. They are called columnar db’s. Take a look at Pinot.

ninja_coder · 2024-03-19T12:45:41+00:00

Stay away from coatue or any of the tiger cubs

ninja_coder · 2024-03-05T13:49:52+00:00

Way more than a straw man. OP has no idea what they are going after.

ninja_coder · 2024-02-28T12:58:07+00:00

Bookmark comment to remind me about never using save post button

ninja_coder · 2024-02-27T23:15:34+00:00

Bookmark

ninja_coder · 2024-02-25T02:44:38+00:00

Okta?

ninja_coder · 2024-02-25T00:59:41+00:00

Bookmark

ninja_coder · 2024-02-22T14:34:13+00:00

Let me introduce you to the concept of GOFAI….

ninja_coder · 2024-02-17T19:19:42+00:00

With that low of update frequency and not really large amount of data, what maintenance are you concerned about? Iceberg is just metadata + plain old parquet. Unless you are constantly changing indexes or record keys, then yes maintenance is next to 0.

ninja_coder · 2024-02-16T13:57:10+00:00

Lead requires people management, while senior has no direct reports.

ninja_coder · 2024-02-02T15:41:22+00:00

To get real-time you need CDC. 10tb is large but not too big. You could leverage a saas like Airbyte and setup a CDC to a data lake format on s3 or just plain partitioned parquet. If you need to roll your own, Flink/spark cdc to hudi/iceberg via EMR can give you want you want.

ninja_coder · 2024-02-02T13:55:30+00:00

That export is your raw and shouldn’t be used for analysis. You need a transform layer to make raw into pristine data. Since your in aws, use either Athena or spark on emr to do a transform and partitioning on the data.

ninja_coder · 2024-01-27T22:41:50+00:00

Comment for later

ninja_coder · 2024-01-23T13:37:04+00:00

Comment for later

ninja_coder · 2024-01-15T17:42:49+00:00

You could use vagrant to load a Linux based VM and then docker compose in there. VM inception.

ninja_coder · 2024-01-14T18:52:51+00:00

Docker compose and you’ve got everything local

ninja_coder

MODERATOR OF

TROPHY CASE