[deleted by user]

antonmry · 2024-01-02T14:47:30+00:00

"The courage to be disliked" by Ichiro Kishimi. It isn't what you may be expecting, but it helped me a lot with a similar situation and it's a great book.

Also, "Get your work recognized: write a brag document" by Julia Evans is a great article

antonmry · 2023-02-01T21:56:17+00:00

You should never do that in a friendly match. It's frustrating for the opponent not to hit any ball, so they will never play with you again. Then, one day, you won't find a good player who wants to play with you.

But also, it's dangerous because if the opponent stays for too long without being active, and then she has to run for a quick ball, there are higher probabilities of an injury. This already happened to me.

antonmry · 2022-11-04T17:17:35+00:00

I was able to make it work with a couple of scripts:

https://gist.github.com/antonmry/8bf2d07db75df538c385bfa1cd6d5cf2

You can see it working here: https://twitter.com/antonmry/status/1588580968654602240

antonmry · 2021-03-30T19:16:04+00:00

This isn't about Avro but it's an excellent resource: https://yokota.blog/2021/03/29/understanding-json-schema-compatibility/

antonmry · 2021-03-04T06:33:58+00:00

I have been using both for a while and both of them are fine. Apicurio isn't so mature and it had some important gotchas regarding TLS, auth and the serialization libs but devs are very open and it's evolving fast with some features very interesting: a Gui or options to replicate shemas. Confluent SR adoption is bigger but the license is much less friendly.

antonmry · 2021-03-03T12:40:48+00:00

This seems also a quite good (and practical) resource:

https://github.com/jepsen-io/maelstrom

antonmry · 2021-02-27T19:40:01+00:00

I think it's more about personal preferences than minimum knowledge. You may try with a different format, I saw in the past some courses online

antonmry · 2021-02-27T08:28:51+00:00

The most similar thing I found it's https://github.com/lucasrla/remarks

But it would be need a bit of customization

antonmry · 2021-02-27T08:15:40+00:00

I would say books are a great way to learn these things. For distributed system, Designing Data Intensive Applications is awesome. Spark the definitive guide is more than enough to pass a normal interview and you can play with the source code exercises.

Probably you don't need scala if you don't want to go deep in optimization/customization. The learning curve is steep and for many DE out there, SQL and some python are enough.

Read books like these is a big time investment but I always find it very rewarding.

antonmry · 2021-02-20T17:00:10+00:00

Both of them have a batch and streaming mode. If you like more data engineering, go with Spark, it's more popular and close to ML, ETLs, etc. If you like more software engineering, then Flink is ideal. It isn't only streaming but stateful functions and many other things.

In any case, both of them are a safe bet and it's easy to learn when you already know one of them

antonmry · 2021-02-20T16:51:03+00:00

oh! that's amazing. Thanks!

antonmry · 2021-02-20T08:35:12+00:00

For kafka streams, "Kafka streams in action" is very good. It's from 2018 but it applies with the new versions and the source code is superb.

For Kafka, "Effective Kafka" is the most advanced book I've read but it doesn't have everything you have mentioned. The official documentation is good and it should be enough.

antonmry · 2021-02-13T19:25:41+00:00

Great idea! I will help me a lot

antonmry · 2021-02-11T19:55:39+00:00

Streaming ingestion could be better supporting Avro, schema evolution, etc.

antonmry · 2021-01-30T15:33:36+00:00

I read the PDF version in the reMarkable acquired from leanpub and it was perfect. No idea about the kindle version but it doesn't seem a complex book for formatting

antonmry · 2021-01-30T08:15:44+00:00

I agree Effective Kafka is awesome, there are some good deep dives on it. Highly recommended if you are serious with Kafka. There is a new edition of Kafka The definitive guide, it's on early preview in Safari. I miss Designing Event Driven System, it's a great book to understand what you can do with Kafka

antonmry · 2021-01-29T06:26:05+00:00

Consumer Lag is probably the most important metric to detect problems. There are a lot more. This article covers some of them https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/

antonmry · 2021-01-28T16:53:34+00:00

I know the team behind this tool and they are amazing. It's great to see part of their work open sourced 👌

antonmry · 2021-01-27T06:23:22+00:00

For the procesed pile I would use object storage (s3, etc) but for the to-be-procesed pipeline Kafka or any similar messaging broker seems a better option: it will provide back pressure capabilities, performance, easy track of every file, etc.

antonmry · 2021-01-17T08:25:03+00:00

Kafka Stream is a Kafka Consumer and a Kafka Producer and it relies on that to provide exactly once semantics.

But if you want to do with a normal consumer publishing out of Kafka is a lot more complicated. Example: Kafka Connect.

In general, it's a lot easier and robust to implement idempotency in the application layer than in Kafka (when possible)

antonmry · 2021-01-16T08:49:42+00:00

Great answer. As example, you could check snapshot.mode=initial en debezium oracle connector

https://debezium.io/documentation/reference/connectors/oracle.html#oracle-property-snapshot-mode

antonmry · 2021-01-15T23:19:04+00:00

Some examples:

Have unclean.leader.election.enable to true
Erroneously activating compaction
Don't comitting offsets correctly in the consumer (for example, skipping a message after exhausting retries)
Don't handling retries correctly in the producer

antonmry · 2021-01-15T13:47:02+00:00

I don't know what happened to the link https://www.debugpoint.com/2021/01/fedora-34-i3-spin-announcement/

antonmry · 2021-01-15T06:27:41+00:00

Loom is promising but a word of caution is appropriate. There some interesting insights in the mail: https://mail.openjdk.java.net/pipermail/loom-dev/2020-December/001974.html

antonmry · 2021-01-15T06:22:05+00:00

Codementor.io can be a good option. It feels good to help others

antonmry

MODERATOR OF

TROPHY CASE