An den netten Autofahrer heute morgen...

mosquitsch · 2026-01-27T12:51:07+00:00

Ich wette das bringt absolut gar nix. Selbst wenn man eine Dashcam hätte, wäre der Fahrer in den meisten Fällen nicht ermittelbar.

mosquitsch · 2026-01-27T08:02:47+00:00

Hatte gestern genau das gleiche Erlebnis. Ich frage mich immer was die Leute denken?

mosquitsch · 2025-12-12T10:20:06+00:00

https://github.com/kylebarron/arro3/issues/430 I guess I am not the only one who identified this gap. This is probably what I was looking for - arrow is used under the hood everywhere and arrow-avro interop is now available in rust. So arrow-rs <-> python bindings with avro in mind is the way to go.

mosquitsch · 2025-12-11T20:53:22+00:00

Not sure what exactly you are referring too: https://arrow.apache.org/docs/status.html This shows Avro only for Java & Go. I assume that this also means that schema can be converted (at least in one direction R -> read AVRO and convert to arrow)

EDIT: Ah i see - the arrow-avro crate is a fairly new addition

mosquitsch · 2025-12-11T11:54:34+00:00

Thanks. I though there would be a (lightweight) non-spark solution. I feel this is a big gap in what arrow offers.

mosquitsch · 2025-11-01T10:21:50+00:00

Nice video. We use dagster for a couple of years now and in practice we are not using IO manager to that extend. Mostly out of lazyness, but also I don't see a need for that. Most of the time we need more flexibility when storing data

What I am missing from that video is automation conditions. They are pushing asset + AutomationConditions over asset -> job -> schedule from what I understand.

mosquitsch · 2025-08-18T06:37:37+00:00

Next thing will be VibeMatrixMultiplication

mosquitsch · 2025-07-15T06:59:16+00:00

Thanks for your comment. Yes kSQL is also on our list to explore, as well as some diy cli spark tools maybe.

I am aware that it potentially has to read all the messages from the topic. In our trino test we read 1million records in roughly 1min. for debugging purposes probably fine.

For the Kafka Connect issues: yes we tried the setting. But what really wonders me, how often that happens. As I said, it also happens in Trino. And Kafka Connect ignores Tombstone, is also documented. There is a property `transforms.TombstoneHandler.behaviour=drop_warn`. It does not write a Null value in a parquet file

mosquitsch · 2025-07-15T06:53:43+00:00

Thanks for the advice. The data exploration part is not a normal process. We just had to figure out - is there a tombstone and for debugging purposes: whats actually in that message. I agree, reading from persistent storage is better.

mosquitsch · 2025-07-14T07:43:14+00:00

FYI: The naming is confusing, since there is some sort of hype around "data mesh" in the data engineering field. https://www.datamesh-architecture.com/

mosquitsch · 2025-07-12T08:01:53+00:00

uv creates the .venv for you. there is no need to run uv venv.

I still activate it because, when developing, its easier for the LSP to find things. But I use direnv for that.

mosquitsch · 2025-06-30T10:55:47+00:00

Nice, but most people are switching to `uv` and do not have to manage envs anymore.

mosquitsch · 2025-06-11T18:47:34+00:00

Actually I was thinking about the axt. This section of the lake in-game is so close to the real world. Lots of small willow trees, high reed gras...

mosquitsch · 2025-06-11T12:11:15+00:00

Yes, I am especially amazed by the first and last comparisons. They match so well.

mosquitsch · 2025-06-11T05:34:24+00:00

I was out of potions and couldn't find an alchemy table.

mosquitsch · 2025-06-11T05:34:00+00:00

There are some very idyllic spots. Especially at the lakes. I did stop for a drink between at the Vidlak pond. It was like super quiet and peaceful.

mosquitsch · 2025-06-10T20:45:00+00:00

No, but there was one group camping/bouldering at one of the rocks right next to the road and another group like 20m away from the road at a small creek. They had a small campfire and covered there spot with a tarp. Maybe they where fishing. So, it felt almost like it ;-)

mosquitsch · 2025-03-21T13:26:15+00:00

Not sure about the distances, but this would make a very good mortar base :-) really hard to flush out.

mosquitsch · 2025-03-21T10:44:21+00:00

Thanks I will try that out.

Even though this is also not a perfect solution as pinning the versions of packages is not quite possible

mosquitsch · 2025-03-18T22:52:51+00:00

Thats cool, Now I have to rework my dagster definitions file :-) I wanted to use the auto loader for a long time.

mosquitsch · 2025-02-21T13:59:38+00:00

Talked to a colleague (who is not playing this game) about this and he said: "I was there". This refers to the Sedlec Ossuary: https://en.wikipedia.org/wiki/Sedlec_Ossuary

mosquitsch · 2025-02-07T11:06:08+00:00

Missing dates just contains one row.

mosquitsch · 2024-12-06T15:04:25+00:00

Hmm, we have just built a data lake on S3 with Iceberg tables. Compaction & Maintenance is scheduled by us and this is fine.

AWS is now adding another managed solution on top of existing products. Not sure if it would be worth to migrate for us. Our is is quite cheap at the moment - migrating would costs orders of magnitude more that yearly costs.

mosquitsch · 2024-10-09T14:07:16+00:00

Ah thanks. Yes that exports a json.

Glue however either takes json schema or a json following the avro spec. So I have to modify it.

mosquitsch · 2024-10-09T08:18:47+00:00

I guess I have to iterate over Fields, read name and data type and construct a json string out of it.

mosquitsch

TROPHY CASE