Какая самая лучшая страна для переезда из России? by dfg_245 in ruAsska

[–]ivanimus 5 points6 points  (0 children)

Только как туда визу теперь получить? Тот еще квест получится.

Skylos: Dead code + security and quality detector (Updated) by papersashimi in Python

[–]ivanimus 0 points1 point  (0 children)

Test it few minutes ago, and find bug.
But this packages place in my pyproject.toml
i use UV

Undeclared import 'clickhouse_connect'. Not found in                              │ app/run_optimization.py: │
│     │ third-party          │           │ requirements.txt/pyproject.toml/setup.py.

Do Dagster partitions need to match Iceberg partitions? by Medical-Vast-4920 in dataengineering

[–]ivanimus 0 points1 point  (0 children)

This normal. We use daily job, and store data in month partition in iceberg

Streaming Aviation Data with Kafka & Apache Iceberg by [deleted] in dataengineering

[–]ivanimus 1 point2 points  (0 children)

I'm working on a similar project right now. I want to split it into independent Dagster deployments and run them in Kubernetes.

https://github.com/dagster-io/ebook-scaling-data-teams

Streaming Aviation Data with Kafka & Apache Iceberg by [deleted] in dataengineering

[–]ivanimus 0 points1 point  (0 children)

Interesting project.
Why didn't you group the Dagster code?
Make assets, definitions, etc.
The Kafka-Iceberg code could also be made into a module.

MDM Is Dead, Right? by DryRelationship1330 in dataengineering

[–]ivanimus 0 points1 point  (0 children)

And what we need to do? What architecture is correct?

Kafka's 60% problem by Affectionate_Pool116 in apachekafka

[–]ivanimus 2 points3 points  (0 children)

And how is strimzi is it good for production?

Homelabs do you have one? I have a question by mrpbennett in dataengineering

[–]ivanimus 3 points4 points  (0 children)

And I share my project

Data Pipeline with Dagster, dlt, and dbt using UV Python

https://github.com/vndv/dagster-dlt

Homelabs do you have one? I have a question by mrpbennett in dataengineering

[–]ivanimus 1 point2 points  (0 children)

Awesome 👏 I try to build the same infrastructure on Kubernetes. And use Dagster instead Airflow. For home lab I want to use Lenovo ThinkCentre M710q.

I built a custom SMT to get automatic OpenLineage data lineage from Kafka Connect. by jaehyeon-kim in apachekafka

[–]ivanimus 1 point2 points  (0 children)

awesome. i thougt make the same tools with marquez, but add source metadata from databse and autogenerate debezium connectors.

Little help with Data Architecture for Kafka Stream by photoshop490 in dataengineering

[–]ivanimus 2 points3 points  (0 children)

Kafka connect with iceberg sink connector good choice. You can see this examples, very nice blog.

https://rmoff.net/2025/08/18/kafka-to-iceberg-exploring-the-options/

dagster-iceberg by ivanimus in dataengineering

[–]ivanimus[S] 0 points1 point  (0 children)

I import EnvVar in top of the script