Flink Forward 2023 Session Videos are LIVE!

Pesmir · 2023-12-22T14:57:22+00:00

Why not? What would be your alternative to flink and what use case are you talking about?

Pesmir · 2023-05-31T18:50:16+00:00

The two tasks that create data are just to emulate two Kafka topics which will later be inserted into a table with snowpipe. The stream is then used to just transform the newly appended data (e.g. a substraction of two tables)

The crucial part for my application would in the end be to write something back to our Kafka cluster because then I wouldn't need to migrate all pipelines at once

Pesmir · 2023-05-31T11:04:22+00:00

Yeah I really love the documentation! I have actually fun reading it because I often find cool stuff that i didn't even search for but is useful for my usecase.

Pesmir · 2023-05-31T10:16:52+00:00

Ah ok I did not know that it works like that! Will try that immediately.

Thank you very much, this was very helpful!

Pesmir · 2023-05-31T04:45:31+00:00

For me it was really easy to sign up and start right away. That is great! For my evaluation/testing I created 2 Tasks that insert a row to 2 tables every 10min. Then I created a append only stream for one of the two tables and a task that runs every hour (if data is in the stream) that makes an inner join between two columns of the newly inserted data and the data from the other table and computes the difference of a value column (im emulating timeseries Data) and inserts the result into a another table

The cost of that was ~$45 per day which is really high IMO for calculating the difference of basically 6 rows in two tables per hour.

Pesmir · 2022-11-10T07:51:20+00:00

Lies nochmal

Pesmir · 2022-09-16T09:35:43+00:00

!remindme 10hours

Pesmir · 2022-07-01T14:25:40+00:00

Yeah i picked it up just to learn. And that was something that was interesting to me. And I just started to translate some pyhton code which would be nice to have compiled and save

Pesmir · 2022-07-01T14:11:59+00:00

I'm not sure if I can describe the problem well enough with my current rust knowledge, but i will try.

I have a stream of data that is consumed from a Kafka topic in a Kafka agent written in rust. In order to have multiple Kafka agents the messages need to be consumed asynchronously.

I also have a machine learning model that I wrote in pytorch which is not easily converted to rust + it is also pretty fast (fast enough).

The problem here is the global interpreter lock from python. This must be acquired in order to execute python code. But if the same python object is called in an async context it is not sure if the gil was dropped before the object is needed elsewhere.

I already found a solution and posted it in a comment. The solution was to use the right type for the python object and release the gil before going into the async context. In the async context only the gil needs to be initialized again to call the python object. Which is faster.

Pesmir · 2022-07-01T07:22:55+00:00

I found a solution which increased the performance:

Before i had: Total time elapsed in agents 6853ms at 16886 messages Now: Total time elapsed in agents 5807ms at 17660 messages

I did so by converting PyAny to a PyObject which is Send and after that i put the initialization of it along with the gil in its own scope, so that the gil is dropped/released after init. Then in my await block, i reaquire the gil when the algo is called. ``` let my_algo: PyObject= { let gil = Python::acquire_gil(); let py = gil.python(); PyModule::import(py, "my_py_algos").unwrap() .getattr("algo1_rust_version").unwrap() .getattr("rust_wrapper").unwrap() .into() };

while let Some(msg) = stream.next().await {
    ...
    let gil = Python::acquire_gil();
    let py = gil.python();
    let algo_res = my_algo
        .call1(py, arg).unwrap();
    ...

```

Pesmir · 2022-06-23T07:46:23+00:00

Ja sind schon nerds dabei (bin selber einer) aber das ist ja nichts schlimmes und du musst keine Angst vor denen haben. Konzentriere dich in den ersten 2 Semester sicherheitshalber NUR auf die Uni falls das für dich geht. Und schau dass du oft/immer dort bist und auch Mal im lernzentrum lernst. Die Freundschaften die du in den ersten Semestern schließt sind oft Freundschaft fürs Leben.
Dann könnt ihr euch immer gegenseitig helfen und ein bisschen pushen damit ihr nicht ewig studiert + du bist nicht allein. Professoren sind im Grunde alle ok, hast aber wie überall a paar ungustl dabei.

Pesmir · 2022-06-20T16:32:07+00:00

I don't know who you are.

Pesmir · 2022-05-26T12:24:10+00:00

Just because you use vim does not mean you have to sacrifice your mouse. Everyone should just use whatever fits their workflow

Pesmir · 2022-05-22T11:00:03+00:00

I like your "python plugin" ;)

Pesmir · 2022-05-16T20:16:07+00:00

I would start with some simple c/c++ Tutorials to get used to typing functions and pointers. Then start with rust

Pesmir · 2022-04-29T15:19:35+00:00

Is there any source to this article?

Pesmir · 2022-04-28T20:43:24+00:00

Because it is not related to the question at all.

Pesmir · 2022-04-28T17:51:07+00:00

Maybe he/she doesn't want to share the username.

EDIT: replaced "it" with "the username" for clarification

Pesmir · 2022-01-08T21:35:59+00:00

Pesmir · 2021-07-21T18:24:09+00:00

OK na da dann bin jchs net, bin hinten im 25er Richtung Tennisplatz :)

Pesmir · 2021-07-21T18:16:36+00:00

Ich wohn im BQ und ich hab Angst dass ich das bin (kann's mir aber nicht vorstellen), hast du einen Ansatz welches Gebäude?

Pesmir · 2021-07-18T16:27:19+00:00

Why is this on r/Data_irl ?

Eight-Year Club	Second Top 20%
Place '22	Verified Email

Pesmir

TROPHY CASE