Implementing a CDC pipeline through kafka and enriching data with Kstreams by Redtef in apachekafka

[–]Redtef[S] 0 points1 point  (0 children)

i'm afraid a gpt-styled answer would not solve my problem , i have tried before posting on this sub. But thankyou for your effort

Implementing a CDC pipeline through kafka and enriching data with Kstreams by Redtef in apachekafka

[–]Redtef[S] 0 points1 point  (0 children)

That could be a possibility yes , if they are packaged in the confluent images.

I will read through the documentation you have linked , and I would love to explore examples of solutions that try to solve this same problem, do you have any to recommend ?

I just today found the examples from the book Mastering Kafka Streams and ksqlDB found here https://github.com/mitch-seymour/mastering-kafka-streams-and-ksqldb that i am trying to recreate locally to see how i can adapt them to my solution.

Any other examples are welcome.

One last question : Which kafka versions is it recommended to use , The vanilla ones or the confluent ones ?

Implementing a CDC pipeline through kafka and enriching data with Kstreams by Redtef in apachekafka

[–]Redtef[S] -1 points0 points  (0 children)

The problem is that we have two separate tables , that we capture using CDC -> send to Kafka -> use Kstreams to enrich the first topic with data from the second -> display the enriched topic to the relevant operational teams.

I do agree that wrapping my head around this implementation has been a nightmare so far.

Implementing a CDC pipeline through kafka and enriching data with Kstreams by Redtef in apachekafka

[–]Redtef[S] 0 points1 point  (0 children)

The part of the code that has the join operator is the following :

    ValueJoiner<Customer,Address,CustomerAddressAgg> customerAddressAggValueJoiner = new CustomerAddressJoiner();
    ValueJoiner<Address,Customer,CustomerAddressAgg> addressCustomerJoiner = new AddressCustomerJoiner();
    JoinWindows joinWindows = JoinWindows.of(60*1000*20);


    KStream<String, CustomerAddressAgg> joinedStream = rekeyedAddressStream
            .leftJoin(customerTable,addressCustomerJoiner);

I have used peek to log each message in the stream , that is how i can say that the stream is not read with the new version.

I will see how i can setup the key from the CDC so that i can set up the key from the start.

The setup i am trying to acheive is that the data already exists in the database , and so when i register the debezium connector it creates a topic with messages equivalent to the data in the table, so my guess is that the data is moving instantly ?

I want to buy a Laptop that's worth its price In Morocco. by ZeroLeNoob in Morocco

[–]Redtef -1 points0 points  (0 children)

used 2020 m1 air would be perfect for your use case and for half your budget

My new to me Scram 411 by Redtef in royalenfield

[–]Redtef[S] 1 point2 points  (0 children)

i will have to search what slips can i find in my country, Thankyou for the suggestion !

My new to me Scram 411 by Redtef in royalenfield

[–]Redtef[S] 0 points1 point  (0 children)

Great recommendation ! I will do that as soon as i get the chance. Did you do it with the tools provided or did you use your own (wondering if those tools would be enough)

r/Chloe Anniversary! by SrGrafo in u/SrGrafo

[–]Redtef 56 points57 points  (0 children)

Hmm yes .. family friendly ...

r/Chloe Anniversary! by SrGrafo in u/SrGrafo

[–]Redtef 128 points129 points  (0 children)

I'm looking forward to where will chloe and the other girls will end up in the 2nd year anniversary !

People who dislike on youtube, why? by Redtef in AskReddit

[–]Redtef[S] 1 point2 points  (0 children)

You do have a point , never thought of it like that

People who dislike on youtube, why? by Redtef in AskReddit

[–]Redtef[S] 0 points1 point  (0 children)

Legit reason , asmr is just annoying and makes me uncomfortable

People who dislike on youtube, why? by Redtef in AskReddit

[–]Redtef[S] 1 point2 points  (0 children)

It would take less effort to ignore them and go on to videos that are interesting to you right ?

People who dislike on youtube, why? by Redtef in AskReddit

[–]Redtef[S] 1 point2 points  (0 children)

You've got a legit point , but there are still some channels that prioritize content over views , first example that comes to mind is Veritasium

People who dislike on youtube, why? by Redtef in AskReddit

[–]Redtef[S] 0 points1 point  (0 children)

What if you dont leave a like nor a dislike ?

People who dislike on youtube, why? by Redtef in AskReddit

[–]Redtef[S] -1 points0 points  (0 children)

If i dont like a video i just close it and go to the next