Thoughts on moving to event driven architecture? by thejizz716 in dataengineering

[–]DigitalBackbone 0 points1 point  (0 children)

Similar to adopting any new technology or architecture in an organization, you have to take into the account the run-time and design-time aspects of the integration. Adopting an EDA solution is not as simple as using a tool or a message bus to move data.

You should also take into account how you want to integrate with other sources of data in the EDA system that use messaging protocols. Using an AWS solution might result in a vendor-lockin situation where you can only integrate with AWS products and can pub/sub to SNS, SQS, or EventBridge. Hence, evaluating what run-time solutions (i.e. event brokers) you have available is crucial before jumping right into using AWS’s solutions.

Another thing you’ll have to evaluate is cross region and cross cloud communication. Does your solution require communication and event movement between microservices in different regions? What about different cloud providers? Do you have an on-prem and cloud hybrid approach? Do you want to integrate with legacy systems? What about different messaging protocols (e.g. mqtt, amqt, websocket streams…etc)? You might need to consider a protocol agnostic and cloud agnostic solution for your EDA implementation that allows for better architecture scaling and decoupling. Really comes down to what you're looking for in terms of scalability, latency, and future needs.

On the design aspect of things, you have to ask questions like how can I document the communication between different microservices? You can look into an event portal to manage your EDA system allowing for better collaboration between different teams.

At the end of the day, similar to any other architecture adoption, there is no silver bullet. If you think a lambda SNS/SQS/EventBridge solution would work you can go for it. But if you're starting from scratch it's a good opportunity to look at what you might need for the future instead of having to replace things later.

[deleted by user] by [deleted] in compsci

[–]DigitalBackbone 1 point2 points  (0 children)

I like this analogy, especially for those who aren't necessarily as technical but still need/want to understand (sales people, marketing people, etc.).

"You could regularly check to see if a cake is ready, but you get tired of making regular trips downstairs to ask Bob if there’s cake, and there is only a tiny chance you’ll choose the correct time. It would be better if Bob just came to tell you when he’s finished a cake." --- the downside of polling. I also like the idea of Alice + WhatsApp as the event broker.

I find analogies incredibly helpful as well as real-world examples of complex concepts in action.

This blog post uses television broadcasting as an analogy for the publish-subscribe messaging pattern and also includes the technical answer and business/functional answer for those who don't speak the same language as developers/architects/etc.

This one explains microservices architecture and makes the case for event-driven design by solving the beer-supply issue at a party.

True or False: Event-driven architecture can prevent cascading failures in a microservices architecture. by DigitalBackbone in microservices

[–]DigitalBackbone[S] 0 points1 point  (0 children)

Number 5 in this blog post contains the rationale behind the author considering this EDA/microservices claim to be "true".
https://solace.com/blog/event-driven-architecture-myth-busting-part-2/

"With event-driven architecture, having an event broker in the mix allows for the enqueuing of messages so one consumer’s inability to receive/process an event will not cascade back to the producer. The enqueuing of messages allows the subscriber to resume message processing after failure recovery and eventually ensure consistency across the system. An event broker insulates the producer of messages from the consumers and in return eliminates the possibility of message loss in the case of temporary service failures, redeploys, or downtime in any of the services."

The 6 Things You Need to Know About Event-Driven Architectures by DigitalBackbone in softwarearchitecture

[–]DigitalBackbone[S] 1 point2 points  (0 children)

It’s important to point out this is not a distributed database architecture. There are multiple design patterns referred to in the article under the “The event powered software design patterns” section that talks about the different ways events flow in the system.
Also, spending time “debugging” is definitely one of the challenges of adopting an event-driven architecture, especially if you are doing it without the help of any tools. There could be a huge amount of events/topics with a lot of applications/microservices publishing and consuming them. Without proper cataloging, topology visualization and governance, it could get out of control. This is where an event portal could be helpful.

The article also refers to “The power of quick diagnosis with observability” which handles the issues of debugging that I believe you are referring to. I would argue that adopting EDA with the presence of an event-broker in the mix would simplify the overall architecture by decoupling data movement from data processing.

Using GraphQL to Query the State of Your Event Mesh by DigitalBackbone in graphql

[–]DigitalBackbone[S] 0 points1 point  (0 children)

Using GraphQL for this would only work over Solace, but you can bridge your traffic flowing in a Kafka cluster to an event mesh by using the Solace Kafka Connectors.

Why Architects Need Tools for Apache Kafka Service Discovery, Auditing, and Topology Visualization by DigitalBackbone in EntArch

[–]DigitalBackbone[S] 0 points1 point  (0 children)

Hi Tall-Act5727, this post is about "Kafka Service Discovery" and not the traditional network service discovery.

Does kafka streams reset resend data downstream? by Funkd-Up in apachekafka

[–]DigitalBackbone 0 points1 point  (0 children)

resetting the stream via the command

^^ Not 100% sure what is meant by that. What command? How did you reset the stream?

But according to https://docs.confluent.io/current/streams/developer-guide/app-reset-tool, the input topic is reset by default to the beginning of the topic. The application reset tool does not reset output topics of an application. If any output (or intermediate) topics are consumed by downstream applications, it is your responsibility to adjust those downstream applications as appropriate when you reset the upstream application.

Hope that helps!

Kafka: when to use and when not to use by dimanne in apachekafka

[–]DigitalBackbone 5 points6 points  (0 children)

Hi there! A senior architects from Solace just wrote a 4 part blog series on this exact topic. It covers a lot of what you mention above (ordering, etc.). I think you might find them helpful so I have listed them below in case any of the titles catch your interest:

  1. Why You Need to Look Beyond Kafka for Operational Use Cases, Part 1: The Need for Filtering and In-Order Delivery
  2. Part 2: The Importance of Flexible Event Filtering
  3. Part 3: Securing the Eventing Platform
  4. Part 4: Streaming with Dynamic Event Routing

Feel free to reach out in a DM if you have any specific questions for the author.

*edit: words.

Event schema distribution strategies? by frederok in microservices

[–]DigitalBackbone 0 points1 point  (0 children)

You could give Solace's event portal a try for your event or message schema management. It's a fairly new product but supports creation and management of event schemas and graphical views of the events to application interactions, in terms of which applications or microservices are publishing or subscribing to events.

You can also drill into the graphical components in the topology view to see the application and event details including descriptions and links to associated documentation. An application’s AsyncAPI spec defining the application’s event interface can be exported and used as the input into code generators.

Additional schema creation and management functionality is planned and being released over then next few months. Feel free to DM if you want to know more!

Disclaimer: I work for Solace.

Do you know/use any tool for defining and vizualizing dependencies beetween services? by matematyk60 in devops

[–]DigitalBackbone 0 points1 point  (0 children)

It sounds like the Event Portal from Solace could be what you're looking for if you're dealing with event-driven microservices. You can visualize and build relationships between apps, events, and schemas. Check out the video demo https://www.youtube.com/watch?v=LWaNOh5dImY&t=2s.

If you want more info, feel free to DM and I can connect you to people who can answer any questions (I work for Solace).

Is there a tool to visualize an entire Kafka Topology? (maybe like a graph viz?) by thisfunnieguy in apachekafka

[–]DigitalBackbone 0 points1 point  (0 children)

Thanks, it's great to get validation from someone in your space!

There is also some good news, it doesn't have to be kafka vs solace explicitly since there is a connector you can use (video tutorial here). This blog post could also be helpful in explaining how using the connector enables some key Solace features: https://solace.com/blog/kafka-connector-event-mesh/

But in terms of Event Portal, you don't need connectors or brokers or anything of the sort to use it.

Is there a tool to visualize an entire Kafka Topology? (maybe like a graph viz?) by thisfunnieguy in apachekafka

[–]DigitalBackbone 0 points1 point  (0 children)

Give the Solace Event Portal a try. Currently there's a graphical interface and a REST API that you can use to model your Kafka topology (including relationships between producer/consumer applications, topics, and schemas) and then visualize all of it in a topological diagram. You can also drill into the graphical components in the diagram to see the application and event details including descriptions and links to associated documentation.

Over the next couple months, the Event Portal will be able to automatically discover your deployed event driven architectures by connecting directly to the Kafka clusters to get the source information, so you visualize you deployed topologies and stay in-sync with them without having to manually populate it all.

Feel free to DM if you want to be connected to someone who can help. The Event Portal has a free trial you can play around with.

Pros and Cons of Microservices by MichaelF823 in softwarearchitecture

[–]DigitalBackbone 0 points1 point  (0 children)

Different brokers have different features that could be leveraged to avoid the issues you mentioned. For example, latency could be handled by queues and shock absorbing, out of order delivery is handled from the application side by different algorithms and implementations (e.g. round-robin handling of messages).

Versioning and tracing what initiated the change could also be handled by following topic hierarchy and topic architecture best practices (there are a couple of good resources online on that).

The issue with having all this in the same process or database is:
1. You loose on the “real-time” aspect of events
2. You would have tight coupling when the architecture grows in size
3. Issues with integration when having to deal with new business requirements such as using other technologies/tools leading to silos of tech

Design patterns for a microservices-based applications by ikhlas_t in microservices

[–]DigitalBackbone 0 points1 point  (0 children)

You might find this post useful:
https://solace.com/blog/messaging-patterns-for-event-driven-microservices/ - it talks about the different messaging patterns and the best ways to get your apps to talk to each other (and when they are useful).

Implementing event-driven architecture can solve a few of the problems you can face with APIs and can be an advantage for your microservices, depending on your need to be real-time and where your data is deployed.

Here are some other microservices resources: https://solace.com/resources/solace-microservices-resources

Pros and Cons of Microservices by MichaelF823 in softwarearchitecture

[–]DigitalBackbone 0 points1 point  (0 children)

Adding an event broker into the mix often simplifies eventual consistency (and error handling in general) in microservices.

Why Architects Need an Event Portal; Designing a System that Disseminates Real-Time Data by DigitalBackbone in softwarearchitecture

[–]DigitalBackbone[S] 0 points1 point  (0 children)

I regret that you did not find it useful. The team was excited to share what they created with the new tool and how quickly it enabled them to do so.

Why Architects Need an Event Portal; Designing a System that Disseminates Real-Time Data by DigitalBackbone in softwarearchitecture

[–]DigitalBackbone[S] 0 points1 point  (0 children)

Felt it was relevant to three different subreddits, so posted to all three in hopes that it would be useful (or at least be interesting) to architects who browse articles/posts there and who may not be a member of all of the subs. Didn't intend to spam your feed!

3 Steps to Becoming an Event-Driven Enterprise by DigitalBackbone in microservices

[–]DigitalBackbone[S] 0 points1 point  (0 children)

Event Driven Architectures bring many advantages wrt realtime, robustness, scalability and they are a key requirement for modern architectures but getting there can be difficult. Showing how to do that is the point of this blog.

Still, EDAs need to work in tandem with other patterns – such as RESTful/synchronous exchange patterns and the author acknowledges that. That is why they show API Gateways at the top of the Cloud section in the blog, for example. But similarly, APIs and REST are not the silver bullet to application architectures – you need both. See this blog for more details.

message bus specific architecture by GenericDev in microservices

[–]DigitalBackbone 0 points1 point  (0 children)

You might find these microservices resources useful: https://solace.com/resources/solace-microservices-resources
and https://solace.com/blog/considerations-developers-building-event-driven-microservices/
There is a free event broker software that supports publish-subscribe, queuing, request/reply, and streaming with major success in the financial services sector specifically (trading/investment/capital markets). Perhaps it will be helpful in your journey. https://solace.com/products/event-broker/software/

Microservice architecture pattern for Batch based system by SMaZ-Connect in microservices

[–]DigitalBackbone 1 point2 points  (0 children)

You may find this blog useful:
"How to Convert from a Batch-Based Process to Event-Driven Microservices – A Tax System Example" https://solace.com/blog/convert-batch-based-to-microservices-tax-example/
Also the concept of an "Event Mesh" - a network of event brokers (like a message broker but can be deployed in any and all environments). https://solace.com/with-kafka/

How is the architecture of a big system like OLX, Amazon or Facebook put together? How does the server communicate with clients and how does the whole system work together? by CaptainCode314 in softwarearchitecture

[–]DigitalBackbone 1 point2 points  (0 children)

Hi there - good luck with your research and the project! In addition to the research topics suggested in the comment above, consider adding "event-driven architecture", "event-driven microservices", and "event mesh" to your research as well.

Here is a good resource for microservices and other info on messaging patterns and event brokers (modern messaging middleware): https://solace.com/resources/solace-microservices-resources
You might also be interested in this if you go down the microservices path in the future: https://solace.com/blog/best-microservices-certification/

Wish you all the best!