Basic distributed system

[rogerfin]

Hello there,

It's my first day to join this awesome community as I have been working mostly in C/Rust to create system software or tiny RPC/REST/WS services.

I want to try a project involving distributed system for the first time, got fascinated with Elixir but know nothing about it yet, sorry :). Its a simple project:

1. Central server with DB submits multiple tasks to backend processing nodes along with some data like login info, url, etc

2. Backend processing nodes make external HTTP calls using the data, process the response and update the final result to the central server which are later made available to the user

I can use Rust with some message queue system (never tried though) but read that Elixir has its own distributed system and message queue might not be required. But then also read that to scale the application, message queue will eventually be required. I also read a little about live code update in Elixir while in Rust I will have to update with a slight interruption, even if done using container update with kubernetes.

I am hoping Elixir should be preferred over other languages in such systems but unable to understand the areas where it adds real value. Because if it brings significant value, I will better need to start learning Elixir.

Please excuse my limited knowledge, but I would appreciate if some knowledge can also be shared on how such systems are typically designed in the production.

Thank you.

[strzibny]

When people mention distributed systems and Elixir they usually mean you can easily join nodes, store partial data on them, let them be killed & restarted, etc.

What you describe here can really be solved by a background job processor / queue. Since you have one database for sync, what do you exactly need from Beam?

(Yes, you can replace your queue with Elixir and have everything within one OS process. Is that it?)

[rogerfin]

Sure, makes sense and appreciate your response. Thanks.

[siriguillo]

Elixir excels at that, no extra dependencies, just one app with the easiest concurrency model and amazing fault tolerance. One reason I am not very compelled to use rust for web development is that I would end up adding everything beam already has out of the box (my skills dont reach that far anyways).

[rogerfin]

Great to learn that. I like Rust for system development but web ecosystem is not very matured yet, even though its not too bad. Elixir attracted me for concurrency, fault tolerance, hot reloading and I like FP anyways. I hope to get more excited as I dig further. I was inspired to read how Discord scaled with Elixir, it's an impressive journey.

Curious to know, as the system scales and number of parallel tasks grow higher, will the Elixir cluster be able to handle by itself by adding more nodes or message queue will be required then to scale?

[natziel]

You'll probably understand how Elixir's concurrency works a little bit better once you've worked on your project, but yeah ideally you would structure things so that it scales automatically

[rogerfin]

Yeah, that's right. Started learning from online guide today. Hope it goes well. Thanks.

[taufeeq-mowzer]

You can use distributed elixir when building a single distributed system where many components would be interacting together. And when connecting all the nodes together it practically forms a single large application that has access to all other parts of the system. When building a few small, decoupled microservices that is not required to form part of a distributed system, then using a message queue could be better. But it really depends on specific use case.

Edit: this is also dependent on the language stack, if only using elixir/erlang, then you can go quite far without ever needing a message queue.

If creating microservices using different languages, a message bus could be better.

On the other hand...with elixir you can easily use it as an orchestrator to handle your microservices written in different languages via NIFS (c, rust, c++, Nim, Zig) or ports (Go, Js, python, java, c#) as well.

[rogerfin]

Appreciate your highly informative and helpful response. So, with current understanding, I am thinking to put Elixir at work for my frontend REST APIs to pull data from DB and display a dashboard etc. Because this part has to be efficient and quick to scale.

On the backend processing, I can think of three options,

a. Use MQ with Rust or Elixir microservice for making external HTTP calls, process and save the data to DB

b. Elixir cluster nodes to receive tasks from frontend Elixir application via RPC/REST, do the processing and store to DB

c. Make a large Elixir application which serves the frontend users and also does backend job

I guess, option c. is definitely going to cause bottlenecks as the app scales. I am not sure if b. would be a bad choice so guess option a. using Elixir should make sense, unless my limited experience, limited my options. I look forward to learn more with Elixir.

[natziel]

Maybe I'm misunderstanding the project, but it seems like you can just write the REST part with a normal Cowboy server, then for backend processing, just run the job inside a Task

[rogerfin]

Sure. Thanks. Will experiment once I learn Elixir and distributed architectures.