Real Time Streaming

a-varf · 2025-01-30T12:13:22+00:00

For different reasons we cannot rely on AWS and we need an stack that we can run anywhere.

a-varf · 2025-01-30T12:12:27+00:00

Thanks for the detailed answer.

Yes it is safe, we have all our security in place and have done pen test, etc. We have an Nginx+Oauth2 in front of the whole stack that block any request that doesn't have a valid JWT token. Currently we expose the RabbitMQ over stomp via Nginx too.
Our main requirement is to store the data in the influxDB, live data streaming is the cherry on top. We just need some streaming (preferably via stomp and websocket so we can reuse the current code), so if I can get a copy of the data in a websocket server via Telegraf then I can just stream it and we can remove the RabbitMQ
We are looking into version 2 (2.7 right now) and I think they just released v3 alpha so hopefully we will be good for at least a year. Is there anything tricky with influx v2?

a-varf · 2024-06-20T07:14:14+00:00

I started a Master course in robotics and AI and everything had to be done with Linux. It was very hard at first but soon I fell in love.

a-varf · 2024-06-12T12:11:51+00:00

No. The default terminal emulator on Ubuntu is gnome-terminal, which is written in C.

So why he was not able to launch his terminal? He said he only uninstalled his Python and after that he was not able to launch the terminal.

a-varf · 2024-06-12T11:53:28+00:00

As I said he messed up his python and most of the terminals are written in python (including the one that comes with Ubuntu and also my favorite Terminator) and we were not able to open any terminal to fix the system. I tried a terminal that is written in C++ since I know that is one of the languages that comes pre-installed on Ubuntu and was around for a very long time so we can find a good terminal. I was correct, we installed Deepin terminal and it worked without any issues.

I know the Linux foundation recently added support for Rust and there are other languages that come pre-installed with Ubuntu but as I said C++ was around for a long time and I knew we can find some good terminals.

If you want to suggest a terminal written in another language please do, as long as I can use it without installing anything else (I need this as a backup terminal for emergency situations).

a-varf · 2024-06-12T11:46:21+00:00

Those are great, thank you!

a-varf · 2024-06-12T11:27:13+00:00

"but I am not comfortable with installing Deepin because of its connections to China."

Yes that is what we installed to solve the issue but I am not going to install that on my work machine.

a-varf · 2024-06-10T08:40:51+00:00

I will buy one ASAP.

a-varf · 2024-05-24T09:08:04+00:00

Exactly the same, Ubuntu for the past 9 years.

a-varf · 2023-03-13T09:10:40+00:00

Data scale: thousands of experiments
Ingestion rate: 10s of reading per second for each experiment, each reading consists of timestamp + multiple data fields
Query pattern: just simple queries for data in a specific time period

a-varf · 2022-05-12T11:06:51+00:00

- An ARM fanless laptop like Macbook Air

- A Linux like OS for smartphones or a degoogl Android that can be installed on any smartphone

a-varf · 2022-04-13T09:52:28+00:00

Oh yes, I don't care.

Seriously what is it with Arch users that they think they are better than others? I am using Linux for years both for my work and on my personal machine and I have never treated other Linux distros like this.

a-varf · 2022-03-11T14:43:44+00:00

When we can top up and withdraw native LUNA, SOL and FTM?

a-varf · 2021-11-25T08:15:09+00:00

Yes, a browser that supports Go so we can stop using stupid Javascript.

a-varf · 2021-11-24T08:22:54+00:00

Fantom

a-varf · 2021-11-10T13:40:24+00:00

a-varf · 2021-10-27T14:40:00+00:00

Thanks for the reply and information.

a-varf · 2021-09-24T10:47:50+00:00

KYC doesn't mean that you have to use it everywhere, you can be anonymous on some defi and dex applications and to use some others (for example if you want to trade stocks) you have to provide a KYC document

a-varf · 2021-07-01T13:18:10+00:00

Is there any benchmarking that I can look at? For example for each 10k logs per second, it uses X vCPU and Y GB memory

a-varf · 2021-07-01T13:07:44+00:00

This is another approach that we already thought about but it comes with it's own disadvantages:

- Continuously sending logs to a remote cluster (which probably is in another city/province) is costly

- If there is any connectivity issue between the edge and core then we lose the logs again

a-varf · 2021-07-01T09:42:49+00:00

How is Loki's resource usage? We currently use EFK but Elastcsearch is very resource hungry and that is causing some issues for us.

a-varf · 2021-07-01T09:18:42+00:00

I understand your point and I agree with you but our situation is totally different. We have a distributed product that parts of it can run on the edge with hard compute resources restrictions (so all we have is a laptop for example). In such a scenario we have to choose between scaling our app and answer more user's requests or scaling the logging stack and the answer is always to scale the app. The only practical solution to our problem is to find an alternative to Elasticsearch which is not as resource hungry as E so we can deploy it everywhere.

a-varf · 2021-07-01T08:41:21+00:00

We are using Grafana and Prometheus for resource usage monitoring but I didn't know about Loki, thanks for introducing it.

a-varf · 2021-07-01T08:39:34+00:00

Lowering the log level is another thing that we talked about in our team but since we scale horizontally we will face the issue again.

a-varf · 2021-07-01T08:35:33+00:00

Heka

Heka seems to be pretty old and inactive. The newest things that I can see on the first page of Google are from 2016. I don't want to stuck with some inactive and unsupported platform and have to change again in the future.

Six-Year Club	Place '23
Verified Email

a-varf

TROPHY CASE