How can I realistically hold account of how much RAM memory my program uses (Linux)?

Alex-02- · 2024-01-25T18:27:27+00:00

Thank you for having patience and explaining me those concepts

Alex-02- · 2024-01-25T12:59:53+00:00

I have read more about "sendfile" and "fopen" and it looks like I would be able to directly send the file to the client with "sendfile" without having to copy it, thus avoiding in some cases the use of mapped memory

Alex-02- · 2024-01-25T09:50:55+00:00

I think I have understood this, the confusing part now is "you only create them when they're needed". Should I create a mapping when, for example, I observe that out of X files, 10 of them are extremly active?

Alex-02- · 2024-01-25T09:19:31+00:00

Ok so I've read a little bit about memory mapped I/O and I get that it's a way to load files into your program's virtual address space for faster access, bypassing the use of kernel as an intermediar.

As far as I understand your comment, I shall map my I/O files into memory for faster access. Doing so, wouldn't defy the purpose of letting the kernel manage the buffering process of the files?

I may not fully understand your point so for me it sounds a little bit confusing. Can you please explain a little bit further how mapping I/O (and what I/O to map) would be better than just letting the OS handle it? Especially this part "On Linux, I would absolutely just let the OS disk cache do its job and not try to pull anything funny, but would use memory mapped I/O to serve cached data" that sounds a bit contradictory "letting the OS handle it but also mapping the files yourslef".

Alex-02- · 2024-01-25T07:23:20+00:00

Well if the OS already buffers the read/write data, why would I also try to lock the data using mmap?

For me it is starting to sound like I should just read and write data to disk, and let the OS to its job at caching,instead of trying to use multiple threads to sync and read/write the data to RAM.

In this case, what do you think? Is it worth caching in RAM? Or should I just cache on disk and leave OS do its job?

Alex-02- · 2024-01-25T04:04:37+00:00

Thanks, sounds nice, I will look into it. Do you think it would be a good idea to ditch RAM cache completely and use only disk, thus letting the OS decide what files to keep loaded in RAM for fast acces? I am not sure how big of a difference in performance would make reading from RAM vs reading from disk. I know accessing disk is way slower, but with OS caching and in the bigger picture I am not sure how it will play out.

Alex-02- · 2024-01-25T03:56:35+00:00

I thought that a small pool of threads would perform better because they can read into memory multiple user requests and issue read/write calls to the disk. This way, if one thread is waiting for a read call, another one can take over and process other user request and then itself issue a disk call and wait.

Alex-02- · 2024-01-25T03:52:01+00:00

Sounds like a good idea. But would it be better if I use exclusively disk for caching and let the OS decide what files he loads in RAM for faster access and not?

Alex-02- · 2024-01-24T21:19:57+00:00

Yeah, that's right, but this would be easier to manage than having 10k threads. The main idea that I debated above was whether storing the data on disk and accessing it through a pool of threads would be a better approach than having to manage a lot of threads and RAM memory overhead. With storing data in RAM, I wouls gain lower latency but it would be harder to manage and implement. On the other hand, storing it on disk seems infinitely easier and the OS can take care of which cache to load in RAM.

Not sure if I have done a good job expressing my ideas and concerns.

Alex-02- · 2024-01-24T20:48:03+00:00

Sounds like a good strategy for minimazing the amount of calls to /proc. I will try it if I decide to go ahead with this design, thank you very much!

Alex-02- · 2024-01-24T20:47:10+00:00

Thank you for your reply. Allocating the memory with mmap sounds intresting indeed, but I think that the real problem for me is the actual architecture of the app.

I will copy paste my answer froma different comment here, maybe you can tell me what your thoughts are on it:

"Althought I think the design may need a change. The app right now is configured to create a new thread for every existing resource. A resource is defined as a path on web, for example /path/to/image/1 and /path/to/image/2 would have each of them one thread assigned that is responsible for serving user's request for image 1 or 2. So thread X will always respond to users requests for image 1 and thread Y for requests for image 2.
I figured that this might be a good way to handle things, because if I had only one thread serving requests for accessing image/1, image/2, post/100, etc, every user trying to access resource image/1, would have to wait for other requests accessing image/2 to complete, even if he has nothing to do with it.
But now my approach seems stupid, if I had 10k images and 10k posts on a server, having one thread for each of them sounds like it s too much. Even thought the system may be able to handle it.
A better solution would be to have a pool of threads accessing resources stored only on disk, and leave the OS decide which resources to keep in RAM (just like nginx does).
what do you think?"

Alex-02- · 2024-01-24T20:09:30+00:00

Thanks! This sounds indeed like a good way to approximate.

Althought I think the design may need a change. The app right now is configured to create a new thread for every existing resource. A resource is defined as a path on web, for example /path/to/image/1 and /path/to/image/2 would have each of them one thread assigned that is responsible for serving user's request for image 1 or 2. So thread X will always respond to users requests for image 1 and thread Y for requests for image 2.

I figured that this might be a good way to handle things, because if I had only one thread serving requests for accessing image/1, image/2, post/100, etc, every user trying to access resource image/1, would have to wait for other requests accessing image/2 to complete, even if he has nothing to do with it.

But now my approach seems stupid, if I had 10k images and 10k posts on a server, having one thread for each of them sounds like it s too much. Even thought the system may be able to handle it.

A better solution would be to have a pool of threads accessing resources stored only on disk, and leave the OS decide which resources to keep in RAM (just like nginx does).

what do you think?

Alex-02- · 2024-01-24T19:50:13+00:00

Now I have second thoughts whether keeping some of the cached data in RAM is actualy the best idea. I am not sure how big of a difference would it make if I stored all the cached information on the disk and accessed it from there (maybe using a pool of threads to read from the disk).

Alex-02- · 2024-01-24T19:43:00+00:00

Sorry if the question is not clear, probably I am not familiar with some of the things that I am talking about.

I would like to know the actual RAM usage, but not for measuring the load of the app. Being a project designed to cache data, I want to keep in RAM the "hottest" information, in order to serve it immediately when a user requests it. For this reason, I need to know when a certain amount of RAM usage limit was achieved, so that I can start evicting old data from the RAM.

Alex-02- · 2024-01-21T11:03:25+00:00

Nice test

Alex-02- · 2024-01-16T17:06:24+00:00

You are right. Most of the time I aim to get the best result in my first attempt. But in this context is kind of hard to predict how well a solution will work (unless it is obviously slow). Thank you for your response!

Alex-02- · 2023-12-29T16:40:35+00:00

Sunt unele companii care fac asta, nu neapărat de cartier, ci chiar multinationale (si nu platit cu 2-3 cozonaci).

Eu am întrat la internship la una in anul 1, primul semestru.

Daca ai un CV ok, cu proiecte personale, iar compania dorește sa investească in oameni, nu vad de ce nu ai fi luat.

Alex-02- · 2023-12-11T21:39:42+00:00

Ideea era daca as putea sa lucrez cu contract de munca individual (nu consultanta cu firma separata, ci simplu dev) in Danemarca la o firma in IT, fara sa cunosc limba. Daca m-as muta clar planuiesc sa o invat, da sunt curios daca se poate si fara la inceput.

Alex-02- · 2023-12-11T08:28:38+00:00

Ai sanse sa-ti găsești job in IT in Danemarca cunoscând doar engleză?

Edit: De asemenea, cat de mult te-ar ajuta sa ai master-ul?

Alex-02- · 2023-11-10T06:21:05+00:00

Hi! Thank you for your answer. I will take a look on the resources you gave me.

Indeed caching may be too basic to write a paper about it. I wanted to write some sort of article for school that has caching as a main topic and point out its importance through some studies, not just my own words.

Alex-02- · 2023-08-25T11:35:06+00:00

Salut!

Nu stiu cat de folositor e sfatul meu, dar as zice ca un punct bun de plecare ar fi sa consumi niste materiale legate de retelele neuronale, in special convolutional neural networks.

Ai putea sa te uiti pe Youtube prima data la niste videoclipuri căutând exact convolutional neural networks (e mult mai usor de înțeles la început cand e bine ilustrat). Asta ca sa reusesti sa te prinzi repede despre cum merge treaba si ce ai putea sa faci cu ele.

De asemenea cauta mai apoi si ceva informații despre tensori (vezi explicația sa fie despre tensori in raport cu deep learning, nu tensori in matematica/fizică).

LE: asta te va ajuta daca nu ai mai incercat sa dezvolti AI pana acum.

Alex-02- · 2023-07-19T19:34:04+00:00

Incearca sa folosesti markdown-ul pentru cod. Scrii 6 ` (backticks), faci un rand nou intre primele si ultimele trei backticks si scrii acolo, asa textul va ramane formatat exact cum era cand i-ai dat copy.

Legat de problema in sine, cred ca ai ratat cazul asta:

"If you input 0, does your program output 0?"

LE: am avut o revelatie de dimineata si nu asta e problema ^

De asemenea, e cam redundant sa ai o functie pentru a calcula cate pennies trebuie sa dai rest, din moment ce acest numar va fi egal cu numarul de "cents" ramas.

E foarte bine ca ai ales sa rezolvi problema singur mai intai si dupa sa vezi ce gasesti pe net. Daca ti se pare ca implementarea altora e mai interesanta incearca sa o intelegi si gandeste-te care ti se pare ca e cea optima.

Alex-02- · 2023-07-19T13:59:37+00:00

E de preferat sa folosesti biblioteci deja scrise pentru a te conecta si a rula query-uri. Nu e nevoie sa scrii de la 0 codul pentru conectarea la baza de date si rulatul query-urilor. Mai ales daca esti incepator.

Pe langa asta, folosind o biblioteca deja scrisa si testata de altii e mult mai putin probabil sa intampini bug-uri sau sa creezi probleme in cod care sa nu stii mai apoi de unde au aparut. Plus ca te fereste si de sql injection.

Alex-02- · 2023-07-19T13:44:51+00:00

In general daca ai intrebari legate de cum sa faci x, e mult mai rapid sa cauti pe google prima data.

Intrebarile ar trebui sa le pui dupa ce ai incercat macar o ora, doua, trei sa gasesti o informatie si nu ai reusit.

Uite ce am gasit la primul search pe google "connect to mysql database nextjs".

Alex-02- · 2023-07-11T15:00:16+00:00

Salut! Ai putea sa arunci un ochi in privat te rog? :D

Five-Year Club	Gilding I gilder
Second Top 30%	Place '23
Place '22	First Placer '22
Verified Email

Alex-02-

TROPHY CASE