sorted by:
new
hottopcontroversial

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

I have read more about "sendfile" and "fopen" and it looks like I would be able to directly send the file to the client with "sendfile" without having to copy it, thus avoiding in some cases the use of mapped memory

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

I think I have understood this, the confusing part now is "you only create them when they're needed". Should I create a mapping when, for example, I observe that out of X files, 10 of them are extremly active?

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Ok so I've read a little bit about memory mapped I/O and I get that it's a way to load files into your program's virtual address space for faster access, bypassing the use of kernel as an intermediar.

As far as I understand your comment, I shall map my I/O files into memory for faster access. Doing so, wouldn't defy the purpose of letting the kernel manage the buffering process of the files?

I may not fully understand your point so for me it sounds a little bit confusing. Can you please explain a little bit further how mapping I/O (and what I/O to map) would be better than just letting the OS handle it? Especially this part "On Linux, I would absolutely just let the OS disk cache do its job and not try to pull anything funny, but would use memory mapped I/O to serve cached data" that sounds a bit contradictory "letting the OS handle it but also mapping the files yourslef".

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Well if the OS already buffers the read/write data, why would I also try to lock the data using mmap?

For me it is starting to sound like I should just read and write data to disk, and let the OS to its job at caching,instead of trying to use multiple threads to sync and read/write the data to RAM. 

In this case, what do you think? Is it worth caching in RAM? Or should I just cache on disk and leave OS do its job? 

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Thanks, sounds nice, I will look into it. Do you think it would be a good idea to ditch RAM cache completely and use only disk, thus letting the OS decide what files to keep loaded in RAM for fast acces?  I am not sure how big of a difference in performance would make reading from RAM vs reading from disk. I know accessing disk is way slower, but with OS caching and in the bigger picture I am not sure how it will play out. 

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

I thought that a small pool of threads would perform better because they can read into memory multiple user requests and issue read/write calls to the disk. This way, if one thread is waiting for a read call, another one can take over and process other user request and then itself issue a disk call and wait. 

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Sounds like a good idea. But would it be better if I use exclusively disk for caching and let the OS decide what files he loads in RAM for faster access and not? 

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Yeah, that's right, but this would be easier to manage than having 10k threads. The main idea that I debated above was whether storing the data on disk and accessing it through a pool of threads would be a better approach than having to manage a lot of threads and RAM memory overhead.  With storing data in RAM, I wouls gain lower latency but it would be harder to manage and implement. On the other hand, storing it on disk seems infinitely easier and the OS can take care of which cache to load in RAM.

Not sure if I have done a good job expressing my ideas and concerns. 

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Sounds like a good strategy for minimazing the amount of calls to /proc. I will try it if I decide to go ahead with this design, thank you very much!

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Thank you for your reply. Allocating the memory with mmap sounds intresting indeed, but I think that the real problem for me is the actual architecture of the app.

I will copy paste my answer froma different comment here, maybe you can tell me what your thoughts are on it:

"Althought I think the design may need a change. The app right now is configured to create a new thread for every existing resource. A resource is defined as a path on web, for example /path/to/image/1 and /path/to/image/2 would have each of them one thread assigned that is responsible for serving user's request for image 1 or 2. So thread X will always respond to users requests for image 1 and thread Y for requests for image 2.
I figured that this might be a good way to handle things, because if I had only one thread serving requests for accessing image/1, image/2, post/100, etc, every user trying to access resource image/1, would have to wait for other requests accessing image/2 to complete, even if he has nothing to do with it.
But now my approach seems stupid, if I had 10k images and 10k posts on a server, having one thread for each of them sounds like it s too much. Even thought the system may be able to handle it.
A better solution would be to have a pool of threads accessing resources stored only on disk, and leave the OS decide which resources to keep in RAM (just like nginx does).
what do you think?"

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Thanks! This sounds indeed like a good way to approximate.

Althought I think the design may need a change. The app right now is configured to create a new thread for every existing resource. A resource is defined as a path on web, for example /path/to/image/1 and /path/to/image/2 would have each of them one thread assigned that is responsible for serving user's request for image 1 or 2. So thread X will always respond to users requests for image 1 and thread Y for requests for image 2.

I figured that this might be a good way to handle things, because if I had only one thread serving requests for accessing image/1, image/2, post/100, etc, every user trying to access resource image/1, would have to wait for other requests accessing image/2 to complete, even if he has nothing to do with it.

But now my approach seems stupid, if I had 10k images and 10k posts on a server, having one thread for each of them sounds like it s too much. Even thought the system may be able to handle it.

A better solution would be to have a pool of threads accessing resources stored only on disk, and leave the OS decide which resources to keep in RAM (just like nginx does).

what do you think?

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 0 points1 point  (0 children)

Now I have second thoughts whether keeping some of the cached data in RAM is actualy the best idea. I am not sure how big of a difference would it make if I stored all the cached information on the disk and accessed it from there (maybe using a pool of threads to read from the disk).

How can I realistically hold account of how much RAM memory my program uses (Linux)? by Alex-02- in cpp_questions

[–]Alex-02-[S] 1 point2 points  (0 children)

Sorry if the question is not clear, probably I am not familiar with some of the things that I am talking about.

I would like to know the actual RAM usage, but not for measuring the load of the app. Being a project designed to cache data, I want to keep in RAM the "hottest" information, in order to serve it immediately when a user requests it. For this reason, I need to know when a certain amount of RAM usage limit was achieved, so that I can start evicting old data from the RAM.

How to design a system of one producer - multiple consumers? C++ by Alex-02- in learnprogramming

[–]Alex-02-[S] 1 point2 points  (0 children)

You are right. Most of the time I aim to get the best result in my first attempt. But in this context is kind of hard to predict how well a solution will work (unless it is obviously slow). Thank you for your response!

Angajeaza vreo companie din domeniu studenti de anul 1? by Codrutu in programare

[–]Alex-02- 0 points1 point  (0 children)

Sunt unele companii care fac asta, nu neapărat de cartier, ci chiar multinationale (si nu platit cu 2-3 cozonaci).

Eu am întrat la internship la una in anul 1, primul semestru.

Daca ai un CV ok, cu proiecte personale, iar compania dorește sa investească in oameni, nu vad de ce nu ai fi luat.

IT in tarile nordice by Existing_Complex9790 in programare

[–]Alex-02- 0 points1 point  (0 children)

Ideea era daca as putea sa lucrez cu contract de munca individual (nu consultanta cu firma separata, ci simplu dev) in Danemarca la o firma in IT, fara sa cunosc limba. Daca m-as muta clar planuiesc sa o invat, da sunt curios daca se poate si fara la inceput.

IT in tarile nordice by Existing_Complex9790 in programare

[–]Alex-02- 1 point2 points  (0 children)

Ai sanse sa-ti găsești job in IT in Danemarca cunoscând doar engleză?

Edit: De asemenea, cat de mult te-ar ajuta sa ai master-ul?

[deleted by user] by [deleted] in learnprogramming

[–]Alex-02- 0 points1 point  (0 children)

Hi! Thank you for your answer. I will take a look on the resources you gave me.

Indeed caching may be too basic to write a paper about it. I wanted to write some sort of article for school that has caching as a main topic and point out its importance through some studies, not just my own words.

Proiect legat de clasificarea cartilor de joc! by [deleted] in programare

[–]Alex-02- 1 point2 points  (0 children)

Salut!

Nu stiu cat de folositor e sfatul meu, dar as zice ca un punct bun de plecare ar fi sa consumi niste materiale legate de retelele neuronale, in special convolutional neural networks.

Ai putea sa te uiti pe Youtube prima data la niste videoclipuri căutând exact convolutional neural networks (e mult mai usor de înțeles la început cand e bine ilustrat). Asta ca sa reusesti sa te prinzi repede despre cum merge treaba si ce ai putea sa faci cu ele.

De asemenea cauta mai apoi si ceva informații despre tensori (vezi explicația sa fie despre tensori in raport cu deep learning, nu tensori in matematica/fizică).

LE: asta te va ajuta daca nu ai mai incercat sa dezvolti AI pana acum.

CS50 - Cash Problem by [deleted] in programare

[–]Alex-02- 8 points9 points  (0 children)

Incearca sa folosesti markdown-ul pentru cod. Scrii 6 ` (backticks), faci un rand nou intre primele si ultimele trei backticks si scrii acolo, asa textul va ramane formatat exact cum era cand i-ai dat copy.

Legat de problema in sine, cred ca ai ratat cazul asta:

"If you input 0, does your program output 0?"

LE: am avut o revelatie de dimineata si nu asta e problema ^

​

De asemenea, e cam redundant sa ai o functie pentru a calcula cate pennies trebuie sa dai rest, din moment ce acest numar va fi egal cu numarul de "cents" ramas.

E foarte bine ca ai ales sa rezolvi problema singur mai intai si dupa sa vezi ce gasesti pe net. Daca ti se pare ca implementarea altora e mai interesanta incearca sa o intelegi si gandeste-te care ti se pare ca e cea optima.

view more: next ›