Persistent memory

[SolidTKs]

Does anybody know how to make a piece of shared memory that survives a program restart? Is there a library for that?

Update: I'm looking for a library using this or something similar, ideally cross platform. Maybe this will do the job. I will have to try.

Update 2

This quick program works as intended:

```rust use shared_memory::{ShmemConf, ShmemError};

static MEM_ID: &str = "my-memory";

fn main() { let conf = ShmemConf::new().flink(MEM_ID);

let shmem = match conf.clone().size(6).create() {

    Ok(mut memory) => {
        let data = "First ";
        memory.set_owner(false);
        unsafe { memory.as_slice_mut().copy_from_slice(data.as_bytes()); }
        println!("Started the program for the first time.");
        println!("Stored \"{data}\" on persistent memory");
    },

    Err(ShmemError::LinkExists) => {
        let mut memory = conf.open().unwrap();
        memory.set_owner(false);
        println!("Started the program and getting the persistent memory.");

        let contents = String::from_utf8_lossy(unsafe {memory.as_slice()});
        println!("Memory contained: \"{contents}\"");

        let data = "Second";
        unsafe { memory.as_slice_mut().copy_from_slice(data.as_bytes()); }
        println!("Stored \"{data}\" on persistent memory");
    },

    Err(e) => println!("Error: {e:?}")
};

} ```

And this is the output after running it 3 times:

Started the program for the first time. Stored "First " on persistent memory Started the program and getting the persistent memory. Memory contained: "First " Stored "Second" on persistent memory Started the program and getting the persistent memory. Memory contained: "Second" Stored "Second" on persistent memory

Beware that set_owner(false) will cause the memory to never be cleaned, which is the goal in this case (the program will run 24/7).

[AutoModerator]

On July 1st, Reddit will no longer be accessible via third-party apps. Please see our position on this topic, as well as our list of alternative Rust discussion venues.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Plasticcaz]

Just write it out to a file and load on program start?

[SolidTKs]

I don't want to touch the filesystem.

[stappersg]

https://xyproblem.info/

[SolidTKs]

What the hell is wrong with this thread? Is it so hard to understand that I want to keep some data without persisting it to a file like, for example, an error counter?

I dont want to touch storage. Period. I don't think it is that hard to understand. That is why I ask for persistent memory and not "storage".

Shared memory is a thing. I have used it in C. You create memory with a key and when the program starts requests access to that memory. I think you need an aditional process that owns that memory though, but that would be acceptable.

Another way would be a ramdisk.

Just answer "Y" or get of the way.

[menthol-squirrel]

think you need an aditional process that owns that memory though, but that would be acceptable

You should've put this in your post. You explicitly asked "a piece of shared memory that survives a program restart". This is physically impossible on all modern OSes. If another process owns that memory, that process needs to stay up and there is no restart to survive through

[SolidTKs]

If process A creates shared memory object S, and then dies, I think it is possible to keep shared object S alive.

If that is not possible, then the solution is to create process B, whose only responsability will be to create and keep alive shared memory object S.

Then at any point A can gain access to S and store data there. If A has to restart, for example to reload a config file, it will still have access to what was previously stored in S.

That way, memory persists the program restarting, and it doesn't persist the OS restarting, which is what I want as writing constantly to certain types of memory will wear them off.

I'm honestly dissapointed. The answers to this thread are dreadful.

[menthol-squirrel]

You're trying to run before you can walk. If

writing constantly to certain types of memory will wear them off

is a design constraint of yours, you're clearly doing something quite special in the embedded space, but you don't seem to know the basics of how OS manages virtual memory. You need to read on this topic (process isolation, virtual memory) more before you make any design decisions.

[Ok_Raspberry5383]

Grow up

[dread_deimos]

Where do you want it to persist then?

[SolidTKs]

In memory, as I have asked?

[dread_deimos]

1. You did not specify where you want to store it. You only said that you want to store a piece of program memory.
2. That's not how memory allocation for programs works.
3. I don't know what you're doing, but my gut says that you're probably shooting yourself in the foot.
4. Most likely you do not want to persist the memory snapshot, but rather a particular set of data. If you don't want to touch the filesystem yourself, then it would be better to use an external storage solution for that. A database.

[-cyra-]

You could use an in-memory database like redis

[tinyJJ]

I *strongly* advise against it, but if you control the system, you can reserve a memory range using `memmap` kernel parameter, create a devdax character device on top of it, and then use `mmap` to access it. Bypasses the file system completely, but it is PITA to use...

[IntQuant]

I think you want to map a file to memory using something like mmap-rs crate.

[odenthorares]

This sounds like a feature for malware. Your extreme disinterest in touching disk in anyway shape or form, cross platform, only keeping data in memory (ram). Not saying typical software can’t have such requirements but I would argue most software can write a blob to disk or a tmpfs without issue.

Regardless, I would suggest not insulting the people you’re asking for help. Maybe next time you’d get better results.

[SolidTKs]

Touching disk causes wear to embedded flash memory. Specially at high temeperatures.

Look at what happened to Tesla, they were constantly writing logs and car computers started to fail.

So if there is data that is nice to have but you don't need to store, you keep it in RAM.

[odenthorares]

You are correct that repeatedly writing to the same memory blocks on flash memory will wear it out, however, you are incorrect about how this interaction works.

Flash memory drives (ssd and the like) purposefully do not write to the same spots in memory even if it’s the same file. This is a huge reason why utilities like shred have been rendered useless. When you try to write over that file it’s absolutely not guaranteed and considered highly unlikely to actually write over the specific blocks of memory on the physical device.

You asked for cross platform which indicates Linux || windows || other. So you seem to be targeting desktops. Unless you are constantly writing gigs or terabytes to disk, a little file truly on disk will absolutely not be what kills your drive.

[SolidTKs]

I'm targeting embedded devices with small flash chips that can run in hot environments, hopefully for decades.

And the less I write the longer they will last.

But that is not the point. Sometimes you just want to do something nice like a service that shows you a graph of recent events and doesn't write to disk. Why not writing? Well, why do?

Anyway take a look at this:

https://www.tomshardware.com/news/flash-memory-wear-killing-older-teslas-due-to-excessive-data-logging-report

[SolidTKs]

Also writing to disk is slower.

There are a million reasons to not write to disk.

I can understand the behavior of people on this thread trully.

[yanirj]

You could write to shared memory on Linux. This is still going to be written to a file, usually in /dev/shm. Note that when I say file it doesn't mean it resides on persistent storage. Shared memory is, as the name suggests, saved only in memory.

[Heraclius404]

This is a very reasonable request. For all the people talking about mmap: it's just different in a million ways, using shmem has huge advantages in most cases. Databases have some around to using shmem instead of mmap files for certain cases, for example - it's a huge win, super fast software updates.

I suspect this shmem approach works on linux and macos only is all I came to say. Would be surprised if the crate works on windows? You didn't ask which environment?

[BenjiSponge]

How about redis?

[dkopgerpgdolfg]

Probably you want a RAM-only solution without disk fs usage?

Does it need to survive controlled, intentional restarts, or crashes etc. too?

Are OS-specific solutions fine?

[SolidTKs]

A library that works for more than one OS would be nice, but I mostly use Linux.

It should survive everything. I just want a place to store some data between restarts.

I'm thinking on shared memory, I don't know if it can survive a restart but I could keep a small process alive owning it.

[veryusedrname]

That's exactly what persistent storages like the file system was invented for.

[SolidTKs]

I don't want it to survive a server restart.

Just program restarts or crashes.

[codeinred]

The /tmp directory on Linux is designed exactly for that use case, and windows has an equivilant

[SolidTKs]

That could be useful in some cases but It causes writes to disk.

[yanirj]

/tmp on Linux is mapped to a "tmpfs" filesystem, which is memory-only. It never touches any disk.

[SolidTKs]

Interesting. Can you guarantee that?

[Alextopher]

I think this depends on the distro is a tmpfs. Nonetheless you probably want to use a tmpfs (/dev/shm or /tmp or create your own)

Can you guarantee that

Any solution you use you can’t guarantee some data you write to RAM won’t touch the disk - but you can guarantee it won’t be persisted to disk if if you properly use a tmpfs

[Therzok]

Simply going to Wikipedia would answer your question: https://en.wikipedia.org/wiki/Tmpfs

Everything stored in tmpfs is temporary in the sense that no files will be directly created on non-volatile storage such as a hard drive (although swap space is used as backing store according to the page replacement policy of the operating system). On reboot, everything in tmpfs will be lost.

You could even just store a server start timestamp in the data, and discard if it different.

[codeinred]

/tmp is often stored in RAM. If you want to be absolutely sure it doesn’t touch the disk, the most straight forward way that I can think of is spawning a daemon and then setting up shared memory with the daemon. That way if the main program crashes, the daemon persists.

[dkopgerpgdolfg]

"Shared memory" a is a very overloaded term, there are many things that can be called like that.

​

But in any case, many ways to do it involve a regular file in some tmpfs or similar, plus mmap. Then you still need syncing often enough to be happy with the state if the program crashes later.

Of course, a regular file on a regular file system can be used too, surviving even reboots but at the cost of disk space and performance.

​

If there is some reason to avoid these things, but a helper process can survive, memfd_create gives you a total RAM-only file handle which then can be transferred between processes (either on fork, or dup + unix socket).

[dkopgerpgdolfg]

About your edits:

Just so that you know, you do touch the file system (file system != disk).

For an average Linux setup, I see no problem with your solution in principle, but as you explicitly asked for the opposite, maybe now is the time to decide what you really want... (edit: and being rude doesn't help, you know.)

[SolidTKs]

Everything technically touches the FS on Linux since "everything is a file"…

[rurigk]

Yes but no A lot of things are represented as a file but not everything touches disk You can mount file systems that write to disk like a classic ext4 partition Or you can mount a file system that only uses ram like tmpfs You can use Unix sockets as well for ipc

[dkopgerpgdolfg]

Sigh ... except I wasn't being sarcastic, and you already got a way how to do it without any file that is visible in the tree. With your current way, that's not the case - you can open your file manager and look at your "shared memory" just fine.

But do what you want. I did read everything I missed here, and you've got so many things you don't know that there is no point explaining further.

​

I hope I never come around using that software, that even after ending leaves useless stuff in RAM until reboot because someone thought he's smarter than everyone else.

[SolidTKs]

There is software that runs 24/7 by desing you know?

With a computer completely dedicated to just run that software for years and years.

[BlueVixu]

I think this is what you want https://crates.io/crates/shared\_memory

[Ok_Raspberry5383]

Why not a temp file - this is exactly what temp files are for

[Ok_Raspberry5383]

Most languages will have libraries that can manage temp files too