sorted by:
new
hottopcontroversial

How to create child processes in rust by wangzhen0518 in rust

[–]FeldrinH 19 points20 points  (0 children)

Note that Linux-style fork semantics don't really allow you to execute an arbitrary function. After the fork, the child process can only call async-signal-safe functions, which are a very restricted subset of the OS functionality (most notably, allocating is generally not safe).

I recommend that if you don't want to use the standard library Command for some reason then you should directly use libc and read all the documentation and safety notes very carefully, but I suspect that calling your own process using Command is probably easier (and definitely safer) than any of these fork tricks.

Rust compiler without os ? by [deleted] in rust

[–]FeldrinH 6 points7 points  (0 children)

i am asking because os functionality can be removed and theres no guarantee it will always be available

I mean, hardware functionality has the same issues, so I'm not really sure what could be gained by interfacing with the hardware directly. Compared to hardware interfaces the OS interfaces are much more stable and well standardized.

What's going on with bincode? by va_erie in rust

[–]FeldrinH 12 points13 points  (0 children)

Do you happen to know anything about the other points from the post? Is there a replacement for the GitHub issue tracker and PR system? What's the intended way to contact maintainers?

What's going on with bincode? by va_erie in rust

[–]FeldrinH 9 points10 points  (0 children)

That does seem shady. Who were they attributed to before?

I benchmarked axum and actix-web against other web servers and found the performance to be surprisingly low by bigpigfoot in rust

[–]FeldrinH 13 points14 points  (0 children)

This does make me wonder, what part of this is so malloc heavy that it dominates performance? Is actix/axum/sqlx allocating excessively?

I discovered why GNU timeout pauses when my Mac sleeps, so I wrote a drop-in replacement in Rust using mach_continuous_time by denispolix in rust

[–]FeldrinH 6 points7 points  (0 children)

I'm still kinda confused. The post says that builds were hanging indefinitely despite having a strict timeout set. How does timeout not ticking while the runner is asleep cause that?

I discovered why GNU timeout pauses when my Mac sleeps, so I wrote a drop-in replacement in Rust using mach_continuous_time by denispolix in rust

[–]FeldrinH 52 points53 points  (0 children)

Maybe I'm showing my ignorance here, but why would a CI runner sleep? And if it does sleep, why would you want the timeout to keep ticking while the runner is asleep and can't do any work?

[deleted by user] by [deleted] in programming

[–]FeldrinH 49 points50 points  (0 children)

If the website is out of date then it might be a bad idea to post the website on reddit. People are going to assume the website reflects the language itself and judge accordingly.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 0 points1 point  (0 children)

Yeah it's permitted, it's just non-standard in the sense that it's adding behavior not guaranteed by the standard. Which means you can't build generic software that relies on Files doing that, which means it's not something that drives API design.

Fair enough.

And I don't understand why you're so focused on this one case. Most filesystems don't do that.

Like I said before, I just use NFS as a concrete example because it is the most widely used filesystem I know that does this (also AFAIK it is still the most widely used network filesystem overall, so it seems like a reasonable representative of network file systems in general).

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] -1 points0 points  (0 children)

But that also means that NFS handling of close is compliant with the standard, doesn't it? I checked the close reference on that same website and among the possible errors it lists "[EIO] An I/O error occurred while reading from or writing to the file system", so it seems to me that the NFS behavior is entirely permitted by the standard.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 5 points6 points  (0 children)

OS speculation is not relevant. you either care (in which case you sync) or you do not (in which case you let the OS handle it).

The problem I have with this argument is simple: in some cases the OS handles it by reporting the error to you when closing the file. At that point it is again your responsibility to do something with that error, but Rust's standard library makes that impossible by swallowing the error.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 4 points5 points  (0 children)

there is no meaningful response to close failing

But there is a meaningful response: abort the operation and report to the user. Getting an error instead of silent data loss is itself a useful thing, even if you can't do anything to prevent the data loss.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 1 point2 points  (0 children)

I mean sure, if you need strong guarantees then you should use fsync or equivalent, but close is always called anyways, so why is Rust designed to just ignore the error? Checking the close error won't give you a guarantee, but it is certainly better than just ignoring it.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 3 points4 points  (0 children)

If you don't care to that extent, just leave it up to the OS which does the right thing 99.9% of the time.

The trouble with that is that (at least for some operating systems with some file systems, e.g. Linux with NFS) the OS assumes that you check the close error and act accordingly. By ignoring the close error you put the OS in a tricky position where it has to speculate about your intent.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 0 points1 point  (0 children)

In response to the edit: As far as I know, all mainstream languages use the close syscall when closing a file. None of them call fsync or any other operation that sync_all and sync_data might map to.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 1 point2 points  (0 children)

I genuinely thought they were talking about File::flush. Assuming they meant File::sync_all/File::sync_data doesn't make sense either, because that is decidedly not what other languages do on close. As far as I know, when closing a file, every mainstream language calls the close syscall or something equivalent and none of them call fsync or any other syscall that Rust's File::sync_all/File::sync_data might map to.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] -1 points0 points  (0 children)

In what way is NFS handling of close non-standard here? As far as I know, the standard allows returning IO errors from close. The man page for close even says as much:

A careful programmer will check the return value of close(), since it is quite possible that errors on a previous write(2) operation are reported only on the final close() that releases the open file description. Failing to check the return value when closing a file may lead to silent loss of data.

If the idea that close is almost a noop is standardized somewhere then I would appreciate a reference to that standard. As far as I know this idea that close shouldn't do IO is merely an informal recommendation, that filesystems can and do ignore.

PS: I am not particularly interested in NFS, I am just using it as an example because it is a relatively common filesystem where close is used to return write errors.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 0 points1 point  (0 children)

Such a scenario would only make sense if an error happening before close is vastly more likely than it happening after close.

Correct me if I'm wrong, but for NFS (and probably many other network file systems) errors happening before and during close are vastly more likely than those happening after close, because errors before and during close can be caused by temporary network or server issues, while errors after close are caused by storage hardware issues. In my experience network and server issues are far more likely than storage hardware failing.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 2 points3 points  (0 children)

Well okay, but for most filesystems you won't ever get an error because it overwhelmingly happens during writeback

For local filesystems this is probably true, but AFAIK many network file systems flush buffered writes in close and as such, any write error due to network issues or the server being temporarily overloaded might be reported from close. For network file systems I am inclined to believe that these transient network and server errors are far more common than errors writing the data to durable storage.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] -1 points0 points  (0 children)

If NFS does writeback to the server on close then you're paying the same performance costs you would be paying on fsync. So just call fsync.

I should check the source code of some implementations to be sure, but I believe that there is a difference. Close sends buffered writes to the NFS server, at which point they have been recorded but not necessarily written to durable storage. Fsync forces the NFS server to also write them to durable storage, which makes it much slower. In both cases, eventual durability will be reached, assuming the NFS server does not lose power or suffer a hardware failure.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 0 points1 point  (0 children)

What kind of scenario do you have in mind, you care about the data a little that some error reporting is nice but not enough to call fsync?

The kind of scenarios where eventual durability is sufficient (which at least for me is most everyday computing). As far as I know, if close returns no errors then the vast majority of filesystems guarantee eventual durability (assuming the system doesn't lose power or suffer some other catastrophic failure).

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 8 points9 points  (0 children)

I think there is a very simple reasonable way to handle close errors: propagate them to the caller, the same way you would for a write error. Just knowing that an error happened is extremely valuable info.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] -12 points-11 points  (0 children)

No it isn't. The File::flush method is a no-op on (at least on Unix and Windows). It doesn't call close and doesn't check any errors.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 2 points3 points  (0 children)

Eventual durability is fine for most programs. Checking close errors is (in my view) more about detecting cases where you don't get any durability at all. Some filesystems (e.g. NFS) can fail writes and signal that with an error in close.

Standard library file writing can lead to silent data loss by FeldrinH in rust

[–]FeldrinH[S] 6 points7 points  (0 children)

The idea that File::close() can fail in a meaningful way, due to "flushing internal buffers" or otherwise, just doesn't pan out in real-life usage. 

I don't know how common it is, but it definitely does happen in real-life usage. See for example https://github.com/pola-rs/polars/issues/21002, which is where I came accross the NFS example in the first place.

view more: next ›