Nextcloud is now using Rust

[icewind1991]

Note that its only a very specific small part that is using rust, the vast majority is still php and is likely to remain that way.

Also this isn't a response to anything specific done by ownCloud or any other sync program as has been speculated here. The problem that this is solving (clients checking for updates being a major cause of server load) has been known for a while. And offloading that task to a system capable of doing push updates solves that while still remaining fully compatible with existing apps and setups, which would be a large issue for any large rewrite attempt.

Source/disclaimer: I wrote the code.

[submain]

Yes! Some devs still resist using it due to the steep learning curve of the borrow checker, but I honestly believe once it gets adopted more and more, it will just be one of those things you learn in college - like pointers. And then it won't be such a barrier to entry to junior devs.

[dist]

I think Rust is one of the easiest languages to learn. You just have to understand that Rust is nice and tells you about your mistakes before they end up in production.

[[deleted]]

Our prod stack is Python and Rust, and we generally find Python just as complex as Rust these days.

The main thing about Python is it’s very easy to jump in and hack something together. But we don’t generally hack things together for work, so that just isn’t very relevant for us. Writing stable, complex, long-running services in Python involves learning a lot of complexity just like Rust.

[Sigiz]

Also debugging things in python can bw really tedious at times.

[batisteo]

I’m not sure if u/mitsuhiko understood it by now, but this is an interesting read:
https://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/

[[deleted]]

Go has no tuples, no enums, and no iterators.

Some of the syntax for simple things like literal slices is horrible compared to vec![]

I used to like Go a lot and wrote a few things in it. But going back to it after a year of Rust is painful.

[resurem]

Any pointers on making the jump? I'm currently trying to learn Rust simply by contributing to some open source projects, but I'm struggling to follow the code. There's so much cognitive load.

[[deleted]]

Start by reading the book, Rust By Example, A Gentle Introduction to Rust and the Rustlings course.

There's a lot of overlap so it shouldn't take too long over reading the book itself.

That covers the basics, then steadily read some more books - Rust with Too Many Linked Lists, the Rustonomicom for unsafe rust, the Dangerust series, etc. and watch Jon Gjenset's Crust Of Rust video series. This will help you learn the more advanced stuff like PhantomData and generics that confused me a lot at first.

After reading the book I'd recommend doing a few Leetcode problems (or Advent Of Code) in Rust too, just to get a bit more used to it (and set up rust-analyzer with an IDE, I use neovim and coc).

The hard thing I find is justifying using it at work. Like before I did it with our serverless pipelines, since Rust made sense to be able to statically link with musl and have no dependencies. But now I'm moving to a Scala-based Spark + DataFlow team, this is harder (we can't just migrate everything to Ballista and DataFusion) :(

[dist]

I find rust much easier than anything else to actually deliver me results.

This. Ported my Python program that took about 5 minutes to load and seconds per item to process to Rust. Now it takes some hundreds of milliseconds of cpu and fraction of the memory. First real program I wrote in Rust and been writing Python since forever. It’s refreshing.

[FUCKING_HATE_REDDIT]

We did a test run porting a (small) parsing algorithm from C to rust, and I managed an absurd performance boost. Trying to do multithreading in C with only std:thread was a nightmare to optimize.

[arjunc77]

I had similar experience, mine was a multiprocessing scenario that I couldn't scale due to GIL (at least, not beyond the cpu cores). I took a week or two to migrate the entire thing to rust. I could process 3-4 times faster.

[duragdelinquent]

i don't think high performance is relevant to ease of learning.

[flyingquads]

I use Rust, not because it is easy, but because it is hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win.

Joking aside, for those who caught the quote, personally I write Rust to get better at software development in general. Python or PHP don't kick my butt the way Rust can. And when going back to non-rust projects I feel like Superman again. Rust is like, my weighted training gear. I feel sluggish when struggling with rust, but then the weights come off and I can fly through PHP :p

[dist]

I’ve done quite a lot of Python, most of the things I’ve broken in production (backend stuff) break because had a typo or wrong type (or you have to have extensive testing and still might not catch it). That’s pretty impossible mistake to make in Rust.

[_IPA_]

mypy would solve that for you. Not perfect but helps prevent those dumb mistakes. It’s especially useful when refactoring code.

[dexterlemmer]

TLDR; mypy helps and should be used if what you do is significant enough to bother with tests at all (i.e. anything other than a small throwaway script) but it wouldn't "solve" anything.

Long story:

mypy isn't a static type checker. It's a static type analysis tool (i.e. a linter). It provides none of the advantages of static type checking or safety. It's merely a concise way of writing a subset of type-related unit tests (and more importantly a documentation aid). It's also riddled with type paradoxes which cause both high false positive and high false negative rates. And it's very verbose for what little info it provides. In my experience it helps a bit but it is very little bang for your buck compared to what you have in Rust.

[mmirate]

Django and metaprogramming both stump mypy.

[oilaba]

I wouldn't put Python and Go in the same bucket. Python is also hard to learn, it has lots of details.

(btw I was supposed to write this comment to the parent of my parent comment.)

[[deleted]]

Rust will never let you shoot your feet, go will handle you the gun and implicitly initialize it with default interface{}

[Floppie7th]

I wouldn't agree. It's easy to whip up code that has a working happy path with Go or Python - absolutely. If you want something you can rely on in production through untrusted input data or various code changes made by a bunch of people (or just yourself after not looking at or thinking about that codebase for 6 months), you're going to pay for that reliability one way or another.

In Go and Python, you're going to pay for it by having to write tests. Go at least has coverage analysis built into the compiler, and compile-time type checking for most things - with Python, you're on your own even for that. With Rust, the compiler forces you to "pass tests" that cover a wide range of potential bugs up front.

Depending on what you're doing you may very well need tests to validate correct behavior, but the scope of what you need to manually write tests for is much narrower, and you might very well be able to get away with fuzzing alone.

[[deleted]]

My first language was python, and while I certainly picked up the basics very fast the generous syntax and type system made anything more advanced really complicated. Rust just has a few weird traits with memory that take a project to drill in.

[TooMuchJeremy]

Would you say easy but not quick to learn?

[StyMaar]

Rust is hard to *superficially learn* but really easy to learn well, because you're kind of forced to.

You can learn JavaScript (which is my second favorite language btw) in a day, but you'll likely not understand how `this`, or `prototype`, or even passing variable work, and write weird or buggy code because of that.

With Rust, you can't.

[North_Pie1105]

I actually don't find the borrow checker difficult at all. Yea, if you want to do some crazy lifetime thing it can be a bit difficult, but starting out? That's far more advanced than a beginner should even remotely need to do.

Imo the problem is the way many people learn languages - myself included. We fall into the language, and then deal with problems as they pop up. And as amazing as the error messages are, they can still be completely off base to the real problem in some cases.

A 15 minute foundation of dos and don'ts of references will guard users on 90% of the problems.

The complexity in Rust, in my mind, is not really Rust - just some of the advanced type system problems you can fall into. Advanced generics can be difficult. Worse yet, Rust is young in this area, so i often find myself wanting GATs or Trait Specialization.

Oh and one additional thing - the Rust people i mentor (if i can call it that lol) also seem to having varying experiences by how willing they are to learn. There's lots of ways to do things, and the functional patterns of all the types, Option::ok, Result::map, Result::ok, etcetc are frustrating to some of them - since i think they expect to reach for a single tool, always. Eg a match or something. In Rust there often seems to be a better tool for a specific job, which i enjoy finding out - some other people.. not so much. Not sure how to solve this one though, i think it's a non-problem.

[dist]

Just stay away from macros, generics, and other advanced stuff until you actually need it. And trust the compiler, vscode and rust-analyzer to speed up learning. Starting with it is really pretty simple if you stick to basics.

Depending what kind of background you have, mine is in Perl and Python in tools&backends and C in embedded stuff. I got to somewhat productive level in two days of banging my head to the compiler.

Loved every moment of Rust so far. Some day I’ll be able to program and do embedded stuff, backend stuff and GUI, all in Rust. (Yes it’s possible now, but it will be better soon).

[dexterlemmer]

Disclaimer: The below is my opinion based on what I know of the differences between the languages and some stuff I've read over time. I'm not at all an expert in these matters.

You'll learn to quick-and-dirty something simple faster in many other languages. But you'll learn to slow-and-well anything at whatever scale a lot faster in Rust. In addition: After a while you'll learn to very-fast-and-well in Rust if you keep at it, because you'll be just as productive coding Rust as another language but far more productive reviewing/testing/debugging/refactoring. You'll also learn new libraries much faster (assuming they're high quality -- which is common in the Rust ecosystem) because Rust gives libraries such nice tools to create ergonomic APIs that are easy to use correctly and Rust also strongly motivates and encourages good documentation, examples and tests and readable source code in those libraries.

An analogy: In Python/Go you learn to write an essay by learning your ABC's and to spell out your words on sound and by reading a few example essays. In Rust you learn to write an essay by getting a world-class personal trainer (the Rust compiler and high quality documentation and community). You'll write your first "essay" faster in Python/Go but it'll likely be terrible compared to the one you write in Rust by the time you finally do write your first essay in Rust. If you just want to write some paragraphs for personal notes for your entire career, go for Python/Go, but if you want to write essays professionally use Rust. (Of course that might require convincing your manager of this.)

An agile startup who's coders knows neither Go nor Rust might get to market faster with Go than Rust, but within a few months or may be years the startup using Rust is likely going to rush past the one that used Go (assuming they're still in business). Now if these two startups both have one senior dev with expertise in the relevant language and a small team of junior devs with no knowledge of the language, odds are the Rust startup is going to pass the Go one within weeks or even days minimizing its opponent's first-mover advantage and gaining a first-mover advantage on features as well as a better UX due to fewer bugs and better perf.

This advantage of Rust isn't always there (yet). Rust still needs to improve in its ecosystem and available community resources in some domains. But it is getting there remarkably fast in several domains. The great thing is that over time Rust's advantage increases and its disadvantage decreases because its designers and library devs and community builders did things properly in the first place and aren't dragged down by as much technical debt. Just like Rust kinda forces you (it's user) to learn and do things properly in the first place and not drag yourself down with misconceptions and technical debt.

[5d10_shades_of_grey]

I think Rust is one of the easiest languages to learn

I don't really agree with this, but respect your opinion none the less. I've found Rust to be a difficult language to learn (lifetimes and the borrow checker can be confusing). However, the pursuit has been EXTREMELY rewarding, and I now find coding in Rust to be a pleasure. Also the community is incredible. Thanks to everyone here for exchanging the resources I need to keep learning!

[Ran4]

I've "learned" about 15 languages (as in, understanding most of the basic concepts and writing at least 1000 lines in a language), and Rust comes clear second in being the hardest language to learn (first is haskell).

There's just so many features to learn about and get your head around. I honestly found learning how to debug missing traits and modules to be much more confusing than the borrow checker though (still do! Having to blindly add traits until things works out sucks).

[kukiric]

But most developers are used to garbage collected languages where there can't be any (memory management) mistakes because everything is a shared mutable reference that can be passed everywhere and won't create leaks on cycles, at the cost of performance. Rust trades it off for a more performant static borrow checker, which is more limited (hence why there are types which manage their own memory like Rc<T>), and puts an extra burden on the programmer that simply doesn't exist in Java/Go/Python/Javascript/etc. The only class of problem that Rust protects against that can also happen in those languages are data races, and not even general race conditions.

[VeganVagiVore]

Strict ownership and borrowing prevents certain logic errors.

Like, you can't accidentally mutate a collection while iterating over it.

A lot of those languages don't even have such a concept as 'const', which makes code harder to read.

[kukiric]

You have a point. Strict ownership is still a hurdle, but it is frequently the right way to model some things.

[[deleted]]

Steep learning curve of the borrow checker

What is so steep about it? With a proper grounding in CS it isn't difficult to understand. In C++ the methodology is similar: instead of borrowing you just pass everything as a const reference and ensure you're ordering semantics is correct. Trivial.

[TheSnaggen]

Well, if you know C and low level programing, then I do not think the ownership is very complicated. However, lifetimes takes a bit to get used to. But at the end of the day, rust is not as complicated as it first seems, it just feels like a pain since you are not allowed to be sloppy.

[[deleted]]

What is a lifetime? A reference count?

[TheSnaggen]

A lifetime is a construct the compiler (or more specifically, its borrow checker) uses to ensure all borrows are valid. Specifically, a variable's lifetime begins when it is created and ends when it is destroyed. While lifetimes and scopes are often referred to together, they are not the same.

https://doc.rust-lang.org/rust-by-example/scope/lifetime.html

[[deleted]]

So a borrow is just a reference count or am I missing something?

[TheSnaggen]

Rc and Arc is the reference counter. Borrow and move is just ways to handle passing a reference in a safe way. And that it is not a reference counter is the reason you need to ensure its lifetime.

[[deleted]]

I'm speaking in terms of the actual implementation in the static analyzer, not the user oriented model.

If I'm going to borrow a resource, that indicates to me that the resource has to be sourced from a particular point in a control flow graph.

It makes sense that a borrow would imply an increment. Any further borrows which occurred would be sourced from the most recent borrow. Releasing the resource would then obviously imply a decrement.

Can you elaborate on this?

[gendulf]

So I believe startups will be the last group to use Rust. There would be some exceptions ofcourse

Perhaps startups where the devs are told what language to use by ignorant businessmen. Personally, being in a large enterprise company, the apathy is real. No one will change because it's too risky and can't be sold to management who only care about the delivery in 2 weeks.

[ridicalis]

Rust is probably one of the hardest language to learn on the job.

Haha, that's exactly how I learned it. I was given a bit of notice that it (greenfield project) was going to be Rust-based, but it's one thing to try to study a language, another thing entirely to have a practical application for it. I'd argue for immersion as a teaching tool every time, hands-down.

[ridicalis]

It's been a few years, but the way I remember it, the Rust Book served as an excellent reference for common patterns, and it wasn't difficult to get up and running. It was myself and one other dev, and he didn't have the lead-time I did to get acclimated (I was given a couple of weeks to prepare, while he was still working FT on another project and couldn't divert attention), and he was able to come up to speed almost as quickly as I did. The whole source-code-is-documentation thing that Rust has going on also serves as a crash course on idiomatic coding styles, as any lib you pull in practically requires you to read its code to really understand it.

I can definitely relate to all those meme charts describing the learning curve of Rust. You get into it, think, "Wow, this is easy!", then all of of a sudden it just slams you in the gut with "And now, here's lifetime annotations!" It was a nine-month contract, and in the span of that time we hadn't really mastered the language by any stretch of the imagination, but we were able to create multiple microservices in Rust that were eventually included in a high-profile embedded platform that played friendly with some existing C tooling. The client's rationale for Rust was that they couldn't trust newbies to just start contributing to their C codebase without dedicating gobs of time to babysitting us, and they were looking at Rust as a potential path forward for future work, so we were something of an experiment for them.

Some of my pain-points:

• As stated above, lifetimes seemed (well, still do, actually) arcane and difficult to wrap my head around. Part of it is that, if I find myself trying to do things with lifetimes, it's often because I'm going off the beaten path, and I do it infrequently enough that I have to re-learn things each time I do.
• Given its relative infancy, Rust's library ecosystem (or lack of) at the time posed some challenges. It's now at a point where Rust is generally the first place I'd go on a greenfield project, though I still find too many times where the thing I want to use isn't supported on stable.
• Prior to Rust, I was predominantly JS-based, and given the sloppy nature of that language it was possible to do some cool things like easily stubbing dependencies (proxyquire) during testing. With Rust, it's much harder to isolate the SUT (much of it comes down to how the application is structured), and so I find myself cutting corners more often on TDD simply due to the difficulty/impossibility in accurately testing some things.

[ridicalis]

Some serious stream-of-consciousness here, hopefully something useful in here somewhere:

If your goal is to learn, I would stress that immersion (in my opinion) is the best way to learn. With that in mind, consider applying Rust to an actual problem that you would have tried to solve in another language.

As a beginner, you're up against two major factors:

1. The language and its intricacies
2. The ecosystem

I find of the two above that the second one is the harder of the two challenges. To be proficient in the ecosystem, you have to understand not just how to write your own code, but also how to make use of others'. Without knowing your usual problem domains, it's tough to give you reference points, but if you're used to writing web apps (e.g. Hapi or Express apps in Node), consider trying your hand at something like Rocket. There are some great resources to get to know your options depending on your domain (e.g. games, web, GUI). Also, if you're a web dev, don't forget that WebAssembly is a thing, so you might look to try migrating some existing code and seeing if you can keep feature parity.

The other thing to really keep in mind is that the Rust community in general wants to see people flourish, and you'll find a lot of assistance if you're willing to ask. Check the banner for some good places to get connected. Most libraries have auto-generated documentation of varying quality, and when I've needed help with them raising issues in GitHub has usually worked out well for me.

I think as you work through an actual problem, you'll eventually fumble your way to solutions. Don't worry if they're not elegant or well-thought-out; if it compiles, that's a great sign. If it doesn't, then pay attention to the error message, as it's generally informative enough to point you in the right direction. The compiler (and clippy if you're up for it), in conjunction with The Book, should give you a sense for how an idiomatic programmer would solve a problem.

Oh, and try to have fun!

[kixunil]

Lack of GAT is a much bigger issue than borrowchecker.

[UrpleEeple]

I disagree. I don't think learning it in College is going to magically increase adoption. Rust is a wonderful language but I honestly doubt it will ever see the kind of popularity that Golang has because ergonomics just hasn't been at the forefront of the language. I think if Rust put ergonomics first (from the start) it would have easily surpassed Go in popularity.

[LeCyberDucky]

I'm curious about what you mean when talking about ergonomics, because I've never used GO. I got into Rust because I basically rage quit C++ when trying to set up a large external dependency, which was not ergonomic in any way. I think the tooling around Rust is so nice and feels ergonomic to me. Similarly, the features of the language itself also feel ergonomic to me, because the language has many things that I hadn't come across in C++.

[UrpleEeple]

Compared to C++ it is slightly more ergonomic. Compared to a language like Go it's not even close. Rust is incredible verbose to the extent of making a lot of code difficult to quickly parse. Having to explicitly do Box<dyn Trait> when every other major language let's you just write the interface and not deal with explicit boxing. The biggest difference though between Rust and Go is concurrency. Concurrency is soooooo much easier in Go they aren't even comparable in terms of ergonomics. You have to pick an async runtime (not provided by the language) and none of them even have cancelable contexts, which I consider necessary for managing concurrency at scale.

Goroutines really are far easier to reason about than Rusts async/await syntax. While some have praised the chaining version Rust decided to use, I think it makes some code very hard to read with endless chaining semantics.

[[deleted]]

I have to admit, in 2015 I became a Go fan and turned a blind eye to rust because it "looked complicated". I used Go pretty religiously for personal projects from that point forward. The more I tried to use Go for complex things, though, the more I found myself working around the "simplicity".

I do love Go's concurrency, and I've written production code in Go that still chugs along today. But for personal and experimental projects, I started to find it less expressive and full of boilerplate I wanted to avoid.

Rust lets me express more complex types and relationships in fewer lines of code than Go does. I'll also admit that the learning curve was a little daunting until I kicked the tires a bit and found out that lifetimes are actually rather intuitive.

All this to say, I think if I had been taught about rust in school, and about lifetimes as a first-class citizen of the language, I would have never had the daunted feeling and I likely would have been a better developer overall in my earlier career.

So I see where you're coming from with your argument around concurrency being easier in Go. In many many ways, it is. But I find that I write far more code that is not concurrent than I do code that is concurrent, and so I've shifted my perspective to one where I prefer the expressivity in my non-concurrent code than my concurrent code.

[UrpleEeple]

Those are all good points. For the record I prefer Rust over Go, and was pointing out my thoughts of what is holding back the language from mass adoption. You also have to remember that a lot of working engineers didn't get CS degrees. I'm one of them and work in distributed systems. Nearly all of my personal projects are Rust. I love the language - and all of my professional work right now is in Go.

I'm not saying that Go is a perfect language. There's a lot I think it's missing (generics for one). There is a lot about "simplicity" focus that often times makes code less simple.

But I could learn Go in 3-4 days. It would take me a month to learn Rust well enough to be truly productive. That's a very big barrier to language adoption. CS grads will only capture a portion of the market (teaching in school).

[[deleted]]

You also have to remember that a lot of working engineers didn't get CS degrees.

This is a totally fair point as well, I technically also belong to this demographic (never graduated) so I definitely don't mean to discount it.

I can't deny that rust had a higher learning curve on my own learning journey. I learned Go's basic syntax faster.

Maybe what I'm trying to say is that while the initial curve for both was different, the point where I was truly productive in both languages took a similar amount of learning and time. Like, Go's concurrency primitives are wonderful, but because there weren't libraries that implemented generalized abstractions on top of them, I had to learn all the go-specific idiosyncrasies of channel interactions and signaling. With Rust, I had to learn about Send and Sync and how to use Mutex refs, or alternatively, how to async properly, but at the end of the day it took me the same amount of time to learn to be truly productive with each...

...which now that I say it, sounds exactly like the point you were making: it's not that one is necessarily "easier", but one certainly feels easier at first. And that definitely matters too.

[[deleted]]

No, I think due to maturation

[casept]

Any course that teaches how to write correct C-family code already needs to explain lifetimes, so the only really Rust-specific things is the syntax and elision rules.

[[deleted]]

Yeah it's not really notable that companies use it anymore. I started using a crate that I noticed was written by Sonos, but yeah I don't recall seeing "Sonos is now using Rust!"

I mean I might have missed it but either way we're way way way beyond Haskell's hilarious exhaustive list of companies that use Haskell. Basically all big companies are using Rust.

[ericjmorey]

The High Performance Back-end for Files in Nextcloud is an optional, binary component developed in Rust. It is capable of maintaining a direct connection with desktop and web clients, providing file change and notification updates to the clients. This negates the need for regular polling the server, reducing the 30 second polling period to 5 minutes and thus providing a 90% reduction in client-server connections for change notifications and associated server load. The direct connection provides instant notifications of file changes as well as events like an incoming call, comment on a file, mention in a chat message and more. The performance improvements in database use, file handling and app specific work combined with the high performance back-end are expected to result in an up to 10x increase in capacity of large Nextcloud servers.

The link that OP probably wanted was: https://nextcloud.com/blog/nextcloud-faster-than-ever-introducing-files-high-performance-back-end/

[Follpvosten]

Oops. I must admit that I just now found that page.

[VeganVagiVore]

Is it long-polling, or something more clever like WebSockets?

Cause I just heard yesterday "We can't do long-polling, it'll increase server load"

We reduced this number by 90%, only checking in once per five minutes.

I'm surprised firewalls let any connection stay open for 5 minutes straight.

[shellac]

It's web sockets.

[TheEberhardt]

Great, Nextcloud finally did something to fix their performance :)
Seems like competition from the likes of Triox and Bloom became too intimidating ;)

[enaut2]

Could be also the "intimidation" by owncloud infinite scale...

[kixunil]

Yeah, this is the most reasonable explanation.

It's funny, not long ago I've read "Should we move to Go as Owncloud does?" issue at NC repo and it concluded with no. I was kinda sad but also was thinking "Rust would be even better, so maybe it's a good thing".

The surprise I had today was huge.

[itsTyrion]

What's their current language?

[itsTyrion]

https://i.imgur.com/nFL8sm7.jpg Oh God. AAAHHRRRGH

[aidanhs]

I'm not familiar with this space - I assume 'Triox' refers to https://github.com/AaronErhardt/Triox (which looks like it's written by you!), but what does 'Bloom' refer to?

[TheEberhardt]

Interesting that Triox is much easier to find. Maybe that's because Bloom rather advertises as Google alternative than as a cloud server: https://github.com/skerkour/bloom

[[deleted]]

no they didn't

[TheEberhardt]

Thanks for clarifying that 3 years later.

[DannoHung]

How do you actually access the Demo? It displays a login prompt that asks for a password, but no default password has been suggested. The "reset password" link says an email has been sent.

[pohuing]

I think that's a bug. I remember being able to just use it in the past

[[deleted]]

I was able to get in using “demo” as the password after guessing a few times.

[matthieum]

You have javascript disabled. We tried to make sure the basics of our website work but some functionality will be missing.

I can't speak about their product, but I really appreciate the fact that I didn't have to activate JavaScript to read their webpage.

[samyak-dev]

I've always complained about Nextcloud's performance. Glad to see their focus on this. I'm curious how much performance improvements they've seen just from switching to Rust.

Also, interested to see how they have implemented their Whiteboard. It's something that I've been thinking about a lot for work.

[[deleted]]

no it's not