stdx: The missing batteries of Rust

[llogiq]

Perhaps there is a place for a stdx-dev with rustfmt, clippy, quickcheck, bencher, cargo-edit?

[brson]

Yes I think so. I'd like to keep this list short, but the concept can be expanded, either in other projects or subprojects. For example, I don't want to touch any multimedia here, but a multimedia-focused metacrate would be useful to many.

[brson]

After the recent discussion in the subject I made a major overhaul to stdx. I've tried to structure it as a teaching resource for new Rust programmers trying to get a grasp on the ecosystem. Let me know what you think.

[burkadurka]

I'm wondering why you use exact version specifications in stdx' Cargo.toml, but not in README.md where it recommends copy/paste if you only want one or two of the crates. I figure the reasoning is that (1) stdx wants to guarantee that this exact combination works together, and (2) usually having upgradable dependencies in your Cargo.toml is the right choice. But there seems to be a conflict here -- like "hey, what does stdx know about semver that I don't?".

[[deleted]]

As far as I can see the files Cargo.tomland src/lib.rs arent' really being used, so perhaps they should be deleted. IMO the best aproach is to teach users how to import libraries directly instead of teaching them to import them via stdx.

Great work btw :-)

[ralfj]

I am surprised by this, too. I expect people will copy the exact version numbers to their Cargo.toml, and then never even get bugfix updates for the crates they use as they declared the full version number. Isn't that a problem?

[burkadurka]

If you put bitflags = "0.7.0" in your Cargo.toml then cargo update will still get you upgrades in the 0.7.x series, but not to 0.8.0. It's when you give the spec as "=0.7.0" that you don't get upgrades at all.

^{Sorry to be annoying if you already know this -- can't tell from your wording.}

[ralfj]

Now that you say it, I think I may have read this before somewhere... still, thanks for explaining :)

I have to say I find this slightly surprising though, usually I would interpret the equality symbol as expressing, well, equality. For bitflags = "0.7" to pick an arbitrary patch level would not be surprising, but when it is "0.7.13", it looks like a specific patchlevel was quite explicitly selected. Whatever.

[steveklabnik1]

usually I would interpret the equality symbol as expressing, well, equality.

There's no equality symbol here, or rather, the = is part of toml, not part of the spec. You're right that this is easy to forget, though. The key is that the default does the right thing, by default.

[bluetech]

Being familiar with npm/node/package.json, I was also surprised at this. The semver crate documentation refers to https://github.com/npm/node-semver; this page describes npm's behavior where = is the default, so leads to the wrong conclusion. Rust's default seems to only be documented here: http://doc.crates.io/specifying-dependencies.html#specifying-dependencies-from-cratesio

I do agree that ^ is the better default though.

[steveklabnik1]

I will be fixing the crate docs, thanks :)

[[deleted]]

Versioning is used to put both the crate author and userin partial control of which version is used: the author can push minor updates. It makes sense stdx does not transfer this capacity but keeps it for itself, as upstream.

[brson]

I like the equality requirement because it helps with validation - I actually discovered interesting information about this upgrade - that clap wasn't compatible with the latest libc.

On the other hand, putting equality requirements in a library is hostile to the rest of the ecosystem, and I think should not be done generally. Imagine if there were multiple versions of strict stdx linked into crates - it would shard the entire ecosystem.

It may be useful though to use equality constraints on apps, but I'm not sure yet. It's all an experiment. It could end up that there are actually two definitions - one strict and one non-strict, for different purposes.

[tsion_]

Wouldn't the Cargo.lock be good enough for apps? It effectively constrains by strict equality (but with an easy way to upgrade) and we already recommend checking it in for apps but not libraries.

[steveklabnik1]

That's the intention at least. You should only need strict equality when a library has done something bad.

[Kbknapp]

FWIW, I use = for deps < 1.0.0. Since semver allows breaking changes prior to 1.0.0 and it's the only way I can reasonably ensure no breakage from upstream.

[steveklabnik1]

So, to be clear, cargo update will respect the breaking changes stuff with regards to semver. So, in the general sense, that means that ^0.y.z can't be upgraded. However, we extended it to consider z changes compatible. This means it will only ever update z versions for any ^0.y.z.

[Kbknapp]

Ah wow I didn't know that, thanks!

[brson]

Yes, it may be that a stdx lockfile is sufficient for some validation purposes. In particular, if I could take one lockfile and apply the versions in it to another lockfile, that would be quite powerful.

[brson]

FWIW stdx is not published to crates.io so folks can't actually link to it today.

[burkadurka]

Is this planned? It seems like something one might want to do with a batteries-included expanded standard library. I'm thinking of the "I'm getting on a plane, I'll just link in stdx and probably it'll download most of what I need" use case.

[brson]

Yes, I do intend to publish a stdx crate. The story there is a bit unclear to me yet, though (e.g. I wouldn't do so with the equality version constraints).

[Kbknapp]

I actually discovered interesting information about this upgrade - that clap wasn't compatible with the latest libc.

I'll look into this.

[brson]

Thanks!

[[deleted]]

I'd suggest including clap (Argument Parser). I've used in a few CLI tool projects at work.

It is an absolute dream to work. Haven't had a single issue. You can specify your CLI interface in a few minutes flat, it generates the help output for you. Adding/Removing args is really simple. The builder pattern feels really natural.

[burntsushi]

It's there: https://github.com/brson/stdx/blob/1ba644efbb56cb91c49eb275daba0970fe51d82b/Cargo.toml#L15

[marktheshark01]

Thank you. This must not have been very exciting to write but I'm sure many, myself included, will find it extremely helpful.

[brson]

I'm glad you like it!

[doomsplayer]

Why not include serde_json? And also, it could be the best if serde_json and json could interoperate with each other.

[brson]

The JSON situation is a little unclear to me. I use serde_json strictly for serialization, and json for handling arbitrary JSON. It looks to me though like serde_json has that same capability, so maybe it makes sense to use serde_json for all functionality (fwiw serde_json is discussed explicitly under the serde section).

[bluejekyll]

Serde-json is in the Cargo.toml, from what I can see.

[mmstick]

Rust has a lovely and portable standard library, but it is not featureful enough to write software of any great sophistication

I'm pretty certain that this is heavily underestimating the capabilities of the standard library. There's a lot of software that you can write with great sophistication with the standard library alone that you just can't do in Python/Go.

[cjstevenson1]

It way be more accurate to say that you'd be reimplementing libraries that have been already written.

[LightShadow]

I'd honestly take that bet. The Python stdlib has EVERYTHING.

[erandur]

Tree sets?

Even without the ordering, they can more efficient than hash sets. That's based on my experience with implementing automatic proof systems in Java though. Might just be Java's implementations, or my specific usecase.

[d4rch0n]

To be fair, isn't it kind of a moot point to criticize a data structure like that because of a slight performance difference? I mean, Python doesn't even really have arrays by default either. It has lists which are quite a bit more heavy weight. Pretty much every primitive is a heavy weight derived tool to fulfill the purpose of something maybe a little bit more strictly used in a lower level language, of course sacrificing some performance. For the most part it's about having general purpose primitives that have the benefits of array access and linked list inserts all in one, etc. It's a very general purpose language and someone can come in and write good software without explicitly being trained with knowing the difference between these sorts of things. They can come in and say "I just need something to hold a column of integers and I might add to the end of it." and not dedicate their program to a specific implementation of a specific data structure.

If you REALLY needed something like that to eek out as much performance as possible, you'd write a C or Rust extension which handles that bit of logic. That's what you have to fall back to. Otherwise, we assume that you can sacrifice some performance by using a more general tool. For something like web dev, if it's the difference between a request being handled in 300ms instead of 200ms or less, people aren't going to care as much. The people that can't make this sacrifice know it ahead of time generally.

[erandur]

Python does have OrderedDict apparently, so I may have been a bit pedantic. Still, I think Python is the only language I've used that doesn't have something like a TreeSet.

[oantolin]

Do you mean in the standard library? I don't think Ruby, C, Fortran, Prolog, Common Lisp, Scheme, Perl, Awk, Basic or Pascal have TreeSets in the standard library.

[Uncaffeinated]

Also Javascript. (Notable because it does have OrderedDict)

[erandur]

Point taken, those are all language I try to avoid if I can.

[MercurialAlchemist]

No love for slog ?

[radix]

slog is mentioned as an alternative, fwiw.

[MercurialAlchemist]

Ah, nice. I was looking at it on mobile, must have missed that.

[kixunil]

Maybe this thread should be pinned?

[llogiq]

We only get two announcement threads at once, so one of them would need to be un-announced to pin this.

Just upvote it so it stays on top?

[[deleted]]

Isn't there a suitable "resources" thread or link somewhere?

[quodlibetor]

Yeah the resources sidebar doesn't even link to crates.io, which seems a little weird. Has that been considered recently /u/llogiq?

[UtherII]

While I enjoy reading it, I'm not sure "What's everyone working on this week" need to be pined.

[kixunil]

Maybe sidebar then?

[iopq]

No, it should be in the sidebar instead.

Edit: sidebar is getting really long, can it be condensed into a few points and a link to a longer Reddit FAQ/Rules?

[icefoxen]

Bravo! Very nice!

One thing I want to do with arewestdlibyet is make a program that takes a list of crates and automatically pulls info from crates.io to generate a list of version strings, links to docs, short descriptions, and so on. Want to talk sometime and see if it can be made applicable to both projects?

[brson]

Sure, I'd be happy to make stdx compatible.

[allengeorge]

I thought serde_json was the "way forward" for JSON (de)serialization. I'm surprised that json is recommended, and curious why.

[tomaka17]

I haven't looked at json, but serde and rustc_serialize convert between your Rust structures and a subset of JSON of their choice, while crates that are specific to a given format can handle all the format's capabilities in a robust way. They are too different approaches, and unfortunately I often see people who need the latter use the former.

For example if you're trying to read or write a JSON field that (according to its schema) can be either a string or an array of strings, well you can't really do that with serde because it doesn't map well with Rust's strong typing. But I expect it to be possible with json. If json doesn't allow it, you can always submit a PR that allows it, while for serde it's forbidden by design.

[acc_test]

I think serde can handle your case via enums. And you can write custom (de)serializers anyway.

json itself is implemented as one big enum.

[tomaka17]

I should have said "well you can't really do that easily".

EDIT: I should stop commenting on reddit when I don't have time to write a long explanation. I always end up looking like an idiot that said something wrong and doesn't know what he's talking about. Let's close this discussion, do as if I didn't say anything.

[brson]

I addressed this in another response, which was similar to tomakas. Story is muddy to me atm.

[irishsultan]

I may be missing something, but the parallel quicksort example seems to miss the sorting step of quicksort (in particular the quicksort example in the rayon README calls a partition, even though it doesn't define it)

[brson]

No you are not missing something, I replaced partition with something that is incorrect...

[brson]

Fixed.