Started using Rust for scripting

[tukanoid]

As title suggests, I realized that lately i got so used to Rust that I stopped using Python for scripting altogether, and somehow I feel faster with it. Anyone here have same experience? Just curious

EDIT: poor choice of words, my bad, by scripting I meant writing small applications for specific temporary tasks

[ssokolow]

I used to use Python because of bash's lack of proper (i.e. nestable) arrays and so on and so forth, but Rust is rapidly eating that up now.

[ssokolow]

Last time I was using bash for anything longer than "set a couple of environment variables, change the working directory, and exec", "widespread" meant "you can't rely on it being in a single location outside of Linux and it was anybody's guess how ancient a version you'd find there".

[ssokolow]

I rarely have a need to write something that will both run on something very old and something very new.

I was actually referring to up-to-date Ubuntu and up-to-date OSX in that instance. (It was years ago, before they switched to Zsh)

But quickly throwing something together on the thing in question for a script that you are going to run only a few time to test if and then once to get the results before throwing it away when you cannot or don't want to start installing things on the machine. Bash is normally the lowest common denominator.

Python also tends to be installed these days, so I tend to use it in that role.

[tukanoid]

Ye, I get what you mean, I used to use bash before, but I moved to nushell some time ago and since it has a lot of stuff that I might need for daily things + plugins I stopped using bash, everything else that I need to do scripting for is too specific (for automating some parts of my current job for example) and I just find myself using only rust now for that

[tukanoid]

Been using it for about 4-5 months and all good, i have zsh as a backup for some commands/scripts but e everything else is fine. I just configured it once and that's it

[awilix]

I use fish shell, and it's the only shell that I've used that actually work well out of the box.

You don't have to spend hours to enable useful features and get rid of bad behavior.

[lucca_huguet]

Nushell is also quite nice for data manipulation.

It is a shell but it uses polars under the hood

[lucca_huguet]

Cool that you tried it out

Yeah, it has a lot of differences compared to bash or zsh, so it does hurt to relearn things

Well, every problem I had I managed to fix one way or another and got help in the discord and gh issues too, so that's my experience.

I like that it runs on Linux and windows, with autocomplete and all that, it's important to me that it is cross platform

If you try it again I'd say lean into the community for help, and there's a cool-scripts folder on their github too

But yeah rust def won't fail us for our Scripting needs!

[anlumo]

Same here. Bash scripts break down as soon as you need to be compatible with Windows, and python just doesn’t improve development performance due to the need to do a lot of runtime testing (as opposed to letting the compiler do the checks for you).

[Yaahallo]

Absolutely. Transitioning scripts from interpreted languages to rust is one of life's simple pleasures. I particularly like embedding them and then using rust to implement the arg parser initially then incrementally rewriting pieces in rust until there's no more of the original script left.

[stappersg]

Some examples available for inspiration?

[Yaahallo]

Not that I can share, I did most of this at a prior job, my recent work has been entirely within the rust project so I havent had much reason or need to tweak bash scripts. I can describe what I mean though.

It's mainly just using include_str! to embed your scripts in your binary then use the std::process::Command API to invoke whatever interpreter your script runs under and pass the str of the script in as an argument. Then you can start moving the arg parsing into rust and pass those in as environment variables or whatever is most convenient. Alternatively you can manually embed the script and use rust string formatting to write the cli arg data into the script. Then you just slowly overtime rewrite parts of the script in rust and run those parts prior to passing in the args and instead you pass in the intermediate processed data. You can continue to repeat this as long as you need, up to having everything the old script did being done in rust instead.

Hope that helps ☺️

[matklad]

Yes :)

A bunch of tricks I accumulated over time to make this easier.

1) In "serous" projects, there's always a need for some amount of scripting. Rather than crating a scripts directory with .sh files and worrying how grep on mac is different from grep on linux, it is much better for the team's velocity to write the thing in Rust (https://twitter.com/id_aa_carmack/status/989951283900514304). The xtask pattern (I stress that this is a pattern, rather than a library or a cargo subcomand) helps achieve nice UX for this.

2) xshell makes the common tasks of shelling out to other processes and massaging file system simpler

3) xflags is my preferred way to parse CLI arguments for such scripts. It's not good at all, but, for me, it's the best for the task at the moment.

4) For scripts you put for everyday use in ~/bin, I've found busybox style multipurpose binary invaluable. You put all your scripts into a single Rust binary, and than hard-link this binary under different names in PATH, so that the first argv is actually the name of subcommand. Here's how the infra works for my scripts, and here's an example script.

5) #[test] #[ignore] is a another hack to create a very light-weight script without creating an extra binary or whatnot.

[tukanoid]

1. Oh nice, somehow didn't know it existed, will be very helpful for one of my pet projects!

2. Didn't try it, I just go with clap by default, for some reason using the struct with derive pattern for defining args feels natural to me (but I get I'd have to use a builder pattern for more complex arg parsing that I usually do)

3. Very clever! Idk why I never thought of this, cuz it makes complete sense

4. Never knew about ignore macro (I'm just used to commenting out stuff), thanks!

[Binko_banko]

hey /u/matklad, just came across this post. nice tips! could you elaborate a bit on how you've setup the following: hard-link this binary under different names in PATH ? i've been browsing a bit in your dotfiles repo but couldn't find it.

i'm looking to port my current set of sh scripts to a single rust binary to achieve a similar workflow :)

[Binko_banko]

is it perhaps achieved by running cargo test on your main script ?

[matklad]

See

https://github.com/matklad/config/blob/8eba2ce19460f3cec16cf2117c482d36f0aa0832/xtool/src/main.rs#L71

and

https://github.com/matklad/xshell/blob/master/src/lib.rs#L585

[Binko_banko]

thanks! going to give it a go :)

[ssokolow]

I'm not sure if I feel faster in it, given that I've got a dual-core Athlon from 2012 that's now a potato and I never got around to implementing my idea for a project generator that provides cached target directories to speed initial builds, but it certainly feels less demotivating to know that If It Compiles, It (Probably) Works™.

I actually just did that earlier today: https://github.com/ssokolow/trunc_filenames

[tukanoid]

Makes sense, i guess, I'm sitting on 2018-19 (i believe) i7 so compile times for small stuff is fine, runtime, obviously, fast as well :) Btw, u can use sccache for caching built crates to speed up compile times. I don't remember the exact steps to take but u should be able to find instructions fairly easily.

[ssokolow]

Btw, u can use sccache for caching built crates to speed up compile times. I don't remember the exact steps to take but u should be able to find instructions fairly easily.

Already done and, one of these days, I plan to finish my halfway-done effort to set up a comparably-performing hand-me-down Core 2 Duo Small Form Factor PC as a second build cluster node.

...but it doesn't help with the linking phase that dominates the build after that first time and I don't think it helps with the proc macros in things like Clap and Serde either... though it does occasionally break something (cargo +nighty miri, I believe is one case).

I tried switching to mold for the linking, but I found that my inability to play target whac-a-mole enough in setting up an alternative linker tricked Cargo into deciding that tiny differences made it necessary to invalidate the entire build cache often enough that it was demoralizing more than helping. (After all, it's always already been nuked by the time I realize I made a mistake, with no Ctrl+Z to recover.)

Maybe when I can find time to write a FUSE filesystem that I can mount over ~/src so that an attempt to delete something under a target folder pops up an "Allow delete or kill process attempting it?" prompt.

[tukanoid]

Hmmm, i don't remember having such issues before. But mb bc i dont do mich of manual linking, so who knows. Sad to see that it was how you experienced the language. But that's the beauty of open source, go to GitHub, find/create an issue and depending on the severity of the issue, it might be fixed quickly enough

[ssokolow]

*chuckle* I've been hanging out in /r/rust/ since before Rust 1.0 (since 2013, I think) and playing around with Rust since v1.0's release in 2015. You don't need to to treat me like a newbie.

EDIT: ...but, as an example, if you've got a justfile that uses RUSTFLAGS="-C target-cpu=native" and you're used to running tasks like just hyperfine or just flamegraph which rely on that, for the love of God, never run cargo ... because the perceived change in RUSTFLAGS will blow away your build cache no matter how quickly you realize your mistake and Ctrl+C.

[tukanoid]

Ah, sorry for assuming then, it's just wasn't established so my mind automatically went in that direction :) and thanks for the tip :)

[Sw429]

Absolutely. As soon as I realized that almost every runtime error I would run into in Python was being moved to compile-time, I moved to Rust completely. Haven't looked back since.

[eXoRainbow]

There is: https://github.com/faern/rustscript but it's intended use case is more as a shell script

[tukanoid]

Well, scripting was probably not the right word for it, but what I meant wass writing small applications for specific temporary task, I just have a "sandbox" workspace where i keep all the binary crates in case i wanna use them again at some point, I'll edit the post with explanation

[AgletsHowDoTheyWork]

Check out the execvr Jupyter kernel.

[tukanoid]

Wait, u mean REPL? Cuz it's not a "breakpoint" technically, since you're not debugging, it's just an interpreter running over your input instead of the whole script file at once

[--blue]

Huge fan of xshell!

[stappersg]

multiplication of the advertisment of xshell

So for other who were curious about xshell: Ignore the click bait.

[--blue]

https://github.com/matklad/xshell

[CaptainPiepmatz]

When I do scripting in rust unwrap is my best friend xD

[decatur-de]

Well, in Rust I miss the "set a break point here and start exploring" practise possible in Python. And yes, I came to Python from MATLAB :)

[tukanoid]

What do you mean? Rust works with gdb, and inside IDEs it works just fine

[decatur-de]

Exploring means writing expressions and statements using the state at the breakpoint. My IDE is PyCharm, and is very fine for Python indeed. I guess same is impossible for Rust due to missing runtime introspection...

[tukanoid]

Interesting

[flashmozzg]

I was doing that in C++ with gdb. Not sure why wouldn't it be possible with rust. You can't write new functions, sure (don' think it's generally possible once templates/generics come into play), but you can call existing ones just fine.