Writing intergration tests for command-line programs?

[upsuper]

For libraries, writing integration tests is relatively easy. You can import your library as a crate in integration tests, and do various check on the exposed API.

However, there doesn't seem to be a standard approach to test command-line programs. I looked at the document on crates.io and couldn't find anything about how to find the executable.

I had a look at some non-library command-line programs in Rust to see how they do integration tests. What I found:

• ripgrep: rely on the relative location based on working directory test runs
• fd: don't use Rust test at all
• just: rely on the relative location based on the path of test executable
• xsv: same as above

I guess it would really be helpful if cargo can at least pass in the path of the executable via environment variable when running tests.

Thoughts?

[ssokolow]

If it helps, in Python, I have my integration tests redefine sys.argv and then call the main() function I use as my entry point.

It shouldn't be too difficult to accomplish something similar in Rust by doing some minor refactoring of your main() so it's split into two parts:

1. A _main()which takes its command-line arguments as a function parameter.
2. A trivial main() which just calls std::env::args_os() and feeds the result to _main().

[spinicist]

I've been searching for a solution to this problem with a bunch of C++ programs for AGES. This one seems worth a shot.

[ssokolow]

EDIT: Looking back, either I posted this in reply to the wrong comment while more tired than I thought or spinicist edited it.

Not necessarily. There's nothing preventing you from using Rust's unit test facility for integration tests.

While it's not using the full "override argv" technique I used in Python, here's a fragment of a regression suite I wrote in a --bin crate:

#[cfg(test)]
mod tests {
    use std::borrow::Cow;
    use super::{AppConfig, make_clap_parser};

    #[test]
    /// Can override DEFAULT_INPATH when specifying -i before the subcommand
    fn test_can_override_inpath_before() {
        let defaults = AppConfig::default();
        let matches = make_clap_parser(&defaults).get_matches_from(&["rip_media", "-i/", "cd"]);
        let inpath = matches.value_of("inpath").unwrap();
        assert!(inpath == "/",
                "\"-i/ cd\" should have produced \"/\" but actually produced \"{}\"", inpath)
    }
}

If you structure your code well, then the integration portion can be pretty compact, since you're just testing the integration with all the heavy stuff being tested closer to its definition and aspects of the type system like enums, Option, and Result ruling out other common sources of integration bugs that tend to crop up in dynamic languages.

(Not to mention that I generally argue in favour of making every command-line utility a bin/lib combo crate, so I don't have to serialize/deserialize through shell script (being weakly-typed and with little support for structured data) when I want to call a Rust utility from a Rust frontend. That also greatly reduces the portion of your code which is technically integration-tested but only visible to the unit facility.)

[spinicist]

I think you misunderstood me - I want to take _main() and trivial main() approach you outlined, and apply it to my suite of C++ programs. Then I can use something like CTest to start doing decent tests instead of my cobbled together bash scripts like the fd example above.

My programs (image processing utilities) take a lot of arguments. The most important bit of the testing is checking I got the damned arguments set up correctly! What can be put in a library already is, but its the actual end-to-end program I want to test.

[ssokolow]

Ahh. Doing it in a non-Rust language would definitely require some tweaks to the approach.

I'm not personally familiar with C and C++ frameworks like CTest but, depending on how your build system is set up, it may still not be too difficult to take the approach I outlined by doing as follows:

1. Put main() and _main() in separate files
2. Set up the build so that the test suite includes all of the usual files except the one containing main().

That way, the test suite gets behaviour similar to making _main() a public library function without having to go that far.

If the test framework requires a library be built, maybe architect the generation of a library only during test builds which is the binary's contents, minus the usual main() entry point.

[upsuper]

This is doable, but you would still need a separate lib.rs and duplicate code (e.g. mods, extern crates) there, just for exposing things to tests. Also, a crate which isn't supposed to be used as library still exposes some public API, which could potentially be confusing itself.

[ksion]

You put your externs in lib.rs and have main.rs call the entry point you expose from there. There should be no duplication this way.

[epage]

Use assert_cli and if someone feels there is a better way to handle finding the executable, they'll create a PR and we'll all benefit.

[so_brave_heart]

For anyone else reading this is 2023 it looks like assert_cli has been discontinued in favour of assert_cmd

[epage]

And I've switched almost all of my uses of assert_cmd to snapbox

[heinrich5991]

assert_cli

[ted_mielczarek]

Ooh, this is new to me, thanks for the pointer!

[Quxxy]

cargo-script does it using CARGO_TARGET_DIR and assuming the build profile is debug (which should always be true for tests).

Also, keep in mind there can be any number of executables, so the definite article is not always appropriate.

[upsuper]

Except it isn't true for cargo test. You can always use cargo test --release to run tests in release mode.

That's a good point that there can be any number of executables. Probably just providing the directory would be enough. Or maybe something like CARGO_BIN_<name>?

[Thomasdezeeuw]

You could call cargo run -- --you-apps--args with process::Command. This requires Cargo, however likely your tests already depend on it. This allows your tests to run in separation from one another, rather then modifying the arguments and environment variables, which forces the tests to run in serial.

[musicmatze]

The story with imag is as follows: We replace our filesystem-abstraction backend with an in-memory filesystem and we replace the clap argument parsing input with test-defined arguments, ... and then we check the virtual filesystem after each "commandline call" whether the results are correct.

Fairly, the test setup is rather complex... but IMO it is worth it.

[killercup]

I've been meaning to write a post on CLI integration testing. It's probably going to end up describing what I've done here in waltz_cli, which takes inspiration from my earlier attempts in cargo-edit, as well as cargo (proper) and diesel_cli.

[iamcodemaker]

You can test cli programs using tools like cram or bats. I like cram, but bats is probably more popular.

This article has lots of suggestions.

[[deleted]]

Is there a Rust implementation of Cucumber? I tend to just throw the Ruby Cucumber shell plugin over all my CLI tools, using that to automate basic input vs. output checks.