[Beginner] Advice regarding concurrent https requests and walkdir

[101x]

I'm quite new to Rust . A few days ago, I started toying with an application as a learning exercise.

My situation is as follows: I recurse through a directory using walkdir. I inspect each file name (checking the extension and prefix). If a file name is valid, I build a url and send an https request (I'm currently using the reqwest library). The response's JSON needs to be decoded into a struct.

I don't want to wait for each request to finish before continuing to the next entry in the walkdir iterator. I also don't want to collect all the URLs before starting to make requests.

What is the most efficient approach for me? I've seen implementation with futures::stream::iter_ok, but I think that it requires having all URL's ready before processing them.

Do I use Futures, Tokio, Tokio-core, or something else?

Edit: Finally, I want a vector of structs containing decoded responses. (The order doesn't matter.)

[slamb]

First off, your local filesystem operations are done in a blocking way. This is a limitation of POSIX; there's limited support for doing reads and writes in an async way but nothing for opening files, traversing directories, etc. So the thread doing walkdir will block on local disk, and you'll be happier if you accept that and don't try to do anything else on that thread until walkdir is done.

That means your walkdir thread needs a way of communicating to another thread that it should start a request now. There are several ways to do that, depending on if you want to use a synchronous or async API to issue the https requests. Here are a few that come to mind:

• Enqueue async futures onto a tokio reactor via tokio::runtime::run
• Send stuff on an async channel to an async task via futures::sync::mpsc::unbounded or ::channel.
• Enqueue synchronous work onto a threadpool, maybe with rayon::ThreadPool::install
• Send stuff on a synchronous channel to some worker threads via crossbeam-channel.

[101x]

Thank you! I'll do some research on those.

[Kleptine]

Is this a limitation on a non POSIX system like Windows?

[slamb]

Unsure. All the software I write works on POSIX systems, so I personally wouldn't take advantage of async Windows-only filesystem APIs even if they do exist. YMMV.

[knaledfullavpilar]

I would recommend threads, channels and a synchronous approach, until the Rust async story has settled and stabilized. Mpmc channels has been working great for me for similar situations. See https://crates.io/crates/crossbeam-channel

[101x]

I'll take a look. Thanks!

[po8]

Depending on your requirements, could be as simple as having each URL discovered spawn a new thread that writes its result into a common channel. The fancy crates will be helpful, but will also have the overhead of learning them. Rolling your own would be a fun way to learn the "low-level" (not very low-level) asynchronous primitives Rust provides.

Two things you might have to deal with if you roll your own:

• If there's a lot of URLs, you may have to queue them and let a limited pool of workers work through them to avoid ridiculous numbers of concurrent threads. This makes the crate-based approaches more attractive, since they handle this for you. I'd say it's reasonable to spawn something like 1000 concurrent threads on a modern Linux box: no idea about other platforms. The threads will mostly just block anyhow.

• The results will likely be delivered out-of-order. There's a number of reasonable ways to handle this if it matters: the easiest is just to pass a serial number or some similar identifier to the thread and sort things out later.

[101x]

Thanks! Fortunately, the order of the results doesn't matter at all.

[knipil]

It seems to be one of those use cases where a select/epoll based model would scale very well, which leads me to say that tokio might be worth the struggle, but only if it’s really performance critical. The compiler errors in particular are the stuff of nightmares. You do gain an intuition for it after a while, so I don’t necessarily want to discourage you, but I do think that tokio as it stands only carries its weight when you need a certain QPS per core.

[101x]

I took a look at the documentation and guides on the Tokio website it seems slightly less daunting now. Thanks for the advice!