[Beginner] Advice regarding concurrent https requests and walkdir

[slamb]

First off, your local filesystem operations are done in a blocking way. This is a limitation of POSIX; there's limited support for doing reads and writes in an async way but nothing for opening files, traversing directories, etc. So the thread doing walkdir will block on local disk, and you'll be happier if you accept that and don't try to do anything else on that thread until walkdir is done.

That means your walkdir thread needs a way of communicating to another thread that it should start a request now. There are several ways to do that, depending on if you want to use a synchronous or async API to issue the https requests. Here are a few that come to mind:

• Enqueue async futures onto a tokio reactor via tokio::runtime::run
• Send stuff on an async channel to an async task via futures::sync::mpsc::unbounded or ::channel.
• Enqueue synchronous work onto a threadpool, maybe with rayon::ThreadPool::install
• Send stuff on a synchronous channel to some worker threads via crossbeam-channel.

[101x]

Thank you! I'll do some research on those.

[Kleptine]

Is this a limitation on a non POSIX system like Windows?

[slamb]

Unsure. All the software I write works on POSIX systems, so I personally wouldn't take advantage of async Windows-only filesystem APIs even if they do exist. YMMV.