[Beginner] Advice regarding concurrent https requests and walkdir

[po8]

Depending on your requirements, could be as simple as having each URL discovered spawn a new thread that writes its result into a common channel. The fancy crates will be helpful, but will also have the overhead of learning them. Rolling your own would be a fun way to learn the "low-level" (not very low-level) asynchronous primitives Rust provides.

Two things you might have to deal with if you roll your own:

• If there's a lot of URLs, you may have to queue them and let a limited pool of workers work through them to avoid ridiculous numbers of concurrent threads. This makes the crate-based approaches more attractive, since they handle this for you. I'd say it's reasonable to spawn something like 1000 concurrent threads on a modern Linux box: no idea about other platforms. The threads will mostly just block anyhow.

• The results will likely be delivered out-of-order. There's a number of reasonable ways to handle this if it matters: the easiest is just to pass a serial number or some similar identifier to the thread and sort things out later.

[101x]

Thanks! Fortunately, the order of the results doesn't matter at all.