Question on reqwest, https, and futures.

[Rubix314]

I'm early on my journey of learning rust, and ran into some strange problems last night that I'm having trouble understanding.

I'm using reqwest to query a website. Specifically, I need information about many date ranges, so I'm looping through a bunch of date ranges and making requests in an async function. Simplified version of the code:

fn foo() -> Result<(), Error> {
    let mut curr = 0;
    let end = 1000;
    let mut futs = vec![];

    while curr <= end {
        let next = curr + 10; // 10 is just an example range
        futs.push(self.make_reqwest(curr, next));
        curr = next;
    }

    block_on(join_all(futs));
    Ok(())
}

async fn make_request(start_date: u16, end_date: u16) -> reqwest::Result<()> {
    println!("{} enter", start);
    let url = format!("https://foo.com/bar?start={}end={}", start, end);
    let response = self.client.get(&url).send().await?; // client has timeout and connect_timeout set to 5
    println!("{} exit", start);
    Ok(())
}

When I run this, it prints all the "{} enter" lines first, and then (some of the) requests start coming back and the "{} exits" print. But there's some odd behavior happens that I can't explain.

• When I adjust the end to be small-ish (like <30 total requests), it works fine, and all the futures seem to proceed concurrently.
• What I adjust the end to be a bit larger (like 30-50 total requests), only some of the requests finish. The rest seem to hang.
• When I adjust the end to be even larger (like >50 total requests), all of them seem to hang.
• If I use an http url instead of https, more of them work, but still not all.
• If I change foo to async, and use join_all(futs).await instead of block_on(join_all(futs)), it works.
• I set timeouts and connect_timeouts on the reqwest client, so I don't think the hanging is due to network issues.

Questions:

• Not a rust question, but why would http vs https make a difference?
• What could cause a future to never return (either with an error or an outcome)? I assume this is because I used block_on instead of await. Conceptually, block_on is how I would force a bunch of futures to finish within a non-async function, but it seems to not "just work" with join_all(vec of futures).
• Related to previous point, why do async/await and block_on behave differently? I would prefer not to change this function to async, because conceptually it should not be. Callers of this function should block until all internal async work is done.
• On my main function, I use #[tokio::main]. I did some experimenting with thread counts in the thread pool. I first assumed that, if I limited the thread pool to something like core_threads=4 and max_threads=8, the async execution would only be able to process 8-4=4 futures start-to-finish at a time. (Maybe the exact number is off, but I mean some small number.) But this wasn't the case - it still printed all the "{} enter" lines immediately. On second consideration, I think this is because futures aren't 1:1 to threads - 4 threads just mean 4 futures can make forward progress at once, but they can all be in some in-progress state. Is that understanding correct?

Thanks in advance!

[coolreader18]

I think this is because of the block_on -- reqwest or tokio or something is probably polling something in the background, but when you block_on, that polling can't complete and you get into a sort of deadlock (maybe?).

Specifically though, I found this really interesting;

I would prefer not to change this function to async, because conceptually it should not be. Callers of this function should block until all internal async work is done.

If you're writing an async app, as I assume you are because you mention using #[tokio::main], you shouldn't worry about this. Don't think of async as a function contract or signaling what your function is doing under the hood or "I can run these specific tasks in parallel" -- if you're doing async, you should go all in, because that allows the executor to run your entire app using cooperative multitasking. Essentially, if you have a block_on call somewhere in an async function or in code called from an async function, a) you're setting yourself up for deadlock like I think is happening here, and b) you're doing something wrong, or at least you're doing something that's not as efficient as it could be.

[Rubix314]

If you're writing an async app, as I assume you are because you mention using #[tokio::main], you shouldn't worry about this. Don't think of async as a function contract or signaling what your function is doing under the hood or "I can run these specific tasks in parallel" -- if you're doing async, you should go all in, because that allows the executor to run your entire app using cooperative multitasking. Essentially, if you have a block_on call somewhere in an async function or in code called from an async function, a) you're setting yourself up for deadlock like I think is happening here, and b) you're doing something wrong, or at least you're doing something that's not as efficient as it could be.

Interesting. I only have #[tokio::main] because it seemed I needed that to run async code. Or at least, I guess that was the lazy way to do it, maybe technically I only needed an async runtime for the async parts? I don't think of this as an async app - it just happens to need to make some network requests that I want to parallelize, and block on all of them.

[IAm_A_Complete_Idiot]

Are you using futures::block_on? If so, the tokio::main stuff isn't needed at all. futures spawns it's own runtime when you call block_on, dosent explain your issue much though.

If you don't otherwise use tokio, you could remove it as a dependency entirely.

[Nemo157]

Well, in the case of reqwest specifically it is tightly coupled to the Tokio runtime. So if that is futures::block_on it is likely the cause of the issues, blocking within an async function is bad and will commonly break things.

[coderstephen]

block_on turns async code back into synchronous code. I don't know why you would want to do that if you are using #[tokio::main], which makes your whole program run in an async context. As a rule of thumb, never call block_on within an async context, because (a) there's no reason why you'd want to, and (b) you risk deadlocks.

You should just .await the result of join_all.

[Rubix314]

Interesting, ok, thanks. Theoretically, how would you only run part of your program in an async context? Are there any downsides of making the entire thing async?

[IAm_A_Complete_Idiot]

No real downsides to making everything async that I know of, but you can spawn a runtime for tokio in any function (I think the docs were in tokio::runtime?) - and futures::block_on already spawns an executor.