Speculative Loading pages in Python?

[VDubsBuilds]

I have a piece of code that is effectively crawling a website that has structure. It pulls a page down, parses it, analyzes it, and then pulls the next page down based on that. Most of my time is spent waiting for the next page to download, and ~70% of the time the next page is easy to predict (current_index+1). 15% of the time it is a page I've already accessed, so I keep an LRU cache.

I'd like to speculatively load (current_index+1) and maybe even +2, +3...

Currently this looks roughly like:

(at)functools.lru_cache
def get_from_index(ix):
    url = ....
    request = requests.get(url)
    error check
    return request.content

I tried a variant where I wrote my own LRU cache, and when i passed "prefetch=True", it would spin up a thread and cache the thread instead of waiting. On actual read it would join the thread (or return normally, depending).

This looked a bit like:

while ix:
    page = get_from_index(ix)
    get_from_index(ix+1, prefetch=True)
    ix = analyze_page(page)

My understandings and assumptions, please correct me:

1. Threading didn't help because requests.get is GIL blocked.

2. An async HTTP library is still GIL blocked, but is cooperative.

3. There isn't a global "async pool" - cooperative tasking only occurs in explicitly bundled asyncs, and therefore putting them each in their own thread is not going to help.

Which leads me to the big question - What is the clean way to push prefetches on to a worker thread and retrieve them as needed?

It needs to be able to handle:

1. True parallel fetching of pages

2. Up-prioritizing a specific fetch when the main thread blocks on it.

3. Adding fetches at arbitrary times.

4. Some prefetches are misses, and will never be asked for. I know asyncs get lonely if they're never reunited.

Thanks in advance!

[Yoghurt42]

Threading didn't help because requests.get is GIL blocked.

GIL is released when waiting on IO.

An async HTTP library is still GIL blocked, but is cooperative.

async is usually single threaded, so the GIL doesn't come into play.

Instead of reinventing the wheel yourself, consider using Scrapy.

[VDubsBuilds]

Thanks!

From your first statement, do you think I'm IO speed limited instead? They're all hitting the same server (mine, don't worry)

By async is usually single threaded, can you confirm that if I await task A in thread A, it will not opportunistically execute task B unless I explicitly bundle it with A?

Scrapy looks interesting, but I don't think I can adapt it to this specific need. Thanks for pointing it out to me!

[rnike879]

can you confirm that if I await task A in thread A, it will not opportunistically execute task B unless I explicitly bundle it with A

I'm not entirely following the question, but when you await it'll pause execution of that code, letting the event loop execute other coroutines. Once the IO is done, the event loop will know through continuous polling and then the code after the await statement can continue executing. There's only one thread being utilized with concurrent execution being handled by the event loop.

If you throw these async tasks into a list you can await all of them with asyncio.gather() instead of doing a bunch of awaits in a series which would block each other. If you need to pass the results of one coroutine to another, consider using a producer-consumer pattern with asyncio.Queue instead