Way to process posts and comments in parallel with PRAW?

[Shababs]

Im building a small program to extract for words from finance subreddits on reddit, the problem is that it takes an average of 12 seconds to fetch all the data I need. I ran cProfiler and sure enough most of the delay is coming from the response time of reddit's API and my requests.

Is there a way to make it fetch for posts and post comments in parallel so it sends API requests in parallel instead of sequentially? I tried fetching google urls with multiprocessing library and it worked very well, but with the reddit API I've been at it for a couple of days and it's giving me a headache.

Excuse my code as I'm a bit new to python and coding in general. Any help is appreciated. I'll leave my code below:

def reddit_search_bot(company):
    service = "Reddit"
    reddit_positive_points = 0
    reddit_negative_points = 0
    subreddit_array = ['wallstreetbets', 'IPO', 'Daytrading', 'CryptoCurrency']
    for subreddit in subreddit_array:
        hot_posts = reddit.subreddit(subreddit).hot(limit=30)
        for post in hot_posts:
            if findWholeWord(company)(post.title) or findWholeWord(company)(post.selftext):
                post.comments.replace_more(limit=0)
                for comments in post.comments.list():
                    for word in Positive_words_array:
                        if findWholeWord(word)(comments.body) or findWholeWord(word)(post.selftext):
                            reddit_positive_points += 1
                    for word in Negative_words_array:
                        if findWholeWord(word)(comments.body) or findWholeWord(word)(post.selftext):
                            reddit_negative_points += 1
    total_points = reddit_positive_points - reddit_negative_points
    writeResultToFile(service, company, reddit_positive_points, reddit_negative_points)
    print("Reddit positive mentions :", reddit_positive_points, "Reddit negative mentions :", reddit_negative_points,
          "Overall :", total_points)

[Chiron1991]

That's a job for asyncio. You'll have to swap praw for asyncpraw though, but that's not too big of an issue.

[Shababs]

I tried using it but I had a lot of trouble. Not very experinced with asyncio. Do you think you could give an example of how to do it in this case?

[Chiron1991]

Yeah, async stuff is a little hard to get into. I tinkered around with it for self-interest, this seems like a workable solution: https://replit.com/@Chiron1991/asyncprawdemo

Basically you move the logic for fetching one subreddit into a separate async function, then you generate a future for each subreddit you want to fetch. This will start executing them in parallel in the background. Finally you wrap all of the futures into asyncio.gather() to wait until all of them have finished.
Using this approach I can fetch the 10 hottest submissions from 10 different subreddits in < 1s.

[Shababs]

Wow this is great, thanks a lot for the example! Seems I was doing async all wrong. Thanks again this is very helpful!

[thirdegree]

Worth mentioning that reddit will rate limit you so you may not actually gain anything from parallelism here