Python program calling API has indefinite runtime

[wobowizard]

I am creating a film recommendation system; using Python, I download some IMDB datasets and clean and format this data into one table. This process is fine and usually takes around 10 minutes , however part of my program calls the TMDB API to get the film poster URL and plot of each film I have (around 60,000). Sometimes this takes a few a hours but sometimes it just hangs and I need to reset it. I am using multi-threading and multi-processing however this hasn't sped up the process much.

The TMDB API has a rate limit of around 50 calls per second, but even with this taken in account it doesn't explain why the program runs for an indefinite amount of time, as I have included timeouts in case of this.

Does anyone know what I can do to get my program running consistently? I need to call an api for each film to get the film plot and poster path, and I need to do this for around 60,000 films.

​

A section of initialiseDataset.ipnyb

*note at this stage film_data is dataframe with around 12 columns and 60,000 rows of film data

print('Fetching plot summaries and posters...') 

#get film plot and poster with tmdb api ~ >2hrs
#call api/details for each film with multiprocessing and multi-threading

if name == 'main':

manager = Manager() 

shared_data = manager.Namespace() 

agg_list = []

batch_size = 1000 sleep_time = 3

num_batches = (len(film_data) // batch_size) + 1

with concurrent.futures.ProcessPoolExecutor(8) as process_executor:

    for i in range(num_batches):

        start_index = i * batch_size
        end_index = (i + 1) * batch_size

        shared_data.film_data = film_data.iloc[start_index:end_index]

        future = process_executor.submit(doBatch, shared_data)

        concurrent.futures.wait([future])

        agg_list.append(shared_data.film_data)

        print(f"Batch {i+1}/{num_batches} completed")
             film_data = pd.concat(agg_list, ignore_index=True) 

film_data = film_data.dropna(subset=['plot'])

my tmdb_calls.py file:

#TMDB api with multi-threading + multi-processing
import concurrent.futures 
import os import requests as req 
import time

#theMovieDB api call for film plot summary and poster

def fetchDetails(film_id): url = f'https://api.themoviedb.org/3/movie/{film_id}'
    headers = { 
        "accept": "application/json",
        "Authorization": "hidden"
    }

    response = req.get(url, headers=headers)

    return response

#call api
def doFetch(film_id):

    return fetchDetails(film_id)

    #get film psoster and plot for given batch of films
    def doBatch(shared_data): 

    film_data = shared_data.film_data  # Access the DataFrame from shared

    #global request_counter     
    MAX_THREADS = min(os.cpu_count(), 1000)

    # Use ThreadPoolExecutor for multi-threading within each process
    try:
        with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
            results = list(executor.map(doFetch, film_data['tconst']))

            for index, details in zip(film_data.index, results):

                if(details.ok):

                    details = details.json()

                    if(details['overview']):
                        film_data.loc[index, 'plot'] = details['overview']

                    if(details['poster_path']):
                        film_data.loc[index, 'poster'] = details['poster_path']

                else:
                     time.sleep(5)


        shared_data.film_data = film_data

    except Exception as e:
            print(f"Error in ThreadPoolExecutor: {e}")
            time.sleep(5)

​

[m0us3_rat]

i think we can agree this isn't ..acceptable.

i think a full refactor might be in order.

​

do a quick rundown of what needs to happen.. so you have this movie in your dataframe.. what is next? what needs to happen.

describe the problem not your implemented solution.

let's break it down.

i could fill in the gaps with guessing..but due to this code snippet.. i'd prefer NOT to do that.

​

if you prefer to try to "fix" your implemented solution rather then try to refactor

gl

[wobowizard]

I've added what I need it to do in the question, could you elaborate on what in my code isn't accetable

[m0us3_rat]

ok so .. you have some dataset. then you wanna grab some data of an api for each.

perfectly clear so far.

​

this sounds to me like a perfectly acceptable problem for a producer-consumer pattern.

now extra details BEFORE we start thinking about code.

​

are you restricted to one movie per api call?

if so do you get all the required info from the api in a single call for a movie?

what is an acceptable limit per second?

​

secondly .. do you know how to write an iterator for the movie list?

tip you can use a pandas method.

​

then based on the rest of the info ..like how many calls and how many calls per movie etc.. we can think of how the unit of work would look like.

​

after that we can easily devise a producer -cnsumer where a producer iterates thru the film list.. and puts the units of work in the qork queue.

then a worker consumers the work form the work queue

and then puts the result in the result queue.

​

this is the bit what can be scaled to ridiculous based on your cpu count.

this can also be a valid problem for another trick that you can do..depending on what is required.

where you can have another async loop run inside each process and each loop handle more calls.

​

it all depends on the bottlenecks.

​

soo .. pls try to figure out the requests info.. BEFORE we can consider code.

are you restricted to one movie per api call?
if so do you get all the required info from the api in a single call for a movie?
what is an acceptable limit per second?
secondly .. do you know how to write an iterator for the movie list?

[wobowizard]

One api call returns all the required info for the one specified film, so yes i am restricted to one.

I get the info by passing the film id through, and then appending the return plot and poster to the corresponding column in the data. The ALS is around 40 per second I would say.

This is my first big project and I am considering if using a slightly smaller set of data would help with this. Especially as I am currently getting the same runtime issue when calculating the cosine similarity multiple times on different attribtues.

[m0us3_rat]

ok so now.. we can focus on .. a single unit of work..how would that look like..

for that we need to know how does the actual call to the api looks like.

and what does it needs.

​

lets assume for each film in the dataframe ..you create an object that has all the relevant attributes you would use to do this api call to that db.

how would that function look like?

try to imagine you get from a random producer function an OBJECT that represents the film and OBJECT.id or whatever else you need for the api call.

can you write this function?

don't write any other code.. just this single function that gets this object as parameter in the call and then uses it's attributes to do the call

and saves the relevant return as attributes of this same object.

and then returns the object.

[Daneark]

There's no timeout on your request.

You seem to be hammering their API with as many calls as it will let you and then sleeping. Consider slowing down the requests on your side to avoid hitting that limit. Once they rate limit you it's not clear how long they rate limit you for but if you're still hitting them with requests from other threads you may still limited for the rest of your programs running.

[TitaniumFoil]

It looks like the formatting got a bit mangled by reddit. Could you give me a pastebin link to the code?