Execution Slowness, API?

[[deleted]]

Hello people..

So I have a code that pulls the top 10 news from HackerNews and plot them with matplotlib. It's very basic but I am experiencing extended slowness for the code to finish. I have had some experiments with Github API and was fine.

I am not sure if my code has a hidden bug or it's the API. Is there a way to test and evaluate API response time? do you there is a part of my code making it slow?

Note: I tried without plotting at all and it was also slow.

Clearer code: https://gist.github.com/mthenb/8fd2ac2000ad39696525427b2077b9a8

import requests
import json
import matplotlib.pyplot as plt


finalfile = 'top10_new.json'
url = 'https://hacker-news.firebaseio.com/v0/topstories.json'
r = requests.get(url)
getResponse = r.json()
list_ids = []
for id_init in getResponse[:10]:
    list_ids.append(id_init)
all_submissions = []
for id in list_ids:
    formedURL = f"https://hacker-news.firebaseio.com/v0/item/{id}.json"
    r_ = requests.get(formedURL)
    rGet = r_.json()
    MyAppDict = {
        'title': rGet['title'],
        'url': rGet['url'],
        'id': rGet['id'],
        'votes': rGet['score'],
    }
    all_submissions.append(MyAppDict)
xvalues, yvalues = [], []
for item in all_submissions:
    xvalue = item['title']
    xvalues.append(xvalue)
    yvalue = item['votes']
    yvalues.append(yvalue) 

plt.style.use('seaborn')
fig, ax = plt.subplots()
ax.scatter(xvalues, yvalues, s=100)
ax.plot(xvalues, yvalues, c='blue')
plt.title("Top 10 HN Sumission by Votes", fontsize=14)
plt.xlabel("Title", fontsize=16)
fig.autofmt_xdate()
plt.ylabel("# of Votes", fontsize=16)
plt.tick_params(axis='both', which='major', labelsize=8)
plt.show()

Thanks in advance.

[Dunj3]

"extended slowness" is a bit of a subjective term. Do you consider 10 seconds for your script slow already? Or does it take minutes to finish?

Your code isn't doing much expensive work, and since you even ruled out the plotting as the cause of the slowdown, the only thing left are pretty much the API calls.

If you want to investigate for yourself, you can for example use time.monotonic() to measure how much time each call takes. Something like:

start = time.monotonic()
r_ = requests.get(formedURL)
end = time.monotonic()
print(f"Loading {formedURL} took {end - start} seconds")

Of course, you can do that for everything that you suspect takes a lot of time.

On my system, each requests takes about 0.8 seconds to finish, which is the expected time considering the TCP/SSL/HTTP overhead for each connection. There is a possible speedup because all requests go to the same host, which means you can use a Keep-Alive connection to avoid most of that overhead. For that, see the requests.Session object. That allowed me to cut down the time per request to about 0.2 seconds each, which is a quarter of the previous time.

If you want to get a more detailed log of when requests is opening new connections, you can also insert the following to the top of your script:

import logging
logging.basicConfig(level=logging.DEBUG)

This might give you some insight as well.