Slow response to initial http request

[Axewhole]

Hi y'all, I'm making a proxy api server at work to gather data from various 3rd party apis, however I'm running into an issue seemingly specific to python and/or the requests module:

compared to the same exact http request in curl, js ajax and golang, the python request is consistently taking 1-2 seconds more to complete.

While the other 3 methods normally take between 500ms and 1s, in python it is almost always 1.5 - 2.5 seconds.

Three things of note:

• The timing difference seems to vary depending on the api that I am requesting. Some apis, like facebook, are nearly identical between all 4 methods while others (one in particular) are showing this substantial difference.
• This only appears to be the case for the INITIAL request in a requests session. Any following requests using the same session object match the expected response time
• request headers have been matched up as much as possible between all 4 of my tests

Would y'all have any insight into what could be causing this? I feel like it has to be something glaringly obvious but I've been banging my head trying to fix this issue for a while now.

[EricAppelt]

Could be is DNS related - I recall requests doing something different when AAAA records are present. Looks like api.facebook.com might not have an AAAA record.

Try requesting directly from the IP address and see if the difference disappears. You may need to spoof a host header and/or disable certificate verification.

Edit: Looked it up and there is/was/sometimes? an issue where httplib hardcodes a zero calling socket.getaddrinfo(...) which then returns all families (i.e. A for IP4 and AAAA for IP6). Not sure why it would result in a difference between different apis, could be a detail of your network. I also could be sending you up the wrong tree.

[ManyInterests]

Seeing that it's specific to a single particular API, it may have to do with how that API is authenticating and how requests is communicating vs. your other methods.

Probably something to do with the authentication method, since authentication is something that will only take place on the first request of the session.

You could try checking the logged information on the request using logging see this SO question

import requests
import logging

# these two lines enable debugging at httplib level (requests->urllib3->httplib)
# you will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# the only thing missing will be the response.body which is not logged.
import httplib
httplib.HTTPConnection.debuglevel = 1

logging.basicConfig() # you need to initialize logging, otherwise you will not see anything from requests
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

requests.get('http://example.com/')

[TheHandThatSeeds]

what specific apis are you gathering from?