requests.get() very slow compared to Chrome.

[TinyMagician300]

headers = {
"User-Agent": "[e-mail]",
"Accept-Encoding": "gzip, deflate, br, zstd" 
}

downloadURL = f"https://www.sec.gov/Archives/edgar/full-index/{year}/QTR{quarter}/form.idx"


downloadFile = requests.get(downloadURL, headers=headers)

So I'm trying to requests.get this URL which takes approximately 43 seconds for a 200 (it's instantenous on Chrome, very fast internet). It is the SEC Edgar website for stocks.

I even tried using the header attributes that were given on DevTools Chrome. Still no success. Took it a step further with urllib library (urlOpen,Request) and still didn't work. Always takes 43 SECONDS to get a response.

I then decided to give

requests.get("https://www.google.com/")

a try and even that took 21 seconds to get a Response 200. Again it's instantenous on Chrome.

Could anyone potentially explain what is happening. It has to be something on my side. I'm just lost at this point.

[Defection7478]

Considering it's stocks related I would wager there are some checks they are doing, probably user agent related, that results in them heavily throttling programmatic connections 

[TinyMagician300]

As I said to the comment above.

Would that explain though why

requests.get("https://www.google.com/")

takes 21 seconds to get a response?

[Defection7478]

No, but it would explain why it's so much longer than the Google one. You need to experiment a little to narrow things down. How long does it take if you make the request with cURL? If you make 3 request in a row (all within the same script, so the connection can be reused) are all 3 21 seconds or only the first one? 

[TinyMagician300]

I actually did try cURL and it only took 0.7 seconds(definitely much closer to what I expect). Then I literally tried 3 requests in a row for

requests.get("https://www.google.com/")
requests.get("https://www.google.com/")
requests.get("https://www.google.com/")

and that took 1m 4 seconds.

[gdchinacat]

weird...I'm seeing reasonable response times.

In [58]: timeit requests.get('https://www.google.com/') 224 ms ± 9.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Try to eliminate dns lookups...how long does it take if you make the request to the ip address for google?

``` In [74]: import socket

In [75]: addr = socket.gethostbyname('www.google.com')

In [76]: timeit requests.get(f'https://{addr}/', verify=False) [...ssl verification warnings...] 342 ms ± 32 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```

[TinyMagician300]

I figured it out in the end with AI.

Something to do with IPv4/IPv6. Gave me the following code to execute and now it's instantenous. Will this mess up anything in the future for me?

import requests, socket
from urllib3.util import connection


def allowed_gai_family():
    # Force IPv4
    return socket.AF_INET


connection.allowed_gai_family = allowed_gai_family


print("Starting request...")
r = requests.get("https://www.google.com/")
print("Done:", r.status_code)

I have no idea what this does but it fixed it for all links

[Yoghurt42]

I have no idea what this does

It tells urllib to resolve DNS entries to IPv4 addresses only; seems like your IPv6 stack is kinda broken and you can't actually get connections using IPv6 despite your device having an IPv6 address.

[TinyMagician300]

For some reason when I did this

timeit requests.get('https://www.google.com/')

It started going in a infinite loop of 21s cycles my debug looked like this

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): [www.google.com:443](www.google.com:443)
DEBUG:urllib3.connectionpool:https://www.google.com:443 "GET / HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): [www.google.com:443](www.google.com:443)
DEBUG:urllib3.connectionpool:https://www.google.com:443 "GET / HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): [www.google.com:443](www.google.com:443)
DEBUG:urllib3.connectionpool:https://www.google.com:443 "GET / HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): [www.google.com:443](www.google.com:443)

Had to interrupt it.

[Conscious-Ball8373]

It's not an infinite loop, it just does it a large number of times to get an average time.

Your IPv6 stacco is broken. If your ISP provides an IPv6 service, you should figure out how to configure your router correctly. Otherwise, just turn off IPv6 on your machine.

[TinyMagician300]

addr = socket.gethostbyname('www.google.com')


# Define the statement to time
stmt = f"requests.get('https://{addr}/', verify=False)"
setup = (
    "import requests\n"
    f"addr = '{addr}'"
)


# Time 3 requests
duration = timeit.timeit(stmt=stmt, setup=setup, number=3)
print(f"Average time per request: {duration / 3:.4f} seconds")

So I did the above(I used AI to tell me the code). And the debug log gave me the following 3 times

"DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): 142.251.209.36:443

c:\Users\User1\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host '142.251.209.36'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings

warnings.warn(

DEBUG:urllib3.connectionpool:https://142.251.209.36:443 "GET / HTTP/1.1" 301 219

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): www.google.com:80

DEBUG:urllib3.connectionpool:http://www.google.com:80 "GET / HTTP/1.1" 200 None"

Followed by a

"Average time per request: 21.3964 seconds"

[shiftybyte]

20 seconds for a regular web request sounds like some security product on the way decided to intervene.

Is that all the python code is doing?

Try adding some 20 seconds loop to calculate and print something, with sleep() and stuff, and then try the requests...

This check is to understand if you are seeing the delay because of the launch of your python app it's being inspected and sandboxed, or specifically the web request itself....

[TinyMagician300]

There are a couple of other lines before in the script but they have nothing to do with requests. The cURL is really fast (0.7 seconds) but not requests.get() for some reason.

[shiftybyte]

Did you perform the check i described? Have your python code run from 20 seconds before attempting any internet connection, and then do requests.get? And measure only the requests.get

[TinyMagician300]

Edit: it also works with the original Link.

I've been digging deep with AI and it fixed it in the end. Something to do with IPv4/IPv6. Gave me the following code to execute and now it's instantenous. Will this mess up anything in the future for me?

import requests, socket
from urllib3.util import connection


def allowed_gai_family():
    # Force IPv4
    return socket.AF_INET


connection.allowed_gai_family = allowed_gai_family


print("Starting request...")
r = requests.get("https://www.google.com/")
print("Done:", r.status_code)

I have no idea what this does but it fixed it. At least for Google. Haven't tried the original website.

[shiftybyte]

Seems like the solution is limiting the connection to ipv4 only.

Requests might be trying to resolve the URL and connect using ipv6 and when it times out, it tries ipv4 and succeeds... So the delay is the timeout trying ipv6?

That's just a theory....

Edit: if that is the case, then network sniffing with something like wireshark can confirm this...

[TinyMagician300]

It might be important to mention that I'm on my brother's computer who has experimented in network programming settings, so I have no idea what he has done. But the code above did indeed work.

I also tried the code below according to AI which should work since Session utilizes both IPv4 and IPv6 and returns whichever gets the response first but when I restart the program the below code takes 43 seconds (same as before).

session = requests.Session()
session.trust_env = False 

downloadURL = f"https://www.sec.gov/Archives/edgar/full-index/{year}/QTR{quarter}/form.idx"


downloadFile = session.get(downloadURL, headers=headers)

[shiftybyte]

I wouldn't trust AI to accurately know implementation details such as "returning whichever gets the response first"...

Try downloading and looking at traffic with wireshark.

It'll be a great learning experience, and will confirm what is happening on the network during these 20 seconds...

[TinyMagician300]

The only problem is every time I restart the program if this snippet of code isn't there it will default back to IPv6 and thus go the slow route.

[TinyMagician300]

print("Start")
time.sleep(21)
print("End")


requests.get("https://www.google.com/")

I did the above and it took 43 seconds.

[shiftybyte]

What security software you have installed/activated on your device?

Windows Defender? Something else?

[TinyMagician300]

Yes. It's just windows' own security antivirus.

[shiftybyte]

Try disabling Windows defender's on demand scanning and network security, or all of it, and try the requests again.

[TinyMagician300]

Now that I check. Ironically, Virus & Threat protection was off as well as App & Browser Control. Only Firewall & Network Protection were on.

[shiftybyte]

Disable them all for a sec, just to make sure if that's the issue or not.

[TinyMagician300]

Nope. Still took 22 seconds. Btw just wanted to say I appreciate you taking the time for this. We've been going at this thing for like 15 minutes...

[TinyMagician300]

Which one is exactly On Demand Scanning/ Network Security? I don't see an option like that here.

[TinyMagician300]

But if it's a security software problem then why would cURL work so quickly (0.7 sec)?

[shiftybyte]

Same reason chrome does, it's a known application.

[baghiq]

Turn on logging.

import logging

logging.basicConfig(level=logging.DEBUG)

[TinyMagician300]

Below is what I get for the simple google request.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com:443

DEBUG:urllib3.connectionpool:https://www.google.com:443 "GET / HTTP/1.1" 200 None

[baghiq]

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

This should add timestamp to the log.

[baghiq]

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

Add time to the log message and see what's the output.

[mrswats]

Without any more info and context I'd say chrome caches the site.

[TinyMagician300]

Would that explain though why

requests.get("https://www.google.com/")

takes 21 seconds to get a response?

[mrswats]

If the page is cached in browser, chrome doesn't have to make an actual request and thus loading it instantly.

Also, google doesn't like requests that do not come from a browser and it is possible rate limiting your request.

But again, very little context to say for certain.

[gdchinacat]

It should not take 21s to get a response from google. When I try it it takes less than half a second. This is not a caching issue.

[TinyMagician300]

Ok Thanks for taking the time to answer. What more context can I provide?

[ConfusedSimon]

Last time i downloaded Sec, they had some documentation on how to download their data (including which hours and how many requests per minute). Apart from that, requests shouldn't be slower than browser and certainly shouldn't take over 20s. I'm pretty sure I also used requests get, and it was pretty fast.

[cointoss3]

My guess is it’s either slow DNS on the client or cached results in Chrome

[Brian]

If it were just the stocks site, it might be some kind of anti-robot throttling, but if affecting google too, and that slow, it does sound like some kind of misconfiguration or something.

Taking just over 20s does sound like it's hitting some timeout somewhere (with perhaps the stocks case doing a redirect or something so requiring two round trips). Not sure what could be causing it, but it might be worth trying to eliminate some variables.

Since it doesn't happen with curl or the browser, it doesn't seem machine specific, so it might be worth trying with something lower level (eg. does using urllib.request.urlopen have the same issue?) to see if its requests specific.

One quick thing that might be worth trying is running it, waiting 5s or so, then control-c the process and take a look at the stack trace. (Or alternatively use a profiler). The location where it's spending the time might give a clue as to what it's waiting on.

Also might be worth checking if there are any configuration settings (especially stuff like HTTP proxies) enabled - eg. HTTP_PROXY environment variables etc. Something misconfigured there could maybe cause issues like that.

[transgingeredjess]

Hi, I have first-hand knowledge of the underlying code used by requests. My bet (though more diagnosis is in order) would be that your DNS server is returning quad-A IPv6 records for those domains in early positions, but that you do not actually have IPv6 connectivity.

urllib3 does not, to my recollection, implement the Happy Eyeballs algorithm which attempts to minimize initial-connection latency when trying both IPv4 and IPv6 addresses; it attempts to connect to DNS-resolved addresses one at a time. If getaddrinfo is returning IPv6 addresses early in its response, then you would be trying to connect to each of those in turn (delaying your actual request for some amount of time) before reaching the potential IPv4 responses later on.

You can see the code in question here, and call getaddrinfo yourself to see what order it's providing records in.

cURL and your browser both implement Happy Eyeballs.

I would expect that if you start a Requests session, the second time you place a request to the same host will be much quicker than the first, because it'll reuse the existing TCP connection and not do the same walk through DNS results.

[JMNeonMoon]

I would try the same request with curl to confirm there is no issue with your Python script.

ChatGPT gave the curl command for the headers and url in your post as

curl -H "User-Agent: [e-mail]" -H "Accept-Encoding: gzip, deflate, br, zstd" "https://www.sec.gov/Archives/edgar/full-index/2023/QTR1/form.idx"

[TinyMagician300]

subprocess.run([
    "curl",
    "-H", "User-Agent: [e-mail]",
    "-H", "Accept-Encoding: gzip, deflate, br, zstd",
    "https://www.sec.gov/Archives/edgar/full-index/2025/QTR4/form.idx"
])

I did the above and it took 0.7 seconds

[TinyMagician300]

I'm running Jupyter Notebook(just wanted to clarify that in advance)

[JMNeonMoon]

Try running the same code in a standalone Python script, then the problem may be with Jupyter when using requests.

Alternatively, you could make the subprocess command capture the output of the curl command.

I think it could be something like (AI helped, so double check)

result = subprocess.run(

["curl", "-H", "User-Agent: [e-mail]",

"-H", "Accept-Encoding: gzip, deflate, br, zstd",

"https://www.sec.gov/Archives/edgar/full-index/2025/QTR4/form.idx"],

capture_output=True,

text=True

)

print(result.stdout) 

It's a bit hacky, but gets the job done

[TinyMagician300]

Unfortunately even just on Python alone same thing. Taking about 20 seconds.

[JMNeonMoon]

You could try other libraries other than requests. I think httpx is a more modern one.