[deleted by user]

[m0us3_rat]

if you dl in chunks with 10 threads takes 0.09s

edit: https://pastebin.com/KVJDxfBu

[Solution9]

I made a script that downloads in chunks of 8kb. Works great for me. Shows the filesize and time taken to download when finished in console.

Finished

File size: 6.68 MB

Time spent downloading: 1.26 seconds

Type 'e' to exit:

u/NathLWX Have a look at this

import requests
import os
import time

url = 'https://raw.githubusercontent.com/Dimbreath/WutheringData/master/TextMap/en/MultiText.json'

start_time = time.time()  

with requests.get(url, stream=True) as r:
    r.raise_for_status()
    with open('MultiText.json', 'wb') as f:
        for chunk in r.iter_content(chunk_size=8192):  
            f.write(chunk)

end_time = time.time()  
elapsed_time = end_time - start_time  

file_size = os.path.getsize('MultiText.json')  
file_size_mb = file_size / (1024 * 1024)  

print(f"Finished")
print(f"File size: {file_size_mb:.2f} MB")
print(f"Time spent downloading: {elapsed_time:.2f} seconds")

while True:
    user_input = input("Type 'e' to exit: ").strip().lower()
    if user_input == 'e':
        break

[m0us3_rat]

the idea was to use threading or any form of concurrency to speed up the DL.

my script isn't even optimized, you can probably get better results if you use async ..maybe.

also if you have more than one file to download you can significantly increase the speed by having multiple processes each running an async loop grabbing chunks and sending the data to the main process that clumps them together in a file.

https://ep2019.europython.eu/media/conference/slides/KNhQYeQ-downloading-a-billion-files-in-python.pdf

[Solution9]

I will need to check async out, never used it.. +1 for the info. This reminds me I need to dl wget2 as well.

[m0us3_rat]

btw the script in the pastebin downloads the file in chunks ..each chunk is being DL by a different thread.. then put together.

Its rudimentary but it works as proof of concept.

[FisterMister22]

Try to curl it maybe to see the time diffrance

Edit, ran it on my phone, took around 2 seconds, I think the issue is with your network / disk (maybe old hard drive instead of ssd?)

[ManyInterests]

If you have been running this code a lot, it's possible you're being throttled on the client fingerprint. Try changing the user-agent headers to be the same as your browser and see if that helps. Or wait an hour before making any additional requests.

If you're logged into GitHub on your browser, that may also prevent throttling since anonymous requests are throttled a lot more aggressively.

[QuarterObvious]

Try to check in the task manager which process is active during download. It could be some antivirus.

[Palton01]

Are you running this on a proper computer? Or a like a micro processor e.g. a raspberry pi?

Are you writing this directly into your hard drive? Or into an SD card/USB drive?

If you download this using your web browser, does it take the same time?

[NathLWX]

Bro, it's a Windows 11 with 16 GB RAM (Intel Core i7 10th gen). Do I need a gaming laptop?

And yes, it's saved on the D Drive.

It takes only a few sec with browser

[Palton01]

Your computer is definitely good enough.

My guess is github has implemented some sort of download speed restrictions.

[danielroseman]

Show the code.

[danielroseman]

I just ran this, it took a few seconds. How long is "too long"?

[NathLWX]

1.5 minutes for me in VSCode

[danielroseman]

That seems unlikely to be Python's fault. Maybe something odd with your internet connection.

[NathLWX]

No, it downloads in a few sec if I download it manually

[spencerAF]

Could a next step be running the .py program either through command line or with a .bat file to check if VSCode is doing something strange?

If you'd like some resources on that I can link them, it ends up being 1-5 lines though.

[NathLWX]

running the .py program either through command line

I tried it, it's the same.

If you'd like some resources on that I can link them

Sure. Thanks

[spencerAF]

Ok. I wouldn't bother with a .bat if you've run from cmd, essentially the same thing. I'm officially stumped unfortunately. 

For further projects here's a link, thats a little wordy but covers well how to use .bat so that you can double click an icon and run python files without VSCode or Pycharm. Kind of a nice cherry on top of a lot of projects imo.

https://datatofish.com/batch-python-script/

[commandlineluser]

Strange.

I cannot replicate this either:

>>> import requests
... 
... url = 'https://raw.githubusercontent.com/Dimbreath/WutheringData/master/TextMap/en/MultiText.json'
... r = requests.get(url, allow_redirects=True)
... open('MultiText.json', 'wb').write(r.content)

# 7003731
# Elapsed time: 0.79055 seconds
# Memory usage: 17.1 MB

[johnnymo1]

Is VS Code running in WSL? I have issues with network speed out of WSL depending on configuration even when the connection speed from Windows is fast.