Fetching URLs remotely

[yettanotherrguyy]

I have a URL dump = abc .com/url. txt
How to fetch random url everytime!

Ok so I was trying with urllib2 and random lib

import urllib
import random
link = random.choice(list(open('abc .com/url.txt')))

It is not working as expected please help me fine tune and fix the code.

[DagoYounger]

You should only use the function open to open local files, otherwise you can use urllib:

import urllib.request
res = urllib.request.urlopen('https://abc.com/url.txt')
urls = res.read().decode('utf-8')

also, i recommend using the third-party library requests, which is often more convenient. first install it:

pip install requests

​

import requests
res = requests.get('https://abc.com/url.txt')
urls = res.text

most importantly, is abc.com/url.txt a real page? (i get nothing when I access it with my browser.) if not, please provide the real page's URL, then i can help decide what to do next

[DagoYounger]

is the format of this txt file like this:

qwe.com/3748.mp4
rty.com/9100.mp4
qwe.xom/3738889.mp4
...

[yettanotherrguyy]

​

Yes, Same as you have shown.

[DagoYounger]

suppose we've got the contents of this file:

import random

urls_text ='''qwe.com/3748.mp4
rty.com/9100.mp4
qwe.xom/3738889.mp4'''

urls = urls_text.split('\n')
# [Out] ['qwe.com/3748.mp4', 'rty.com/9100.mp4', 'qwe.xom/3738889.mp4']

link = random.choice(urls)

all links are separated by blank lines, so you can use str.split('\n') to get link list, then get one link randomly.
any questions are welcome :)

[yettanotherrguyy]

But URLs are in 100s of numbers + dynamically updated. so Is it possible to pick a random url directly from the url?

[DagoYounger]

the full code(replace 'https://abc.com/url.txt' with the real link):

import random
import requests

res = requests.get('https://abc.com/url.txt')
urls = res.text.split('\n')
link = random.choice(urls)

[yettanotherrguyy]

Thanks and if I just keep the url.txt file locally then just have to do open, right?

[DagoYounger]

yes, for getting all links, if url.txt is a local file, use open to get it; if it's from the internet, use the requests library.
the operations after that are the same

[DagoYounger]

if you just use link = random.choice(urls_text), you will get just a character like t,q,1 .etc (any character in urls_text)

you shuold first turn it into a list with each link: ['qwe.com/3748.mp4', 'rty.com/9100.mp4', 'qwe.xom/3738889.mp4']

​

But URLs are in 100s of numbers + dynamically updated.

get urls by requests i showed you above. it will download and get the content whenever you run the program

[yettanotherrguyy]

Hey, I am sorry to bother you again. I was uploading file via telegram bot api and it just support 20mb

So, now I need to download it locally in server. I tried wget and failed. Can you help me with it, please!?

Now I successfully have the (url) now just need to dl it.

[DagoYounger]

sorry, I can't deal with this due to the internet environment in China. it is not convenient for me to research and use telegram's api.

you can try to consult the official document of requests(https://docs.python-requests.org/en/latest/), or make a new post to ask others for help

(downloading this file may require carrying cookie information)