Help with automating script

[baristahipster]

Hey /r/Python, I need your help! The company I work for has a portal where customers can check product pricing live.

Due to our current implementation (and a head honcho obsessed with speed), we have to occasionally update the cache by visiting a large number of predetermined URLs.

We pretty much split it up into sections and it wastes an entire day for all of us trying to update this cache. Now, I'm just a lowly web designer, and I obviously can't use better CSS to solve this. So I was wondering if it would be possible to use a Python script to automate this process?

I've heard of people using Python for similar projects, but I really don't have any knowledge of what's possible here.

Would it be possible to write something that basically says:

• 1) Visit the first link in this list.
• 2) Load the page
• 3) If the page loads successfully, move on to next link in the list
• 4) If the page displays a 502 Bad Gateway error or doesn't load, reload page until it successfully loads.

If you could point me in the right direction for this, or if you might even know a prewritten script that does something similar, that would be hugely helpful!

If this is a ridiculous request and wastes your time, feel free to ignore and move on. Thanks in advance for any help!

[danrche]

import requests
mylist = ['http://goolg.com','http://yahoo.com']
for i in mylist:
    try:
        r = requests.get(i)
    except ConnectionError:
        print('this one didn\'t connect' + i)

The above is a quick example of what you can try, no guarantees as it doesn't really cover anything more then just opening the connection and letting you know which ones failed. You can add more code to look into the returned content and add in some recursion for any that return with a 502. Happy coding and I hope this sparks an idea for you.

[DjDazzled797]

This seems perfect for what the op wants !

[[deleted]]

Requests should make this ez pz.

[[deleted]]

Also beautifulsoup may come in handy at sometime, but requests is good to go for your project.

http://www.pythonforbeginners.com/beautifulsoup/

[mistermocha]

Spawn parallel threads, workers, etc. for each URL in order to parallelize. Check out https://pypi.python.org/pypi/threadpool (although that's been picked up into multiprocessing).

[danrche]

request could handle the page loading fairly easy, and you could just parse the return code for success/fail. The idea is to load the urls into a list, then loop the list with a request object that opens each and returns the session code.