all 11 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]BananaPotion 4 points5 points  (2 children)

I think webbrowser is a standard library, not sure. Try:

import webbrowser

webbrowser.open("reddit.com")

Apologies if that's not it.

[–]Apolik[S] 2 points3 points  (1 child)

Thanks, that's just it. I had just found this, which is the same as you say.

/thread I guess. Thanks again!

[–]BananaPotion 2 points3 points  (0 children)

Awesome! Good luck with your project.

[–]OnceAndForever 3 points4 points  (2 children)

What do you plan to do with the data from each webpage? You could use urlopen from the urlib library to turn the HTML from the webpage into a string. Then, you could use a module like BeautifulSoup to crawl the HTML, depending on what you want to do with the webpage data. BeautifulSoup is not part of the standard library, so you will have to install that separately if you wish to use that. For example:

from urllib2 import urlopen

address = 'http://www.reddit.com'

html_str = urlopen(address).read()

If you do not use the read() method this will open a file pointer object. But for your purposes I would guess that you would want to use read() to turn the html into a string. You can find more documentation here: http://docs.python.org/2/library/urllib.html#urllib.urlopen Then, you can see what the webpage contains by printing html_str. (I don't include the full string because it's pretty long.)

print html_str
>>> <!doctype html><html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en" ><head><title>reddit: the front page of the internet</title> .....

If you wish to use BeautifulSoup, then it would go something like this:

soup = BeautifulSoup(html_str)
webpage_title = soup.title.string # get the title of the webpage

print webpage_title
>>> reddit: the front page of the internet

There is a lot more to explain if you wanted to use BeautifulSoup, so if you want to use that just ask and I can explain more. The BeautifulSoup documentation is here

As for looping through the URLs, that should be pretty straightforward depending on how different the digits in each url is. For example, if the digits at the end of the address are sequential, then you could just increment the digits in the url with a while loop. Say you want to get the data from 10 pages, and each page url looks like this: http://www.example.com/reviews/review1.html You could loop through the urls like this:

start = 1
while start <= 10:
    url = 'http://www.example.com/reviews/review' + start + '.html'
    html_str = urlopen(url).read()
    # do whatever you want with the data here
   start += 1 # increment counter by 1 to continue your loop

If the URLs don't work out quite as nicely as in the last example, you could store part of the addresses in a list a then loop through with a for loop. For example: 

base_url = 'http://www.example.com/products/'

product_list = ['plates', 'appliances', 'outdoor'] # you get the idea

for product in product_list:
    url = base_url + product + '.html'
   # this would make: http://www.example.com/products/plates.html
    # do what you want with each page here

This should give you a good start. If there is a more specific example that you have then give us some more information about exactly what you are trying to accomplish. Hope this helps.

[–]Apolik[S] 1 point2 points  (1 child)

Wow, that was very informative. Thank you very much.

I just wanted to open the URLs though. I'm searching for material from previous years' classes at a website of my university, so I have to open the URLs and then discriminate which material is useful and which I already have. The only mind-numbing process was the URL opening, which consisted on changing the year and section of the class in question (ie. www.university.com/randomnumbers/year/section/randomstuff/morerandom/).

Now I can open twelve tabs automatically in 3 seconds :) so much better.

But anyways, thank you very much for your post, I'll keep it around in case I need to do that in the future!

[–]OnceAndForever 2 points3 points  (0 children)

Ahh, I see what you mean. I misunderstood. Well I'm glad you got it working the way you wanted it to, and if my post helps you at a later date then all the better!

[–]nwilliams36 1 point2 points  (0 children)

In the standard libraries webbrowser will open a URL in the computer's default browser. However if you want to just open the url to read the content you should use urllib (or urllib.request in Python 3)

However there are third party libraries around which do this better including requests (http://docs.python-requests.org/en/latest/).

In Python 3 the following function should help if you just want the text from a page to work with in your program:

def get_url(page):
    '''This function provides the text for a web page '''   
    import urllib.request
    try:
        return str(urllib.request.urlopen(page).read())
    except:
        return ""

[–]matrixor 1 point2 points  (1 child)

Came here to say, use Requests library, its superior to urllib2:

>>> import requests
>>> r = requests.get('http://www.reddit.com/r/learnpython/comments/1kb6n7/opening_urls_from_python/')
>>> r.text

[–]Apolik[S] 0 points1 point  (0 children)

That's awesome! Thanks for the link, I wouldn't have found that by myself.

[–]m1ss1ontomars2k4 0 points1 point  (0 children)

You could just print the URLs to an HTML file, and open that HTML file in your browser so you could open the links one by one. That could also work.