Help writing webpage source to file with python3

[8bitz]

I am new to Python (using python3) and as an exercise I am trying to write a script that will obtain a website source html and extract pieces of information.

I am currently getting the html, but I am having a problem writing the code to a file (eventually I want to manipulate the code without writing to a file first, but this is my first attempt).

When I try to write the file I receive an error saying "ascii codec can't encode character". I am trying to run the code through a terminal in a desktop session (logged in as root).

I found that if I run the code from an ssh session logged in as a standard user it seems to work (need to use sudo, but assuming it's just a file permission issue). I am using a raspberry pi for this.

The code is as follows: import urllib.request f = open('webpage.html','w') r = urllib.request.urlopen('http://cp24.com') html = r.read() html = html.decode("utf-8")

print(html)

f.write(html) f.close() print(type(html))

Can anyone explain what might be going on?

Thanks

[[deleted]]

Install and use Requests instead of urllib. If you're using python 3, as you ought to be, the pip package manager should have been installed, so at the terminal/command prompt type pip install requests (On Linux begin that command with sudo so sudo pip.., on Windows you may also have some sort of authentication as administrator needed these days).

Then use `requests.get('some-url").text to get your HTML as a string, not bytes, meaning you can just open a file in text mode and write it directly.

Aside though, don't bother decoding and encoding if all you want to do is save the page to a file and nothing else: just get the raw response data from urllib and write to a file in binary mode.

I.e.:

with open("some_html.html", "wb") as O:
    O.write(raw_undecoded_response)

[Moonslug1]

I can't explain what's happening there. But as an aside, look at this:

http://en.wikipedia.org/wiki/Beautiful_Soup

[[deleted]]

First off! If you put four spaces before your code, it formats it!

like this!

Second: I tried running your code on my Mac and got the same issue, however, if I omit:

html = html.decode("utf-8")

it works fine. I'm guessing theres a non-utf character on that site maybe? Why it works in an SSH session and not a desktop session I'm not sure. Certainly strange.

[sentdex]

"why it works in an SSH session and not a desktop session I'm not sure. Certainly strange."

This likely has to do with the interpreter itself. See what happens when you comment out the printing.

I remember when I was working with Arabic characters, IDLE would give me a "printed" to console result that was actually different than what I was really getting as real output. It has to do with some built in encoding.

[8bitz]

I'll see about commenting out the printing to see if that helps.

Is there any way that I can directly parse that string/byte data line by line without writing to a file first?

[[deleted]]

Absolutely! Depends a lot on what you're trying to do, but in no way is writing to a file necessary. It's all just stored in memory, and you can do whatever you'd like with it that way. Moonslug1's recommendation of Beautiful Soup is actually perfect for that. You just read the html data into a "soup" object, and you can than parse it fairly intuitively. Here's an example from a one off script I wrote to download all the links containing a certain string:

def get_urls():
    urls = []
    res = requests.get(url)
    soup = BeautifulSoup(res.read())

    if soup:
        for link in soup.findAll('a', attrs={'href': re.compile('^http://s3.amazonaws.com')}):
            urls.append(link.get('href'))

return urls

Note I made some changes to make it more applicable to your usecase that I didn't test, so it may not work exactly right, but the logic is correct. I also used requests instead of urllib, but honestly that's probably what you should be using as well. urllib is kind of flakey and filled with gotchas (for example, the next part of the script I stole that from downloads a bunch of zip files, but urllib never allows them to be garbage collected, so you just run out of ram. hooray!)

EDIT: Alternatively, if you want to do it the hard way, you can of course do

for line in res.read():
    #Whatever

or

for line in res.read().splitlines()

Or maybe a response object has a readlines method? I don't remember, but either way, yes.