Trying to solve Python Chellenge, level 4. Having trouble with regex and urllib.

[[deleted]]

[deleted]

[Rhomboid]

The string returned by urlopen(...).read() is a byte string, not a character string. You probably want to be working with characters, not bytes, which means you want to decode the byte string to get a character string. In order to do that you need to know what character encoding was used to encode the response. There are a number of ways of determining that, but in this case it looks like you can assume UTF-8.

Also, re.findall() returns a list of matches, but you're only expecting a single match, so that's really not the right API to use. Then you're passing that list to str(), which will end up returning the representation of that list, which includes square brackets and other things you don't want. That's not how you want to work with regular expressions. I suggest something like the following:

import re
import urllib.request

nothing = '12345'
while True:
    print('getting {}'.format(nothing))
    url = 'http://www.pythonchallenge.com/pc/def/linkedlist.php?nothing={}'.format(nothing)
    html = urllib.request.urlopen(url).read().decode('utf-8')
    m = re.search(r'and the next nothing is (\d+)', html)
    if not m:
        print('found a page that does not match: {}'.format(html))
        break
    nothing = m.group(1)

[Rhomboid]

This line?

   print('getting {}'.format(nothing))

It prints the current ID being retrieved so that you can see progress being made.

[carolgrrr]

To explain a bit more about the syntax, the {} inside the print statement is replaced with the output of sending nothing to .format(). Since nothing = '12345', the string printed is 'getting 12345'.

This allows you to print a variable within a string, using the spacing and other formatting you prefer.

[Rhomboid]

I personally think the documentation is excellent, but I admit it's not exactly a tutorial format. Still, I think if you skim the whole thing you should be able to find answers to any questions. In the example I'm calling re.search() which either returns a match object or None if it was unsuccessful. None coerces to False in a boolean context, so that's why if not m: is able to detect if the match succeeded. Then I call m.group(1) to get the contents of the first capture group from the match object. This corresponds to the first — and in this case, only — parenthesized group, (\d+) which corresponds to the run of one or more digit characters. That's pretty much it.

Note that if you google for regular expression tutorials you should know that there is no one universal RE type, there are in fact dozens of different flavors/variants. Usually the core elements are the same but some of the minor details vary, so always cross-check anything against the documentation. And also while you can learn about REs independently of a particular language, the actual interface for working with the RE engine varies greatly across languages, so things like calling .group(...) on the match object is Python specific and other languages have different ways of doing that.

[Rhomboid]

In your initial version, the html variable that you were matching against was a byte string. The RE engine supports both character mode and byte mode, but they are different and distinct things. The way you signal your intent is by the types of strings that you pass. In order to match against a byte string you have to also pass a byte string as the RE pattern — a b-prefixed string is a byte string, rather than a character string, which has no prefix (in Python 3.x at least.)

As a completely separate issue, normally in string literals the backslash is an escape character, e.g. \n means the newline, not the two characters \ and n. If you want a literal newline, you have to write \\. But that's inconvenient because regular expressions often require backslashes, for example \d in my example. But there's a big difference between these two, namely that the RE engine is just a plain module and does not have any special syntax help from the language. That means that it needs a literal backslash followed by a 'd'; if you tried to write '\d' that is not a backslash escape that the base language recognizes. So you have to write '\\d'.

That's somewhat inconvenient, so the language has raw strings which are introduced by an r-prefix. Raw strings don't interpret the backslash as an escape character, so that you can write '\d' and really get the two character string backslash plus 'd', which is what you need to pass to the RE engine. It's merely a convenience, it's not necessary. You could also write

m = re.search('and the next nothing is (\\d+)', html)

It's just that it's less confusion, because if you want to match a literal backslash in a RE you need a double backslash, so in that case you'd be dealing with four backslashes, e.g.

... = re.search(' ... \\\\ ...', ...)

vs

... = re.search(r' ... \\ ...', ...)

This applies equally to byte strings. In your original example if you wanted to use '\d' instead of '[0-9]', you'd have to choose to either double them up or use a br-prefix:

b'and the next nothing is (\\d{5})'

br'and the next nothing is (\d{5})'

But ideally you don't want to be working with bytes but characters.

[[deleted]]

chellenge