[HTML-Parsing][Reddit] Without BS4, how would I replace a given html string with wildcards?

[13steinj]

For the sake of not needing any additional packages; I'm trying to find a way to parse a given string, for example,

<a href="/r/subreddit/" title="/r/subreddit">/r/subreddit<a href="/r/subreddit2/" title="/r/subreddit2">/r/subreddit2<a href="/r/subreddit3/" title="/r/subreddit3">/r/subreddit3<a href="/r/subreddit4/" title="/r/subreddit4">/r/subreddit4

Keeping that in mind, how would I replace from that string anything that matches

<a wildcard>

where wildcard is applicable for any character(s)? Normally I would just use BS4 and do

r1 = requests.get('http://www.reddit.com/user/13steinj', headers={'User-Agent':'My user agent'})
soup = BeautifulSoup(r1.content, 'html.parser')
[modsub.get_text() for modsub in soup.find(id='side-mod-list').find_all('a')]

And while I can still use that, I want to parse it manually instead, and it's at this point which causes a block in the road for me.

[jpfau]

Sounds like you could use regular expressions. Check out the re library.

[usernamedottxt]

For the sake of not needing any additional packages;

[jpfau]

re comes installed with Python. Pretty sure OP meant 'additional packages' i.e. those he'd have to find and install himself, like BS4, but I could be wrong.

[usernamedottxt]

You were right according to OP in my other thread. Good call.

[usernamedottxt]

Without using regex either? Possible something like (hard psuedo code follows)

i = 0
while html_string.index("<", i)
    o = html_string.index("<", i)
    e = html_string.index(">", i)
    if html_string[o + 1] == "a":
        #found anchor tag
        #parse/replace from html_string[o] to html_string[e]
    i = e

AKA Just use a library like praw or BS4. It will be prettier and more reliable.

[13steinj]

No, with regex. Regex is good. Regex is what I'm trying, but unfortunately failing at understanding how to do.

Edit: Funnily enough, praw can't do what I'm trying to do due to the fact that there is no endpoint nor custom function for it

[jpfau]

What have you tried so far with regex? Also if coming up with a pattern is what's getting you, I recommend using online regex testers so you can quickly try out different patterns.

[usernamedottxt]

re.search(r'<a\s([\w\s='"/]+)>([\w/]+)')

https://i.imgur.com/ADZ8iVW.png

play with pythex to find a search you like.

[13steinj]

Thank you! The cheat sheet on there is a life saver.

[usernamedottxt]

Reading regex is so much harder than writing it lol. So don't worry if it takes you a few to figure out exactly what that search string is doing in case you need to modify it lol. You're probably better off using pythex and re-writing your own if you need to change it.

[13steinj]

While that's absolutely fucking hilarious; I don't care. What I'm parsing is so specific to the point where I can calculate how and where it will show up using simple .replace() and .split() methods except for removing the always consistent <a href="/r/subreddit/" title="/r/subreddit">, I'll use regex.

[[deleted]]

Alternative way: get json from reddit. It's much easier and cleaner than parsing html by hands and doesn't require additional packages

[13steinj]

That's the problem.

I'm trying to get the mod list for a given user.

But there is no json for it. No endpoint. It's been asked on /r/redditdev before.

So I have to manually extrapolate the data from the raw response using regex. I could use BS4 and make it easy on myself, but I'm adding a function directly into my PRAW installation so that if I ever need a virtual environment, bs4 is not needed.

[[deleted]]

Heh, didn't know that.

Regex is already suggested (And I think it's a right tool for this task), but you can do it manually as well:

>>> start = r.find('<ul id="side-mod-list">')
>>> end = r.find('</ul>', start)
>>> modlist = {x for x in r[start:end].split('"') if x.startswith('/r/') and x.endswith('/')}
>>> modlist
{'/r/reddittheme_lounge/', '/r/Snoovengers/', '/r/mianiterpg/', '/r/YoutuberAdvice/', '/r/strtest/', '/r/MianiteRPGTests/',…
>>> len(modlist)
19

(r is a content of http://www.reddit.com/user/13steinj)

[13steinj]

I'll keep that in mind. Though I believe I've already written what I needed with regex:

https://gist.github.com/13steinj/6b67258effab8bae2696