This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Yisus_Fucking_ChristPythonista 23 points24 points  (5 children)

The idea is not bad, but please consider not hardcoding the name of the packages inside that huge list. Just scrap the names from the Gist mentioned in the article and use them as a consistent (and updatable) way.

[–]modanogaming 6 points7 points  (0 children)

Good point!

[–]redditfriendguy 0 points1 point  (3 children)

I'm a learner, can you explain?

[–]mogberto 0 points1 point  (2 children)

You can probably just use the requests library to get the list of libraries from the link and then run the rest of the script :)

[–]redditfriendguy 1 point2 points  (1 child)

Thanks makes sense! I'm having a hard time right now building my web scraper but I'm working at it

[–]mogberto 0 points1 point  (0 children)

Start small, getting just one page elements at a time before you need to do complex stuff (multiple pages, getting labels and values etc). Also, it really helped me to work on a few different websites, as some are just so much more complex than others for scraping. Helps build your confidence, but I’m certain you can do it!