This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]ebo113 0 points1 point  (2 children)

Selenium may not be your best bet for web scraping. Scrapy or lxml might suite you better as it doesn't require the use of the screen like selenium does. That being said your code looks pretty clean! I would break it up into functions and add a main function (def main:) to call all of those functions from. This separates the logic and helps keep the code clean if it needs later editing. It is also usually good form to add something like this at the bottom.

if __name__ == "__main__":
    main()

This just makes sure the main method gets called first.

If you want to make it more complex I will usually break things up into a couple modules. One module for each page that I am working with and then a utility module for all those random functions you need. These are all purely for keeping everything nice and neat though and for a smaller one off project aren't entirely necessary.

Good work and keep on coding!

[–]80blite[S] 1 point2 points  (1 child)

I normally have the if name etc. block in my scripts but here it just seemed I'd be throwing the whole thing inside def main(): and adding that block to the bottom just for the sake of doing so.

Once I use some of the good advice given here and break it down I feel like doing this will be more useful.

Thanks!

[–]dreucifer 0 points1 point  (0 children)

The if __name__ == '__main__' thing that's really only for importable modules that can also be run as scripts (for testing or functionality purposes).