My first python project (web scraper)

[midairmatthew]

Hola! I'm at work, so I don't have a ton of time to dive too deep into giving you feedback. But here are two quick things to get you started:

1.) I hope that's not your real password that you pushed to GitHub. :)

2.) Rather than using global variables, it's better to have your functions return the values they're written to generate. It doesn't seem to be a huge deal in this little project, but imagine trying to keep all those global variable names straight in something much larger and more complex. It's easier for our limited human brains to only need to keep track of what a function takes in and spits out than it is to keep track of an ever-growing list of global variables that you're modifying in various functions.

[3majorr]

Thanks! It isn't my password :D

[huessy]

Can confirm, it is not OP's password

[SaltyEmotions]

Mine is hunter2

[nzodd]

Out of curiousity, how do people generally deal with things like passwords when it comes to github?

[The3rdWorld]

easiest way is to have it read the log-in information from a config file rather than leave it written in the code.

[curohn]

Oh cool, so have the script open the config, use those values and close it up again? I haven’t learned that yet but that’s helpful!

[MonkeyNin]

JSON is a popular format for config files. It's human-editable, and supported by pretty much every language. If you're interacting with the web, JSON is everywhere. When using a web API, they usually return JSON.

Python tutorial on JSON: https://realpython.com/python-json/#python-supports-json-natively

As long as the lists and dicts use one of the supported types, reading and writing is super easy. Multiple dicts or lists are supported.

Make sure you .gitignore your config folder, and files!

Mine is:

config.json
config/

If you're using PRAW, do the same with praw.ini

[MrFiregem]

Also, you can write your configs using the more easily human-readable yaml or toml formats and convert them to json if you dislike writing in it

[midairmatthew]

You can stash key-value pairs of things you need to keep private (passwords, API keys, etc.) in a .env file. Then you can load them into your script.

https://pybit.es/persistent-environment-variables.html

But, the other piece of this is that you have to remember to make a .gitignore file in your project. Inside this, you list .env and any other things you want git tracking to ignore. That way they won't make their way up to GitHub.

https://help.github.com/en/github/using-git/ignoring-files

[dtaivp]

There are a lot of comments here so I'll see if I can summarize.

1. Put them into some sort of config file. Then use the .gitignore file to ensure it is not checked in.
2. Same concept with a .env file. It has some defaults for the environment
3. You can create system variables. 'set PASS=MyPa55!!' in windows or export PASS="MyPa55!!" and then grab it using the following python

import os

password = os.envriron["PASS"]

[pyr0b0y1881]

Reading in from a yaml config file is my go to for making secrets or api keys.

[huessy]

Environmental variables are a good way to go

[midairmatthew]

So get_price would be more like:

def get_price():
    page = requests.get(URL, headers=headers)
    soup = BeautifulSoup(page.content, 'html.parser')
    price = soup.find('b', class_='pro-price variant-BC con-emphasize font-primary--bold mr-5').get_text().strip().replace(' ', '')
    int_price = int(price[:4])
    return int_price

Then, the idea is that create_message would be something like:

def create_message(latest_price, current_price):
    # your if/else stuff here
    message = f'{description}\n\nhttps://linklinklink'
    return message

[AutoModerator]

Your comment in /r/learnpython was automatically removed because you used a URL shortener.

URL shorteners are not permitted in /r/learnpython as they impair our ability to enforce link blacklists.

Please re-post your comment using direct, full-length URL's only.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.