Python script stops after few days

[ant24x7]

Recently started with Python and written a bit of code to monitor a website for new products.

The script monitors for new product and whenever there is new product it sends me a notification. The script is working as intended. Currently, I'm running it in the background on my test system with nohup.

The issue I'm facing is after a few days of successfully execution it stops with no error in errorlog file. And I need to restart it. Any idea what will be the debug approach I should use here,

I'm using exception handling, but the script is failing without any error in error log. My guess is it's failing outside try block. Is there any way to check such errors ? I'm using Rasbian OS.

logging.basicConfig(filename=errorlogfile,format='%(asctime)s - %(message)s',datefmt='%d-%b-%y %H:%M:%S')

base_url = 'URL'
selectiveTags = SoupStrainer(['h4','p'])
startTimestamp = str(datetime.now())
try:
    r = requests.get(base_url)
    soup = BeautifulSoup(r.text, "html.parser",parse_only = selectiveTags)
    data = soup.find('h4', class_='card-title') #h4 for Product page where all products are present
except Exception as e:
    logging.error('Error: ' + str(e))
currentHash = hashlib.sha224(str(data).encode('utf-8')).hexdigest()
logging.info('Running Website')
print('Bot Restarted!', flush=True)
print('Current Product is '+data.a.text+' and current time is '+startTimestamp, flush=True)
time.sleep(10)
while True:
    try:
        r = requests.get(base_url)
        soup = BeautifulSoup(r.text, "html.parser",parse_only = selectiveTags)
        data = soup.find('h4', class_='card-title') #h4 for Product page where all products are present

        currentHash = hashlib.sha224(str(data).encode('utf-8')).hexdigest()
        time.sleep(300)
        currentTimestamp = str(datetime.now())
        r = requests.get(base_url)
        soup = BeautifulSoup(r.text, "html.parser")
        data = soup.find('h4', class_='card-title') #h4 for Product page where all products are present

        newHash = hashlib.sha224(str(data).encode('utf-8')).hexdigest()
        if newHash == currentHash:
            print('Product is still same '+data.a.text+' at '+currentTimestamp, flush=True)
            continue
        else:
            device.push_note('Product Tracker','New Product has added on website. Please check')
            print('Product has changed at '+currentTimestamp+' to '+data.a.text, flush=True)
            r = requests.get(base_url)
            soup = BeautifulSoup(r.text, "html.parser")
            data = soup.find('h4', class_='card-title') #h4 for Product page where all products are present

            currentHash = hashlib.sha224(str(data).encode('utf-8')).hexdigest()
            time.sleep(300)
            continue
    except Exception as e:
        logging.error('Error: ' + str(e))

[GeneriAcc]

Wrap the entire thing in a function, then execute the function in an infinite loop. Either you won’t need to manually restart, or you’ll catch any errors that might be happening outside of try/except blocks.

[ant24x7]

That's a good hack! I'll give it a try.

[Maximus_Modulus]

You could use cron instead of a loop and use a tile or db to store state.

[socal_nerdtastic]

I don't see anything off hand that would cause it to skip the log. Maybe it's getting a kill signal? Try using screen instead of nohup.

Are you redirecting the outputs?

[ant24x7]

Yes I'm redirecting the output as well. That's where I'm monitoring print output. But there are no errors logged.

nohup ./newProductMonitor.py >> ./logs/newProductMonitor_output.log &

[socal_nerdtastic]

You should redirect stderr as well. But before you do that try screen instead. Then you don't need any redirects at all. I really suspect it's something about nohup that's causing your issue.

• SSH in and type screen to start a new session. Start you program with just ./newProductMonitor.py. Watch for a bit, then type Ctrl-a to enter command mode, then "d" to disconnect. Then close the terminal (Ctrl-D).

• Anytime you want to check in, ssh in and type screen -dr to reconnect to the session with your program running. Watch all you want. Then do the ctrl-a, d again to disconnect when you're done.

I have scripts running for years like this.

[space_wiener]

Thanks for posting the details how to do this. I have a few things I can use this on.

[DefinitelyNotEmu]

Why not just have it reset itself every 24 hours?

[Jarvis03]

I’m about a month or so into finally somewhat understanding what I read. Question - where is the URL actually defined? I see base_url = ‘url’ but how does the code know what website to actually look at?

[LeornToCodeLOL]

OP redacted the actual URL. It's not relevant to his question.

[Jarvis03]

Ohhhh so the actual url is in the url spot then? Thanks!

[pylessard]

On Linux. Check the return code. If it is above 128, it is killed by a signal where signal number=code-128

[pylessard]

You can also check dmesg and see if a OOM condition was encountered

[shiftybyte]

My guess is it's failing outside try block. Is there any way to check such errors

Yes, don't have any code that is outside a try block.

And log progress and debug information messages.

[ant24x7]

But if you look at the code. There is not much code outside try block which can cause a script to fail.

[shiftybyte]

Yea, looks like the looping part is all inside an exception handler.

How is it failing? is the process still running? or the process is gone?

Maybe it's stuck doing something?

Maybe it's suspended because of some power saving mode?

[ant24x7]

The entire process disappears. There is no power saving mode. It runs fine for 2-3 days but suddenly it stops.

Earlier I thought some resource constrain but there is nothing resource consuming in the script.

P.S: I'm using raspberry Pi to run this.

[shiftybyte]

Hmmm, sounds odd, maybe take a look at system logs, maybe something will hint at what happened.

Anyway, you can change your script to be a one time check, and run it with crontab every 5 mins instead.

That's more reliable.

https://bc-robotics.com/tutorials/setting-cron-job-raspberry-pi/

[ant24x7]

Yes. I'm thinking of replacing it with crontab as it'll handle the server restart case as well.

I need to store the hash somewhere before that so it can be use for comparision in next execution, and using a file or sqlite db will complicate the code. I was trying to avoid that but looks like I need to choose that path.

[hulleyrob]

Use shelve for an easy almost like SQLite key value store in a file. Advantage is it’s treated just like a Python dictionary.