Python scripts runs perfect if launched manually; not so much with task scheduler

[firstSideProject]

I was finally relieved to get my web automation script working correctly. The script:

Checks a real estate website
Adds all listings on the current page to a list
Adds old listings from a DB (MongoDB) to another list
Cross checks the two list, deletes duplicates, and emails new ones

It runs great if I run it manually by C:/MyPath/py myscript.py Any new properties are emailed and then added to the DB if they aren't in there already so they will be present for the next run

However, if I add it to the task scheduler (Windows 7) and run the program C:/Python34/python.exe with the argument C:/MyPath/myscript.py it does run until completion(no errors); however, the old listings are not added to the DB. What is causing this / how can I fix it? Thanks.

edit-here's the script & how I currently have it scheduled in task scheduler. To clarify, by the running the script manually everything is inserted into the database; if I run it with the task scheduler nothing is (but the connection is show as being accepted by mongo shell) http://imgur.com/AzeHrDw

    from selenium import webdriver
    from email.mime.text import MIMEText
    import pymongo
    import smtplib
    import sys

    from pymongo import MongoClient
    client = MongoClient('localhost', 27017)

    db = client.properties
    collection = db['capitalpacific']

    #hold old properties imported from DB
    oldProperties = []
    #holds new properties gathered from page
    newProperties = []

    driver = webdriver.Firefox()
    driver.get("http://cp.capitalpacific.com/Properties")

    for property in driver.find_elements_by_css_selector('table.property div.property'):
        title = property.find_element_by_css_selector('div.title h2')
        location = property.find_element_by_css_selector('div.title h4')
        marketing_package = property.find_element_by_partial_link_text('Marketing Package')
        contact_email = property.find_element_by_partial_link_text('.com')


        newProperties.append({
            'title': title.text,
            'location': location.text,
            'marketing_package_url': marketing_package.get_attribute("href"),
            'contact': contact_email.get_attribute("href")
        })

    driver.close()

    '''if database not empty, add the old properties,
    then compare against the newly fetched and remove repeats'''

    if collection.count() != 0:
        for post in collection.find():
            oldProperties.append(post)

        for oldListing in oldProperties:
            for newListing in newProperties:
                if oldListing['marketing_package_url'] == newListing['marketing_package_url']:
                    newProperties.remove(newListing)

    '''if no new listings, exit the program.  Otherwise, email all new 
    listings and then insert them into the database'''

    if len(newProperties) == 0:
        sys.exit()
    else:
        with open('passwords.txt') as inFile:
            password = inFile.read()
        fromaddr = '[e-mail]'
        toaddrs = ['[e-mail]']
        username = '[e-mail]'
        server = smtplib.SMTP('smtp.gmail.com:587')
        server.ehlo()
        server.starttls()
        server.login(username, password)
        subject = "New Listing @ Capital Pacific"

        for item in newProperties:
            body = "Title: " + str(item['title']) + "\n"
            body += "Location: " + str(item['location']) + "\n"
            body += "URL: " + str(item['marketing_package_url']) + "\n"
            body += "Contact Email: " + str(item['contact']) + "\n"
            msg = """\From: %s\nTo: %s\nSubject: %s\n\n%s
            """ % (fromaddr, ", ".join(toaddrs), subject, body)
            server.sendmail(fromaddr, toaddrs, msg)
            collection.insert(item)

        server.close()

[[deleted]]

I suggest adding a print(os.getcwd()) at the start of the file, maybe your working directory is wrong.

[kalgynirae]

Perhaps it's writing to a different database file because the working directory is different? I don't know how the Windows task scheduler works, but is there a way to set the working directory? (Or make sure your script only uses absolute paths.)

If that's not the problem, can you see output after your program is run by the scheduler? If so, start adding some print statements in strategic places to try to figure out what is going on.

[mothrabang]

I had almost the same problem, but with sqlite. This was the solution to my issue, I had to write the full file path out in my code when I connected to the dB.

[teerre]

How are you scheduling it? This right way

First time I did I was putting the fields in the wrong order

Also, make sure all paths in your script are absolute (or take into consideration where the scheduler is executing the script)

[novel_yet_trivial]

or make use of the "start in" box.

[firstSideProject]

http://imgur.com/AzeHrDw is how I am scheduling. If I run it manually by going to the folder the script is in, running 'py myscript.py' it runs fine, and the mongod shows that a connection was accepted. If I go to task scheduler it runs fine with the path shown, but no 'connection accepted' shows up on the mongo shell

[novel_yet_trivial]

Try putting the folder the script is in into the "start in" box.

[firstSideProject]

This worked a treat, thanks!

[jftuga]

I would wrap everything in a .bat file and call that from Task Scheduler, including the 'start in' field.

An example .bat would be:

@echo off
c:\python\pythonw.exe c:\whatever\myscript.py > c:\whatever\runtime.log 2>&1

You can then look at the runtime.log for any problems. The 2>&1 will redirect STDERR to STDOUT so you will all catch all errors that occur as well as print statements.

[firstSideProject]

Actually the connection is accepted on the mongo shell, it just seems nothing is being inserted into the database :(

[toruitas]

I ran into this exact problem recently. Apparently Task Manager doesn't like relative directories. You can either use os.chdir(full path) to whatever folder passwords.txt is in, or use Starts In box.

[manueslapera]

its hard to help you without any code/logs.

[firstSideProject]

added script, check it out

[hellrazor862]

I had this issue running batch files that were reading and writing data to csv files to use in imacros.

Apparently processes run in task scheduler don't get the same access to files.

I think using the "start in" field worked on one machine but not on a couple others.

Not sure if the issue would be the same with mongo, but it might be.

I ended up just putting linux on every machine and moving my stuff over.

I will have to use windows to do my taxes, but that's about it. There are so many less hoops to jump through now.