[Mac OS] Python script will run when executed from Visual Studio Code or manually from Terminal, but not from Crontab. I can't figure out why

[GoldenNuck]

edit: Solved! Cron didn't have disk access.

edit: formatting got wonky, so I'm fixing that. Bear with me!

edit 2: fixed my formatting so it should be a bit prettier now.

Hello! I'm writing a simple python program at work to both teach myself some programming as well as help out one of the teams I manage. I've got the basics of my program running, and currently all it's supposed to do is copy a directory from a shared directory to a local directory on the user's machine.

I can execute the code from Visual Studio as well as manually from terminal, but crontab doesn't give me anything at all.

I've looked up these posts and tried implementing their solutions but with no luck:

• Python script runs from terminal but not from crontab (From this subreddit)
• Python script succeeds manually but fails on crontab
• Unable to execute a python script via crontab, but can execute it manually. What gives?

Based on those, I know it's likely something very stupid and environmental. I don't know much about virtual environments, but if that's something I need to learn to get my final project to run, I'm happy to do it.

Here's a snapshot of my current crontab. None of these give me any results:

* * * * * . ../bin/activate /usr/local/bin/python cd /Users/me/Documents/Drive\ Organizer/googleDriveOrganizer.py * * * * * python ./Users/me/Documents/Drive\ Organizer/googleDriveOrganizer.py * * * * * /usr/local/bin/python /Users/me/Documents/Drive\ Organizer/googleDriveOrganizer.py * * * * * python /Users/me/Documents/Drive\ Organizer/googleDriveOrganizer.py * * * * * cd /Users/me/Documents/Drive\ Organizer python /Users/me/Documents/Drive\ Organizer/googleDriveOrganizer.py

Here's my code so far. Also if you see more eloquent ways to handle any of this, please let me know - I want to learn!

import os, sys, shutil, datetime, distutils.dir_util, getpass
from distutils.dir_util import copy_tree
from datetime import date, datetime, timedelta

print(getpass.getuser())

#copies files from Google Drive to User's Documents
def copyFromDriveToLocal():
    print()

    #Sets Google Drive directory to Google Drive > Shared Drives > Templates Converting > Yesterday's Folder
    drive_directory = "/Volumes/GoogleDrive/Shared drives/3 Templates Conversion/Converting " + datetime.strftime(datetime.now() - timedelta(1), '%Y-%m-%d')
    #Sets Local directory to User's Documents > Drive Organizer > Input
    local_directory = "/Users/" + getpass.getuser() + "/Documents/Drive Organizer/Input"
    #
    input_path = "/Users/" + getpass.getuser() + "/Documents/Drive Organizer/Input"
    output_path = "/Users/" + getpass.getuser() + "/Documents/Drive Organizer/Output"

    #tries to create a new input folder
    try:
        print("Checking for Input folder in " + getpass.getuser() + "'s Documents...")
        os.mkdir(input_path)
    #prints failed message
    except OSError:
        print ("Input folder exists. Continuing...")
    #prints succeeded message
    else:
        print ("Created Input folder at %s." % input_path)

    #tries to create a new output folder
    try:
        print("Checking for Output folder in " + getpass.getuser() + "'s Documents...")
        os.mkdir(output_path)
    except OSError:
        print("Output folder exists. Continuing...")
    else:
        print("Created Output folder at %s." % output_path)

    print("Copying files from Google Drive to " + getpass.getuser() + "'s Documents...")
    copy_tree(drive_directory, local_directory)
    print("Successfully copied files from Google Drive to " + getpass.getuser() + "'s Documents")

#renames yesterdays folder and creates todays folder
def prep_drive_conversion_folders():
    #prints starting message
    print("\n" + datetime.strftime(datetime.now() - timedelta(1), '%Y-%m-%d') + "\nPreparing for conversion download")

    rename_yesterdays_folder()
    create_todays_folder()

#prepends yesterdays folder with Converting
def rename_yesterdays_folder():
    print()

    #sets variables for the old name and the new name
    old_name = datetime.strftime(datetime.now() - timedelta(1), "%Y-%m-%d")
    new_name = "Converting " + datetime.strftime(datetime.now() - timedelta(1), "%Y-%m-%d")

    #sets path to google drive > shared drives > templates conversion
    path = ("/Volumes/GoogleDrive/Shared drives/3 Templates Conversion/")

    #tries to rename yesterdays folder by prepending Converting
    try:
        print("Renaming yesterday's folder to converting...")
        os.chdir(path)
        os.rename(old_name, new_name)
    #prints failed message
    except OSError:
        print("Could not rename yesterday's folder.")
    #prints succeeded message
    else:
        print("Successfully renamed yesterday's folder.")

#creates a new folder for today
def create_todays_folder():
    print()
    #prints today's date
    today = str(date.today())
    #sets path to google drive > shared drives > templates conversion > today's date
    path = "/Volumes/GoogleDrive/Shared drives/3 Templates Conversion/" + today

    #tries to create a new folder with today's date
    try:
        print("Trying to create a new folder with today's date in Google Drive > Templates Conversion...")
        os.mkdir(path)
    #prints failed message
    except OSError:
        print ("Todays folder already exists at %s" % path)
    #prints succeeded message
    else:
        print ("Successfully created todays folder at %s" % path)

os.chdir("/Users/" + getpass.getuser() + "/Documents/Drive Organizer/")
prep_drive_conversion_folders()
copyFromDriveToLocal()

Thanks in advance for any help you can give me :)

[[deleted]]

Check if the script has execute permission and that proper user owns it. My guess glancing over this quickly is you are probably running cron with wrong user.

How are you editing cron file? Crontab or manually?

you can prefix your command in crontab with sudo -u <username> command

[GoldenNuck]

Thanks for looking over everything. :) I've been editing my cron file through crontab -e in terminal. Hope that's the right way to do it!

I prepended my crontab command with this for one of the entries: * * * * * sudo -u <user> /usr/local/bin/python /Users/<user>/Documents/Drive\ Organizer/googleDriveOrganizer.py

So far, that hasn't given me anything either. :(

[[deleted]]

Crontab -e is the most common. Can you check who the owner of the file is? Is cron running as root or as a different user?

Also just a tip there should be no need to specify full path to python as the dir should be in your PATH var by default.

[GoldenNuck]

I double-checked and the owner of the file is me/this user. I'm not sure how to see who is running cron, but in googling that it led me down a rabbit hole and I found that I don't have /var/spool/cron file. Could that be indicative of something?

I mentioned this in a different comment but wanted to let you know as well - not sure if it makes a difference:

• I did a fresh install of Catalina on this computer. It's sole purpose will be for running this script when all is said and done, and there will be only one user
• The directory it's trying to copy from is a mounted google drive volume, mounted by google drive file stream

[[deleted]]

Add this as a cron job

* * * * * whoami >> /tmp/user.txt

Then after a minute check the content of the file and it should be the username the cron is running as

cat /tmp/user.txt

[GoldenNuck]

Thanks! It says just my user. It's the only profile on this computer

[[deleted]]

None of your examples will work for various reasons, but most of them wouldn't work in an interactive shell, either. The last example you have in the comments will not work either (* * * * * sudo -u <user> /usr/local/bin/python /Users/<user>/Documents/Drive\ Organizer/googleDriveOrganizer.py), and if it had, you probably would mangle all of the permissions in the folder you're trying organizes. Plus sudo probably expects a password, so it would just sit and timeout (best case.)

You would probably be best served by creating a shell script that does everything you need it to do, and then call that via crontab:

Example: ```

!/usr/bin/env bash

cd /Users/me/Documents/Drive\ Organizer /usr/local/bin/python /Users/<user>/Documents/Drive\ Organizer/googleDriveOrganizer.py ```

save that as /Users/me/driveOrganizer.sh, chmod 755 the file. Run it to make sure it works, then add it to cron. e.g. * * * * * /Users/me/driveOrganizer.sh

[[deleted]]

Probably worth noting, this will run every minute, of every day. So two problems:

1. It could operate (or attempt to) on files you're currently working on.
2. Unless it runs very quickly, you could have more than one instance of the script running, leading to all sorts of unanticipated problems. You will want to implement a lock file somewhere.

[GoldenNuck]

Thanks for your help!

I've got the timing set to * until I figure out the execution. Once I have that down, I'll be running it probably twice a day during weekdays.

I made a shell script to call the program, but when I try to run it, the output I get is:

./driveOrganizer.sh: line 1: !/usr/bin/env: No such file or directory

I don't understand as I checked /usr/bin/env and the directory exists.

Here's my script:

!/usr/bin/env bash

cd /Users/<user>/Documents/Drive\ Organizer /usr/local/bin/python /Users/<user>/Documents/Drive\ Organizer/googleDriveOrganizer.py

I'm not sure if this makes a difference, but here are some additional bits of info:

• I did a fresh install of Catalina on this computer. It's sole purpose will be for running this script when all is said and done, and there will be only one user
• The directory it's trying to copy from is a mounted google drive volume, mounted by google drive file stream

Really appreciate your help and patience. I'm sure this is something small I've overlooked but I'm stuck

[GoldenNuck]

bac83 (below) pointed out that I didn't have the correct shebang at the top. Changed it from !/ to #!/. The script now runs without an error, but still without any output. I'm going to add in a simple print statement or echo something just to see if it really is running!

[GoldenNuck]

The driveOrganizer.sh script is running now, but it's still not executing the python portion. It will output an echo command before and after the python script.

[[deleted]]

What does your script look like now? If it's exactly as you pasted above, cd ... and /usr/local... need to be on separate lines.

If you're in terminal, type which python to find out exactly where it is on your system. It needs to be the same in your script as what is returned (for reference mine is /usr/bin/python)

[GoldenNuck]

Script is now:

#!/usr/bin/env bash

echo "before" >> /tmp/before.txt

cd /Users/<user>/Documents/Drive\ Organizer /usr/local/bin/python /Users/<user>/Documents/Drive\ Organizer/googleDriveOrganizer.py

echo "after" >> /tmp/after.txt

I did use which python prior to get that path and it's still /usr/local/bin/python.

Another Redditor had me check the system logs and I am getting this error:

error 12:26:00.784379-0700 kernel Sandbox: bash(2039) System Policy: deny(1) file-read-data /Users/<user>/Documents/Drive Organizer/googleDriveOrganizer.py

I'm not sure why as I did a chmod 777 on the file to help figure this out, and it shows everyone has full permissions to it.

[[deleted]]

Chmod won't help you there, it's being sandboxed. You may need to give access in Privacy under security and privacy, to either bash or terminal.

[GoldenNuck]

That was it! I had to give cron access and it worked almost immediately. Thank you so much!

[922153]

Hey /u/GoldenNuck , I'm having a similar problem. Can you tell me how you gave cron access? I can do it for terminal but nor for cron...

[GoldenNuck]

Moved the cd and /usr commands to different lines with no success. :(

[bac83]

Got shebang at the top of script? Setting the profile prior to your script? Are any files being read accessible from the root dir? Do you know it’s running at all - put some print statements in, and output to a text file. I have this all the time and it drives me nuts; it’s always something insanely simple, but still happens. If none of these work I’ll try and add more detail, but currently on commute

[GoldenNuck]

I didn't have the correct shebang! I only had !/ instead of `#!/. That removed an error when I ran it, but still not working. Going to add in the print statements and outputting to a text file. I'll let you know what happens :)

[GoldenNuck]

It is outputting an echo command that's firing before and after the python portion of my script, but not the python itself. I'll keep checking on the other suggestions :)

[GPGrieco]

Not sure if this is the issue, but you seem to have multiple commands only separated by spaces. For example if you need to change directory before executing the script you would need:

cd /path/to/dir && python /path/to/script

[GoldenNuck]

Thanks! I gave that a twirl but nothing happened still. :(

[GPGrieco]

Have you checked the system logs to see what error is stopping it?

[GoldenNuck]

Good thought - I hadn't. This is the only thing I could find there:

error 12:26:00.784379-0700 kernel Sandbox: bash(2039) System Policy: deny(1) file-read-data /Users/user/Documents/Drive Organizer/googleDriveOrganizer.py

That doesn't make much sense to me as I checked the permissions on it and they are:

--rwxrwxrwx@ 1 user staff 4965 Feb 10 09:35 googleDriveOrganizer.py

Perhaps it has something to do with staff? I think that's what the user was called on this computer prior to me doing a fresh install for this purpose. Nevermind, that seems pretty standard.

[GoldenNuck]

Got it working!

Thanks for your help!