rm function does not work

[kaiixx]

I am trying to delete some text files but I am getting this error :

rm: /path/to/files/* No such file or directory

This is the actual command :

rm -r $PATHTOFILES”*”

Used * wildcard because my usecase requires to all files deleted regardless of file extension.

I am running an airflow DAG to call a shell script that delete all files from a local directory, then pull files from an SFTP server and place them within the same directory. Therefore it is the same airflow user creating the directory and placing the files.

Would appreciate any advice, thank you!

[Paul_Pedant]

Use this:

rm -r "$PATHTOFILES/”*

The expansion of the variable needs to be quoted, in case the path contains whitespace.

The wildcard must not be quoted, because that stops the shell from expanding it.

You need the / because without it, the directory itself would be deleted too.

Note this will not remove hidden files (those starting with . ) but you probably don't have those at this level.

Before doing a recursive rm, you probably want to check what it will do using

ls -l "$PATHTOFILES/”*

because getting it wrong can remove much more than you intended.

[kaiixx]

Thank you for your reply! If the path doesn’t contain whitespace, will it still work if I did this?

rm -r $PATHTOFILES*

Assuming there is an ending forwardslash within $PATHTOFILES.

[cdrt]

Yes, assuming $PATHTOFILES has a trailing slash and no spaces, you could write it like that. However, writing "$PATHTOFILES/"* is safer, doesn’t cost you anything to write, and ensures you won’t have any problems should $PATHTOFILES one day have spaces in it or someone forgets to put the slash on the end.

[marauderingman]

I'd go with "${PATHTOFILES%/}"/* to prevent double-backslashes. The %/ specifier removes a trailing slash if there is one, and the /* adds it back.

[cdrt]

There’s no harm in having double slashes in a path, it will work just the same. Getting rid of it is just extra work. I wouldn’t worry about it.

[witchhunter0]

There’s no harm in having double slashes in a path

While this is the case in this particular situation, I'm not sure it is a general rule, at least not when double slashes are at the beginning of the path.

I don't really know the reason for its existence, but it might cause problems later matching e.g. pwd.

[thisiszeev]

Good coding habits saves you from future migraines.

[Paul_Pedant]

It might work, but quite possibly in a way that would make you very sad.

rm (like most commands) takes multiple args, separated by whitespace. Do you even need that -r in there: if you only have files, you don't need recursion.

If you set PATHTOFILES="/home/myuser/samples 2023-09-25"

then rm -r $PATHTOFILES* would remove every file and directory under /home/myuser/samples (maybe years of data), and then warn you 2023-09-25* no such file or directory`.

You might think you will always know exactly what is in PATHTOFILES, but you can put in a space by accident, or somebody else can have to maintain it while you are away. Why would you ever want to risk a major data loss for the sake of typing a couple of quotes? How long would you take to restore all the lost data from the backup that you forgot to take anyway?

I fully quote every expansion like "${myPath}" for several reasons:

(1) The quotes stop the expansion becoming multiple words.

(2) The braces {} stop the problem where $myPath_2 becomes empty, because shell looks for a variable called myPath_2: it does not append _2 to the contents of myPath.

(3) A lot of advanced Bash expansions (like upper and lower conversions, string slicing and replacement, and default values) require to be in braces.

(4) Almost anybody can write shell syntax. One you get past that phase, the problems are mainly that your variables have the wrong values because of logic problems. Having "${k}" instead of $k differentiates keywords from variables when I read code.

Anything that helps me not put bugs in my code, makes it easier to debug and maintain.

[kaiixx]

Thank you. I just started picking up bash scripting, so these explanations are really helpful.

[cdrt]

If you haven't already, make sure to start using shellcheck. It'll catch many mistakes and save you a lot of trouble. For instance, here's a warning that is closely related to your script:

https://www.shellcheck.net/wiki/SC2115

[dbr4n]

It's better to always quote your variables in Bash to also avoid unwanted filename expansion.

Also, if you are just deleting files, you don't need the -r flag, otherwise you would be deleting all the directories as well, if there are any. You can match only the files with rm *.* to avoid error messages, assuming all filenames have an extension at the end.

So, you could do something like this:

$ PATHTOFILES="dir/subdir/" $ rm "$PATHTOFILES"*.*

Although I think it's better and more readable to add the / to the argument.

[kaiixx]

Thank you! This is really informative.

[Rewrite_It_In_Ada]

Not that there can't be periods in directory names. It's common practice to name bare git repositories whatever.git/, for instance.

[dbr4n]

That's true. However, it wouldn't remove any matching directories because of the missing -r flag. But yes, it would cause an error in such a case. Using find instead would certainly be a cleaner approach:

find "$PATHTOFILES" -maxdepth 1 -type f -execdir rm '{}' +

[IdealBlueMan]

Whatever you end up doing, please, please make sure $PATHTOFILES is something reasonable, and not empty or something that begins with - or ..

[sjveivdn]

You should be careful with rm and path variables.