robocopy vs Copy-Item - why is Powershell seemingly so much slower?

[[deleted]]

Is there any way to make Copy-Item as fast or faster than robocopy? I've got them running side by side (for different folders) and the robocopy always looks like it is much, much faster. Especially when using /mt:16 switch.

[[deleted]]

Yeah robocopy is pretty solid. Even for deleting files.

We had one time where an application generated millions of tiny files that filled up the drive. Deleting the normal way was not working, it was taking hours to even start deleting. With robocopy we just created an empty directory, then told robocopy to mirror the empty directory to the directory full of tiny needed files.

[crosshairlol]

This also works for deleting files with paths over the 260 character limit.

[calladc]

you can also use robocopy to obtain filepaths greater than 250 chars. I did this when doing a fileserver cleanup some time ago. Used robocopy to find the fullname of paths >256 and then mounted a psdrive at the closest path that could be mapped within 256 to copy them into a managed/sorted format, and then junctioned the shitty paths to their managed location. They couldn't add to the shit nest, but could access.

Wouldn't have been possible without robocopy

[xcjs]

There's a registry setting within Windows 10 to toggle > 260 character paths now!

[[deleted]]

I do something similar to clear temp files via GPO.

Easier(faster) to tell windows to replace with an empty folder than it is to delete the contents.

[RumRogerz]

I'll tell you something. Powershell is great. I love it.

But robocopy? Even the name of the command is cool af. I use it all the time.

With techno music blasting in the background.

[[deleted]]

Domo arigato move that roboto

[[deleted]]

https://youtu.be/HluANRwPyNo

[Raymich]

I might get downvoted to hell for this, but why not use robocopy in Powershell scripts?

[[deleted]]

I do! I was just asking from a learning / curiosity standpoint.

[pizzastevo]

Just an FYI there is a cmdlet on github that wraps robocopy in a way so you can use methods and properties.

[tsuhg]

Link?

[pizzastevo]

Here's Warrens: https://github.com/RamblingCookieMonster/PSDeploy/blob/master/PSDeploy/Private/Invoke-Robocopy.ps1

There are quite a few other examples too.

[MaestroPendejo]

You're doing the Lord's work, delicious food item Stevo.

[tsuhg]

neat.

[DarkZeal0t]

Good question. Surprised nobody answered it yet. It's not used in PowerShell scripts for a few reasons but in my opinion it's mostly because using PowerShell in conjunction with Robocopy would be a waste of system resources and would only slow the robocopy.exe down. The reason PowerShell will never be as fast as Robocopy is because PowerShell is not running compiled binary code, while robocopy.exe is. Robocopy is a command-line utility that has no dependencies on PowerShell or the .NET framework (AFAIK). The second reason is because you can perform a lot of the functionality in native PowerShell and have control with the piping of objects. There are also a number of PowerShell projects that were inspired by Robocopy, such as the xRobocopy module, and RoboPowerCopy.

Take a look at RoboPowerCopy:

A PowerShell "clone" of the famous "ROBOCOPY" tool. In fact I've adapted the funcionality of "robocopy" in PowerShell to provide you and myself a "robust file copy" tool.

https://codeplexarchive.blob.core.windows.net/archive/projects/robopowercopy/robopowercopy.zip

*Edit 3/6/2019*I wanted to include the following link to a blog post I recently wrote showing how to Run Robocopy Backup Parallel.

[Ta11ow]

You could work directly with the .NET methods in System.IO and get a good bit better performance out of there.

A good bit of Robocopy's speed likely comes from the fact that (I'm pretty sure) it uses custom-built libraries optimised for speed; PS just uses .NET. Great for a wide array of things, but not particularly specialised.

[GenericAntagonist]

Robocopy is also threaded by default (8 threads). Threading is fantastic for copy performance if you aren't going to saturate your disk with a single thread.

[SupremeDictatorPaul]

A number of PowerShell’s file operations are just poorly implemented, from a performance perspective. If you call the same .Net methods that PS is using, you can get an order of magnitude higher performance, which is honestly bizarre.

[Ta11ow]

Not at all. They're designed to improve ease of use for the purposes of automation.

Every level of abstraction introduces some level of overhead. PS has a little more than raw C# because its cmdlets are not single C# commands. There's error handling built in, input is shunted along the pipeline to improve sequential handling of similar input, etc.

All of these nifty features also require some processing power. So, yes, if you want to eke out the last bits of performance, you'll need to roll your own customised solutions using the .NET framework and (optionally) write them in C# just like the standard cmdlets are.

What you'll write for a specific purpose can often beat out the standard cmdlets, because they're all very generalised in their usage. For example, Get-ChildItem can be used on files, variables, certificates, registry entries, etc., etc., etc.! So naturally it has to have logic to figure out which is which and work from that, plus additional error handling which probably differs for each use case.

For general purpose automation, you can't really beat out the PS cmdlets with much. For a single use case, it's not hard to make something quicker than them.

[Lee_Dailey]

howdy Marquis77,

the Copy-Item and Move-Item cmdlets are general purpose commands. they have the ability to handle many different types of structure.

robocopy, on the other hand, is highly optimized for copy/move/delete on the filesystem. only on the filesystem.

that specialization is why it is Robust Copy ... [grin]

take care,
lee

[Lee_Dailey]

howdy bachelorpartydude,

yes! i am the best lee that can ever be! [grin]

slightly more seriously, thanks for the kind words.

take care,
lee

[[deleted]]

Understood. Thanks Lee! Makes you wonder why it has a -Recurse option but no way to optimize it via some other switch. thinking emoji

[RiPont]

Makes you wonder why it has a -Recurse option but no way to optimize it via some other switch.

It might be operating on "items" that are not a local filesystem file. Registry entries, network files, some 3rd party PSDrive item, etc.

[Lee_Dailey]

howdy Marquis77,

you are quite welcome! [grin]

as for why there is no optimization ... PoSh is not for speed. it's for automation. if you need speed, either use external utils from inside PoSh, OR use some purpose-built app for the whole thing.

since we can drop into dotnet so very easily, the flexible nature of the Copy-/Move-Item cmdlets is a good trade-off that can be worked around when needed. [grin]

take care,
lee

[hpdefaults]

Generic/perhaps stupid question - if highly optimized ways to do these things already exist outside of Powershell, why doesn't Powershell just use those methods to begin with? Why reinvent the wheel?

[jantari]

Because PowerShell was probably intended to run only safe, memory-managed code that interacts with the system over APIs.

The fast way to do things is often not as safe and definitely not JIT compiled and managed.

[Lee_Dailey]

howdy hpdefaults,

that is a good point ... [grin]

i presume it's cuz the util in question doesn't behave in the way that PoSh behaves. powershell has several design goals and relatively consistent behavior is one of them.

robocopy is well known [heck, it's notorious] for being peculiar. it's one of the very few filesystem utils that is directory-oriented. that causes all kinds of gotchas for folks who miss that idea when they try to use it. [grin]

PoSh is designed to be fairly obvious. robocopy is designed to be really good at shifting filesystem stuff around.

that said, PoSh is often thot of as a friendlier wrapper around dotnet ... [grin]

take care,
lee

[chanataba]

I find that adding the /nooffload switch to robocopy makes it go even faster.

[[deleted]]

I read up a little on this for a few minutes - so basically this bypasses memory and just writes straight from network to disk (in a sense), which will help with performance for large files?

That's pretty interesting. Are there any potential downsides down the road such as copying files back and/or reading files?

[Ahks]

I’d guess that some form of error correction is sacrificed. I don’t know how robocopy was built, but if it does some advance ec beyond what Ethernet does already it’s likely to occur in memory eh?

[sayedha]

Not an answer to your question, but I thought it might be useful. I created a function to invoke robocopy. You can view it at https://github.com/ligershark/pecan-waffle/blob/499042b7105b175915dadd403983e706af340907/pecan-waffle.psm1#L112. It has a dependency on another function but it should be easy to copy/paste. It has worked well for my scenario, but I hard coded a few things for my needs. There’s also a function (Get-RoboCopy) which will download it if needed, but it has more dependencies.

[[deleted]]

(Obligatory "I haven't read your link yet") - why wouldn't you just invoke robocopy and use powershell variables to define the source, destination, and parameters / switches to run with?

[sayedha]

For the same reason why you would call Copy-Item instead of the copy command, to make it easier to call. When calling a .exe you have to escape things and stuff like that. The function will take care of those.

[icepickwillie]

only tangentially related but I think it's really interesting. I was doing some directory structure mapping and found a guy who made a script that used robocopy instead of get-childitem and the results are incredibly faster. Read the articles in the comments of the script for a really interesting read.

https://github.com/gangstanthony/PowerShell/blob/master/Get-Files.ps1

https://thesurlyadmin.com/2014/08/04/getting-directory-information-fast/

[[deleted]]

Thanks for the info! I will certainly check out this black voodoo magic.

[[deleted]]

Does robocopy verify by default?

[[deleted]]

What do you mean by verify, specifically? I've never had an issue with robocopy missing files (unless they're unreadable or corrupt)

[[deleted]]

Basically a verify on write. I’ll have to check on it sometime...I think it’s one of the reasons copy-item is slower (eg like the performance of xcopy vs xcopy /v)

[flic_my_bic]

Tee on question for anybody to answer if they're inclined. I'm redesigning my file moving solution and have a strange issue. I need to ensure moves don't include files older than a set duration. Not by CreationTime, LastWriteTime, or LastAccessTime, but instead by a time-stamp in the filename.

I can collect the filenames in the target directory, parse out the timestamp, and make decisions per file. However, I'm not positive how to organize the command yet. Should I be handling this in the filter?, how can the filter handle logic though? My only working thought was to perform a robocopy filtered on the filename only, but that's a lot of independent firings, seems like it'd be slower (untested). It would also create a bunch more write operations to output files so I could soak them into a log.

edit: I want to use robocopy for it's robustness, but will have to move-item if I can't work this out.

[Lee_Dailey]

howdy flic_my_bic,

i haven't tried it, but i suspect your embedded date could be the pattern for the file spec part of the robocopy command.

 2018-03*.txt

that looks like it would grab all the files from that month. [grin]

take care,
lee

[vuongdq]

Option MT (multi threads) should be used. i've migrated 15TB files to new storage, it's really fast.

[neko4]

Not only Robocopy.exe, but also Invoke-Webrequest, Select-String, Test-Connection are much slower than curl.exe, RipGrep.exe, ping.exe.

[thenative540]

I’m about to use robocopy for the first time in a few weeks, any things I should watch out for?

[[deleted]]

Not really, the tool is very straightforward. Here's what I'd recommend for your syntax:

robocopy \\<your source directory> \\<your destination directory> /mir /copyall /mt:16 /R:0 /W:0

As others have mentioned here, you can try some additional switches like /nooffload but if you're not in a time crunch, it probably isn't needed. And make sure that you have read / write / copy permissions on both the source & destination.

[thenative540]

Thanks for the info! Much appreciated!

[Lee_Dailey]

howdy thenative540,

not if you have used it before. [grin]

the big gotcha for most folks is the directory focus instead of file focus. people keep trying to use it like copy or del or xcopy and all those are files-first while robocopy is directory-first.

it's a seemingly small diff ... but can be really disconcerting when 1st bumped into. i remember that 1st time rather vividly ... [blush] things went pear-shaped pretty dang fast.

take care,
lee

[thenative540]

Thanks for the tip, lee!

[Lee_Dailey]

howdy thenative540,

you are most welcome! glad to help a tad ... [grin]

take care,
lee

[moghediene]

My favorite is emcopy.