Updated : May 21, 2016
Question : What is the fastest solution to count the number of files in a large network share ?
Answer : To answer this question, I will compare 4 different commands:
- [System.IO.Directory]::EnumerateFiles
- [System.IO.Directory]::GetFiles
- Get-ChildItem
- cmd /c dir
- robocopy (I need to include in this test, will be done later)
I will count all the files in a large network share (27.7 GB : 71 694 files and 8821 folders ).
Important : in this test there are no long paths, access denied, etc. I will write another article about how to deal with exceptions.
First, I just confirm that all these commands return the same result before measuring that.
|
1 2 3 4 5 6 7 8 9 |
[System.GC]::WaitForPendingFinalizers() [System.GC]::Collect() $path = 'T:\' $query01 = (Measure-Command -Expression { (([System.IO.Directory]::EnumerateFiles($path,'*.*','AllDirectories')) | Measure-Object).Count}).TotalHours $query02 = (Measure-Command -Expression {([System.IO.Directory]::GetFiles($path,'*.*','AllDirectories')).Count}).TotalHours $query03 = (Measure-Command -Expression {(Get-ChildItem $path -Recurse -File -Force).Count}).TotalHours $query04 = (Measure-Command -Expression {(cmd.exe /c dir $path /a:-d /s /b).Count}).TotalHours |
We can start the measure as we have the same value (71690 files).
Result:
Conclusion : In this scenario, the fastest was :
[System.IO.Directory]::EnumerateFiles($path,'*.*','AllDirectories')
Note 1: The EnumerateFiles and GetFiles methods differ as follows: When you use EnumerateFiles, you can start enumerating the collection of names before the whole collection is returned; when you use GetFiles, you must wait for the whole array of names to be returned before you can access the array. Therefore, when you are working with many files and directories, EnumerateFiles can be more efficient.
Note 2 : I recommend to use multithreading search when working with large amount of data to speed up. I will write another article about that.
Note 3 : If you have a faster solution, feel free to comment below so I can update my article.
I have received an email about someone asking me how to deal with exceptions.
Here it is a temporary solution (which needs some minor improvements) to parse directories dealing with exceptions.
It requires the module AlphaFS but I will start to work to create a function using FindFirstFile and FindNextFile.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
#requires -Version 2 function Get-EnumerateFile { Param ( [string]$Path ) function Invoke-GenericMethod { Param ( $Instance, [string]$MethodName, [type[]]$TypeParameters, [object[]]$MethodParameters ) [Collections.ArrayList]$private:parameterTypes = @{} foreach ($private:paramType In $MethodParameters) { [void]$parameterTypes.Add($paramType.GetType()) } $private:method = $instance.GetMethod($MethodName, 'Instance,Static,Public', $Null, $parameterTypes, $Null) if ($null -eq $method) { throw ('Method: [{0}] not found.' -f ($instance.ToString() + '.' + $MethodName)) } else { $method = $method.MakeGenericMethod($TypeParameters) $method.Invoke($instance, $MethodParameters) } } foreach ($private:folder In (Invoke-GenericMethod ` -Instance ([Alphaleonis.Win32.Filesystem.Directory]) ` -MethodName EnumerateFileSystemEntryInfos ` -TypeParameters Alphaleonis.Win32.Filesystem.FileSystemEntryInfo ` -MethodParameters $Path, '*', ([Alphaleonis.Win32.Filesystem.DirectoryEnumerationOptions]'Files, Recursive, SkipReparsePoints, ContinueOnException'), ([Alphaleonis.Win32.Filesystem.PathFormat]::FullPath))) { $folder.FullPath } } # Download : https://github.com/alphaleonis/AlphaFS/releases/latest # Be sure to select your .NET Framework version (net35, net40, net45 or net451) Import-Module -Name 'C:\Demo\AlphaFS.2.0.1\lib\net451\AlphaFS.dll' Get-EnumerateFile -Path 'C:\' | Measure-Object |
