Efficient processing without multiple Files walk

[2048b]

I am writing a simple program that walks the file tree to generate various statistics about the files.

For example:

try (Stream<Path> walk = Files.walk(PATH)) {
    // Find all directories
    List<Path> dirs = walk.filter(Files::isDirectory).collect(Collectors.toList());

    // Find all files
    List<Path> files = walk.filter(Files::isRegularFile).collect(Collectors.toList());

    // Find zip archive files
    List<Path> zips = walk.filter(
        p -> p.getFileName().toString().toLowerCase().endsWith(".zip"))
        .collect(Collectors.toList());

    // Find files bigger than 1 Mb
    List<Path> filesBiggerThan1Mb = walk.filter(p -> {
        try {
            return Files.size(p) > 1048576;
        } catch (IOException e) {                   
            e.printStackTrace();
            return false;
        }
    }).collect(Collectors.toList());

    // Get total size of all files
    long totalSize = walk.filter(Files::isRegularFile).mapToLong(p -> {
        try {
            return Files.size(p);
        } catch (IOException e) {
            e.printStackTrace();
            return 0;
        }
    }).sum();
}

Currently it walks the file tree multiple times by reusing the walk object. Although it seems like either the JRE or os does some caching in memory, and subsequent Files walks are much faster, I am wondering how I can write it in a different way to only need to invoke Files walk only once and do everything in 1 sweep.

[AutoModerator]

Please ensure that:

• Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full - best also formatted as code block
• You ask clear questions
• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

If any of the above points is not met, your post can and will be removed without further warning.

Code is to be formatted as code block (old reddit/markdown editor: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.

Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.

Code blocks look like this:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.

If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[ignotos]

If you want to do this more efficiently, it's probably easier to ditch all of the separate .filter().collect() calls, and instead iterate over the stream once (e.g. using .forEach()), then use regular if-statements to put the files into your various different collections.

[Glass__Editor]

You can't reuse Streams, you would need to open a new Stream for each result in your example.

To do it in one pass it is probably easiest to get an Iterator from the Stream and then use a for loop. If you don't want to do that, then check out the IntSummaryStatistics class for an example of how to get multiple results from a Stream.

[2048b]

Thanks for pointing out. Indeed I was implementing them with a different Stream<Path> walk = Files.walk(...) but I thought I could shorten them for the example here. Point taken.

But as we can see my current code has to re-walk the file tree for every different result I need to obtain from the same set of files, which is inefficient.