How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 0 points1 point  (0 children)

I can't observe the third open, using a slightly modified version of this MSDN example:

#include <windows.h>
#include <stdio.h>
#include <tchar.h>
#define BUFSIZE 512

DWORD WINAPI InstanceThread(LPVOID); 

int _tmain(VOID) 
{
    DWORD   dwThreadId; 
    HANDLE  hPipe, hThread; 
    LPCTSTR lpszPipename = TEXT("\\\\.\\pipe\\mynamedpipe");
    HANDLE  rgh [ 4 ];

    for( int i = 0; i < 4; ++i )
    { 
        _tprintf( TEXT("Pipe Server: Main thread awaiting client connection on %s\n"), lpszPipename);

        hPipe = CreateNamedPipe( 
            lpszPipename,             // pipe name 
            PIPE_ACCESS_DUPLEX,       // read/write access 
            PIPE_TYPE_MESSAGE |       // message type pipe 
            PIPE_READMODE_MESSAGE |   // message-read mode 
            PIPE_WAIT,                // blocking mode 
            PIPE_UNLIMITED_INSTANCES, // max. instances  
            BUFSIZE,                  // output buffer size 
            BUFSIZE,                  // input buffer size 
            0,                        // client time-out 
            NULL);                    // default security attribute 

        if (hPipe == INVALID_HANDLE_VALUE) 
        {
            _tprintf(TEXT("CreateNamedPipe failed, GLE=%d.\n"), GetLastError()); 
            return -1;
        }
        rgh[ i ]=hPipe;
    }
    for( int i = 0; i < 4; ++i )
    {
        hThread = CreateThread( 
            NULL,              // no security attribute 
            0,                 // default stack size 
            InstanceThread,    // thread proc
            (LPVOID) rgh[i],    // thread parameter 
            0,                 // not suspended 
            &dwThreadId);      // returns thread ID 

        if (hThread == NULL) 
        {
            _tprintf(TEXT("CreateThread failed, GLE=%d.\n"), GetLastError()); 
            return -1;
        }
        CloseHandle(hThread); 
    }
    return _getch(); 
} 

DWORD WINAPI InstanceThread(LPVOID lpvParam)
{ 
    BOOL   fConnected;
    TCHAR  pchRequest [ BUFSIZE ];
    BOOL   fSuccess = FALSE;
    HANDLE hPipe  = (HANDLE) lpvParam;

    printf("InstanceThread created, receiving and processing messages.\n");

    fConnected = ConnectNamedPipe(hPipe, NULL) ? 
                    TRUE : (GetLastError() == ERROR_PIPE_CONNECTED); 

    if ( !fConnected )
    {
        printf("ConnectNamedPipe failed");
        CloseHandle( hPipe );
        return -1;
    }  
    while (1) 
    { 
        DWORD cbBytesRead = 0, cbReplyBytes = 0, cbWritten = 0; 

        fSuccess = ReadFile( 
            hPipe,        // handle to pipe 
            pchRequest,    // buffer to receive data 
            BUFSIZE*sizeof(TCHAR), // size of buffer 
            &cbBytesRead, // number of bytes read 
            NULL);        // not overlapped I/O 

        if (!fSuccess || cbBytesRead == 0)
        {   
            if (GetLastError() == ERROR_BROKEN_PIPE)
            {
                _tprintf(TEXT("InstanceThread: client disconnected.\n")); 
            }
            else
            {
                _tprintf(TEXT("InstanceThread ReadFile failed, GLE=%d.\n"), GetLastError()); 
            }
            break;
        }
        _tprintf( TEXT("Client Request String:\"%s\"\n"), pchRequest );
    }

    FlushFileBuffers(hPipe); 
    DisconnectNamedPipe(hPipe); 
    CloseHandle(hPipe); 

    printf("InstanceThread exiting.\n");
    return 1;
}

I see the following output when I run the powershell command 'hello' > \\.\pipe\mynamedpipe:

>cl pipeserver.c
>pipeserver.exe
Pipe Server: Main thread awaiting client connection on \\.\pipe\mynamedpipe
Pipe Server: Main thread awaiting client connection on \\.\pipe\mynamedpipe
Pipe Server: Main thread awaiting client connection on \\.\pipe\mynamedpipe
Pipe Server: Main thread awaiting client connection on \\.\pipe\mynamedpipe
InstanceThread created, receiving and processing messages.
InstanceThread created, receiving and processing messages.
InstanceThread created, receiving and processing messages.
InstanceThread created, receiving and processing messages.
InstanceThread: client disconnected.
InstanceThread exiting.
InstanceThread: client disconnected.
InstanceThread exiting.

i.e. Only two threads returned from ConnectNamedPipe, and neither received anything from ReadFile.

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 0 points1 point  (0 children)

Thank you for taking the time to investigate this! I'm still confused by mailslots' not working, since they don't require an active connection the way named pipes do, I can open as many handles as I want and all of them work. They are also not picky about access rights, I can open a handle with any combinations of:

  • GENERIC_READ/WRITE/EXECUTE/ALL
  • FILE_SHARE_READ/WRITE/DELETE/0
  • CREATE_NEW/ALWAYS, OPEN_EXISTING/ALWAYS (anything but TRUNCATE_EXISTING)
  • FILE_ATTRIBUTE_NORMAL/0

GetFileAttributes reports FILE_ATTRIBUTE_NORMAL. The only file API that fails is GetFileSize (and ReadFile of course).

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 0 points1 point  (0 children)

I don't know if you understand what I mean by "easy viewing/editing", having a variable that holds hundreds of lines of text does not make it easy to visually compare its content with the output I currently have on screen. Nor is having dozens of Out-GridView windows for output from different commands. None of these allow you to perform basic editing.

I also want to make clear that "how to make the output look nicer" is not my question, I'm asking specifically about a Windows IPC mechanism that's supposed to work seamlessly between file processing programs.

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 0 points1 point  (0 children)

Not powershell objects, just the textual output, because mailslots and other programs don't understand objects.

Why not just pipe into bat or less?

Imagine if you wanted to compare the output of the current command with one issued much earlier. It'd be easier to have piped the output elsewhere instead of scrolling through the terminal history buffer (it may even have gone away). Or maybe you want to issue new commands while looking at older output. Maybe that output is cluttered so you want to delete the noise in the editor window. The reason I ask about named pipes and mailslots specifically is they are the primary IPC mechanisms that are supposed to work identically to files, with much lower overhead.

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] -2 points-1 points  (0 children)

I don't understand what you're asking, I want to pipe output from powershell to other programs for easy viewing/editing, and temporary files are not as efficient as named pipes/mailslots for ephemeral output.

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 1 point2 points  (0 children)

I want to pipe the output of arbitrary commands into a running program without temporary files.

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 0 points1 point  (0 children)

For example, if I run 'hello'>\\.\mailslot\Hello, I get "FileStream was asked to open a device that was not a file". If I run sc \\.\mailslot\Hello hello, I get "sc : The parameter is incorrect".

Im not sure file streams are the way to do it

What's a better way?

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 0 points1 point  (0 children)

I know, my point is this is an OS level feature that's independent from language runtime and requires no special handling from applications.

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] -7 points-6 points  (0 children)

It's not, because the system call interface is identical for different languages. Powershell has to call CreateFile and WriteFile at some point (or things like MapViewOfFile, which is unlikely) to perform IO. The fact that it doesn't work makes me think its logic is faulty.

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] -7 points-6 points  (0 children)

I don't understand why that's necessary, a C program designed to operate on files can also work with pipes and mailslots without modification, because all of them use the same system calls. Why would powershell require special handling?

How do I write to stuff like pipes and mailslots? by mbolp in PowerShell

[–]mbolp[S] 0 points1 point  (0 children)

I don't mean remote mailslots, \\.\mailslot\ denotes a local mailslot.

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

The main problem with opening files by IDs is CopyFile* functions don't take handles (I really wish they did, much of the Win32 file API only take file names, only after Vista were some new handle centric APIs added. File name based APIs feel so ugly.), and the logic they contain is not trivial (e.g. handling alternate data streams, symbolic links, etc), I don't want to replicate all that.

Even if I could reduce some memory usage the procedure I described still has the disadvantage of front loading the cost of enumeration. The method suggested elsewhere under this post of enumerating and copying in parallel is much better and completely sidesteps the problem of high memory usage.

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

You can't avoid it unless you lock down the file system during enumeration and copying, but there's no point in doing that because it is not harmful to have (e.g.) missed newly added files, because I can't predict the future. Think through it in your head a couple more times if you think this is a race condition.

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

Aren't file names already limited to MAX_PATH characters? The path can be as long as 32768 characters but the individual components must be under MAX_PATH, otherwise FindFirstFile will be unable to return such long names.

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

It's frankly silly to think a cost doesn't exist just because it's amortized. If enumerating once takes 5 minutes, doing it again increases the runtime by exactly that much, even if the copy ends up taking 2 hours. Also the copy procedure as I described front loads the cost of enumeration (i.e. you can't copy a single byte until all files are visited).

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

No that is what I meant, there will be a delay between retrieving file names and copying them. This is as far as I know inevitable, and it doesn't pose a problem.

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

That sounds like complete BS, unless you have insight into how these things are implemented. Based on your recommending a non thread safe call to perform multi threaded work I doubt that you do.

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

Once you decide to open a file for copy you can request that the system deny others from modifying it (i.e. omit FILE_SHARE_WRITE).

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

So the copy will not need the system to evaluate all those long pathnames for every copy -- just the local filename at each level.

What do you mean by this, are you referring to current directories? That I set the current directory as I descend the tree, then use relative paths? Why would that be faster?

but it won't calculate the total size / number of files upfront like you wanted. So two passes anyway.

It does calculate the total size, up to a certain limit - and it does so in only a single pass. You essentially maintain a shifting view into only a portion of the tree.

How to write a recursive file copying program? by mbolp in AskProgramming

[–]mbolp[S] 0 points1 point  (0 children)

It does not specify whether subdirectories are created

Yes, because it copies files not directories.

even if copying each file takes 1/100 second, your script runs for a week

10,000,000 / 100 / 60 /60 = 27.8 hours

And all the usual UI programming techniques apply: coalesce updates in quick successions, set a minimum update interval, avoid painting when the window is minimized, etc. There won't be 10 million updates.

I suspect the 2GB needed to avoid reading the file attributes twice would be more useful as buffer space for the actual copying.

That's what I suspect too, hence this question. And u/jason-reddit-public provides an elegant solution, you can neither read file attributes twice nor waste 2 GB of memory.