Problem with C pipes [reprise]

[gsayers]

Hi Paul,
Thanks for your quick response to my first post, and sorry for the confusion over examples I gave.
Appears my edit of this topic got swallowed up somewhere:

Here is the abridged version:

I am trying to pipe the output of one aescrypt decryption to the input of an aescrypt encryption.

This works fine as a DOS batch command on windows7 thus:

Code: Select all

aescrypt -d -p password1 -o - infilename | aescrypt -e -p password2 -o outfilename -

When I try this using a pipe to pipe routine in c (MinGW32 gcc 4.6.1) I get the following messages:

Code: Select all

Error writing decrypted block:: Invalid argument

I am guessing that for some reason, the aescrypt on the decryption side of the pipe appears not to have access to STDOUT. However, the "ls" command piped to "SORT" does work through the same C function.

I have tried every combination of syntax and buffer sizes I can think of, but still get the same result.

These are the values of the two variables I pass to pipe2pipe:

Code: Select all

cmd1="aescrypt -d -p password1 -o - infilename"
cmd2="aescrypt -e -p password2 -o outfilename -"

And here is the function I am using:

Code: Select all

#include <stdio>
#define MAX_PIPE_BUFF 128

int pipe2pipe(char *cmd1, char *cmd2)
{
// -----------------------------------------------------------------------------
// Open two pipes allowing output from one command to be piped to the other
// -----------------------------------------------------------------------------
        FILE *pipein_fp, *pipeout_fp;
        char readbuf[MAX_PIPE_BUFF];
        long bytes;
   
        // pipe IN
        if (( pipein_fp = popen(cmd1, "r")) == NULL)
        {
                perror("popen");
                return(1);
        }
        // pipe OUT
        if (( pipeout_fp = popen(cmd2, "w")) == NULL)
        {
                perror("popen");
                return(1);
        }

        while( (bytes=fread(readbuf,sizeof(readbuf),1,pipein_fp))>0) 
                 fwrite(readbuf,bytes,1,pipeout_fp);
              
        // Clean up
        pclose(pipein_fp);
        pclose(pipeout_fp);

        return(0);
}

Any further light you can shed on this would be appreciated.

Regards,
Gary

[paulej]

The code that would generate that message is in the routine writes out the decrypted data. So, I guess the problem has something to do with writing to the output file on decryption. Here's the code:

Code: Select all

if ((i = (int) fwrite(tail, 1, n, outfp)) != n)
{
    perror("Error writing decrypted block:");
    return  -1;
}

It's apparently reading the input file OK.

Is there any output created at all? I'm wondering if this line of code is the problem:

Code: Select all

while( (bytes=fread(readbuf,sizeof(readbuf),1,pipein_fp))>0) 
             fwrite(readbuf,bytes,1,pipeout_fp);

I wonder if the file is read to the end, but then one more read attempt is made and, rather than returning 0 octets, it's returning an error for some reason. Can you put some debug statements in to show how many octets are read inside that while loop? If you're reading through the file and hitting the end, perhaps an explicit call to feof() is needed?

Another possibility is that the - character is being mangled for some reason. But, I'm not sure why that would happen.

In your environment, do you need to set the mode to be "r" or "rb"?

[gsayers]

Brilliant Paul!
It was the "b"inary switch in read and write that was causing the hitch - can't think how I missed that - I hit the same problem a few months ago on a file read and fixed it myself, so I'm a bit embarrassed.
The code had a few more problems in it that I fixed really quickly once binary mode was on, so I thought I'd post it, just in case someone wants to do the same thing.. Note: I have not played with performance regarding differing pipe buffer sizes yet, so the current value may not be optimum. I note that in your code your buffers are quite small, so multiples of that size is probably the way to go.

Thanks again for providing such a valuable tool to the community.

Code: Select all

#include <stdio>
#define MAX_PIPE_BUFF 512

int pipe2pipe(char *cmd1, char *cmd2)
{
// -----------------------------------------------------------------------------
// Open two pipes allowing output from one command to be piped to the other
// -----------------------------------------------------------------------------
        FILE *pipein_fp, *pipeout_fp;
        char readbuf[MAX_PIPE_BUFF];
        long bytes;
   
        // pipe IN
        if (( pipein_fp = popen(cmd1, "rb")) == NULL)
        {
                perror("popen");
                return(1);
        }
        // pipe OUT
        if (( pipeout_fp = popen(cmd2, "wb")) == NULL)
        {
                perror("popen");
                return(1);
        }

        while( (bytes=fread(readbuf,1,MAX_PIPE_BUFF,pipein_fp))>0) 
                 fwrite(readbuf,bytes,1,pipeout_fp);
              
        // Clean up
        pclose(pipein_fp);
        pclose(pipeout_fp);

        return(0);
}

[paulej]

Great! I thought this might be a bear to fix. I noted the "r" and wondered if "b" should be there. I've actually seen it both ways: some environments require a "b" for binary and some do not. I have no idea what the reasoning is, but I've been bitten by that a few times in my life, so I do take note of it. I saw that the first time, but didn't question it since, as I said, some environments only need "r". I write a lot of Perl scripts for things, too, and have the same issue. On Windows, I have to call binmode() and I do not on Linux.

You asked about buffer sizes. The Linux AES Crypt code makes calls to fopen() to read and write data. fopen() is a buffered I/O interface, so the operating system creates a larger buffer to optimize reads and writes to disk. Thus, I don't have to worry about that.

The Windows GUI code calls the CreateFile() API to create files. I'm not sure exactly why I chose that API call now, but it is not buffered. For that reason, I created a C++ class called BufferedFile that essentially wraps the CreateFile() and related API calls and buffers all reads and writes to the disk using a buffer that is 8192 octets long.

In the Windows console code, I used _wfopen(). I assume that is buffered, since it's just a version of fopen() that accepts wide character file names.

One really should not notice a performance issue, though, until reading and writing over a network. If the performance over a network connection (e.g., to a NAS server inside the office) is good, then you should not need to change the buffering approach.