In the shell, pipes allow us to redirect the output of one command into the input of another. In practice, this is usually implemented in three broad steps:
- Fork twice to run both commands inside child Processes.
- The first command writes to some buffer as output.
- The second command then reads from that buffer as input.
If the two commands are run in completely different processes, how do we pass data between them? It turns out that a syscall named pipe()
creates that buffer for us.
Example
The pipe
|
inls | wc -l
is implemented with a call topipe()
. In other words, the shell pipe uses the underlyingpipe()
system call.
System call
int pipe(int pipefd[2]);
This syscall allows two different processes to communicate with each other, with one process writing a message, and another reading it. It is unidirectional in the sense that only one process should be reading, and only one process should be writing.
In particular, it creates two new entries in the System wide file table, one exclusively for reading, and one exclusively for writing. A pipe isn’t actually a real file on your system; instead, the kernel allocates and manages a buffer.
int pipe_fds[2];
// Now pipe_fds[0] contains read, pipe_fds[1] contains write
pipe(pipe_fds);
Buffer has a size limit
If we
write()
too much without reading, the buffer will become full, and we won’t be able to write to it anymore. We will then have toread()
some data first to free up the buffer.
Reading and EOF
When using a pipe, if there’s nothing in the buffer to
read()
, then the process will block. Forever. Until we write something to it.Only when EOF is sent will
read()
know that it should stop. However, EOF is only sent when all write ends of the pipe are closed. This means if we’re not using the write side of a pipe in a process, we should immediately close it (before we forget to)!
Including flags
int pipe2(int pipefd[2], int flags);
The only difference is that this accepts additional flags. In particular, the flag O_CLOEXEC
automatically closes all file descriptors for us when we call some exec*()
function!