CIS 84.12B               Constructing Pipelines                  P.Grosh


A "pipe" (|) is a mechanism whereby the standard output of one command
is used as the standard input for the next command to be executed. The
resulting series of commands connected by pipes is called a pipeline:

command1 | command2 | command3   (options and filenames omitted)

The best way to construct a pipeline is one command at a time. You could, 
of course, just type the whole pipeline in at the prompt, cross your fingers,
and press Return. You could be lucky and the machine might do what you want.
This is often the case with a short pipeline, such as when you pipe the
voluminous output of one command to the "less" or the "more" command, thus:

ls -al | more

But with a more complex pipeline, it makes much more sense to construct
the pipeline 1 command at a time, adding a new command to the pipeline
only after you have verified that the previous command in the pipeline
does exactly what you want. For example, the best way to construct the
3-command pipeline above so that it actually WORKS, is to start with:
(you can use the "less" command instead of "more" for all that follows)

command1 | more

Then you can check that the output from command1 is exactly what you want.
If it is not, correct it (such as by adding options or using a different
command) and try again. It is usually futile to try to add new commands to
a pipeline before the existing commands do what you want. So it is only
when your first command works that you should add the second command:

command1 | command2 | more

In other words, a pipeline is built in a modular fashion, one piece at
a time. This is the most effective way to construct a pipeline so that 
it actually does what you want it to.

Often you may want a copy of some of the intermediate output produced by
your pipeline. The "tee" command can be used to do this:

command1 | tee outfile1 | command2 > outfile2

The file outfile1 will contain the data that is also sent as input to command2.
The file outfile2 will contain the final output of the pipeline. The "tee"
command duplicates its standard input and sends one copy to the standard output
and the other to one or more files. For those of you who are interested in such
things, you can also use "tee" to see the intermediate and final output on the
screen as well as storing them in files (the following is a direct consequence
of the fact that Unix regards everything including your screen as a file):

command1 | tee outfile1 /dev/tty | command2 | tee outfile2