The Brilliant Concept of Standard Streams in Unix (1)

1. The basic idea of standard streams

In the 1980s, almost every book or class on computer science here in Germany started with an explanation of the EVA-Prinzip, where E stood for Eingabe (Input), V for Verarbeitung (Processing), and A for Ausgabe (Output). In the English speaking world the EVA-Prinzip is known as the IPO (input-process-output) model.

The model describes any computer system as a black box that receives some input, performs some computations on it, and spits out the results of the computations. As simple as this model may be, it does not only apply computer systems as a whole, but also to its components. Any process, any program, any piece of software such as functions or methods can be considered to create output information based on certain input information.

Depending on the purpose of the program, the source of its input could be your computer’s keyboard, a file that resides on your hard disk, or even a movie stream provided by a remote server. The output generated by the program may be some text printed on a computer screen or printer, or saved in a file.

From a user’s perspective it is desirable a program works with any input source and any output target. From a programmer’s point of view, however, supporting a large variety of possible input sources and output targets can be a painful task.

Most operating systems provide so-called standard streams for data input and output to the programmer. In Unix-like operating systems, such as Linux, standard streams are not merely meant to help the programmer be happy and productive. The concept of streams is rather a philosophy that, together with the idea of “everything is a file”, improves the lives of both programmers and users.

2. How to use standard streams on the command line

Unix-like systems implement three character-based standard streams that have numeric IDs ranging from 0 to 2.

By default, the Standard Input is read from your keyboard. Both Standard Output and Standard Error are written to your screen. For a better understanding on how this works, let’s take a look at a Linux command named cat. With no options provided, all it does is forwarding all information from Standard Input to Standard Output. So any line entered (and terminated with the <ENTER> key) will show up on the screen. An End of File (EOF) marker will cause the cat command to terminate. You can send the EOF by pressing CTRL-D.

maierm@abacus:~$ cat

I must admit there may be no reason at all why anyone would want to use cat in such a way. However, Linux and Unix command line interpreters provide specific operators that allow you to redirect streams to files, or to pass them on to another command or program.

The < operator allows you to redirect Standard Input from a file instead of the keyboard.

maierm@abacus:~$ cat < note.txt
Book a table for two at Ladybird, April 4th, 8 p.m.

The > operator allows you to redirect Standard Output into a file instead of the screen. So the following command line will copy file1 to file2.

maierm@abacus:~$ cat < file1 > file2

As streams are based on characters (bytes), the above command line even works for binary files. Note that if file2 already exists, its contents will be overwritten. If you want to append Standard Output to a file, use the >> operator, as shown in the following example.

maierm@abacus:~$ cat < file1 >> file2

By specifying the numeric ID you can also redirect and append the Standard Error stream to a file. The following two command lines will not generate any output on the screen, because the error messages caused by wrong usage are redirected to a file named error.log.

maierm@abacus:~$ rm 2> error.log
maierm@abacus:~$ cp 2>> error.log

Besides redirecting Standard Output to a file, you can pass it on to another command using the | (pronounced “pipe”) operator. Note that Micro$oft implemented this feature in their PowerShell a mere 34 years after pipes became an inherent part of Unix-like systems. 😉

On the command line, all you need to do to get the pipeline working is using the | symbol between subsequent commands. Here is an example:

maierm@turing:~$ ls | sort -r | nl

As you may know, we normally use the ls command to display the contents of the current directory on our screen. In the example above, however, Standard Output is not shown on the display, but forwarded to Standard Input of the sort -r command instead. This command sorts the list of files recursively and then passes it on to the nl command. nl prepends line numbers to each line and, as there is no further pipe or redirection, shows the result on the screen.

You may now wonder if it is possible to swap streams, how it is done, and how streams look like from a programmer’s point of view. This, however, will be covered in a future posting on streams.

Take care,

— Andre M. Maier


About bitjunkie

Teacher, Lecturer, and BITJUNKIE ...
This entry was posted in Linux, Uncategorized and tagged , , , , , . Bookmark the permalink.