Redirection in Linux is a function that can change the standard input and output devices when executing commands in Linux. The basic flow of Linux commands is to receive input and produce output, but redirection techniques can be used to change the source or destination of standard input (stdin) or standard output (stdout). For example, you can save the output of a command to a file instead of displaying it on the screen, or you can use the contents of a file as input to a command instead of typing it from the keyboard.
There are three data flows involved in the execution of commands in Linux:
- Standard input (stdin) is the source of input data. By default, stdin is any text entered from the keyboard. Its stream ID is 0.
- Standard output (stdout) is the result of the command. By default, it is displayed on the screen. Its stream ID is 1.
- Standard error (stderr) is the error message (if any) produced by the command. By default, stderr is also displayed on the screen. Its stream ID is 2.
These streams contain data stored in buffer memory as plain text. These streams can be redirected using some special symbols or commands, for example:
- Output Redirection: Use the ">" or ">>" symbols to redirect stdout to a file. If the ">" symbol is used, the content of the original file will be overwritten; if the ">>" symbol is used, the new output will be appended to the end of the original file.
- Input Redirection: Use the "<" or "<<" symbols to redirect stdin to a file. If the "<" symbol is used, the command will read input from the specified file; if the "<<" symbol is used, the command will read multi-line input from the terminal ending with the specified delimiter.
- Error redirection: Use the "2>" or "2>>" notation to redirect stderr to a file. If the "2>" symbol is used, the content of the original file will be overwritten; if the "2>>" symbol is used, the new error message will be appended to the end of the original file.
What is a file descriptor?
A file descriptor is a non-negative integer associated with an open file on the system. It is important to note that a file descriptor is bound to a process and is unique per process. When opening a new file, the kernel returns the file descriptor associated with it. There are three standard file descriptors associated with your terminal.
When bash starts, it opens three file descriptors (fd) by default
- stdin (fd 0)
- stdout (fd 1)
- stderr (fd 2)
All three of these file descriptors point to your terminal (e.g. /dev/tty0), so input is read from your terminal, and output and errors are redirected to your terminal.
stdin (fd0) -----> /dev/tty0
stdout (fd1) -----> /dev/tty0
stderr (fd2) -----> /dev/tty0This is what the default redirect looks like. But these can be changed according to our needs. The best way to understand how redirects work is to visualize them.
What is redirection?
Redirection is the process of manipulating the default location to where the standard file descriptors (stdin, stdout, stderr) point to.
Redirect the output of the command to a file:
Syntax:
<command> n> file_name
Examples:
$ ls -la > list_file.txt
$ ls -la 1> list_file.txtThe above two commands in the example are exactly the same. If no file descriptor number is specified with the redirection symbol (>), it defaults to file descriptor 1 (the stdout file descriptor). See redirection visualization below
fd0 -----> /dev/tty0
fd1 -----> list_file.txt
fd2 -----> /dev/tty0
Redirect stdout and stdout to a file:
This is one of the commonly used scenarios where the command's stream, stdout and stderr output are all redirected to a file
$ command &> file
$ command >file 2&>1Here is how the file descriptor table changes when executing the command after the above line.
Original After the redirection
0 ------> /dev/tty0 0 ---------> /dev/tty
1 ------> /dev/tty0 1 ------|--> file
2 ------> /dev/tty0 2 ------| The order of redirection is important here. If the redirection order is swapped, only standard output ends up being redirected to the file. Try following the example above to visualize
Enter the contents of the file via the stdin of the command:
Redirect the contents of a file to the command's stdin. This scheme is typically used for read commands, where a file can be passed as input to read, rather than read from a terminal
read -n 5 < input_file.txt
0 -----> input_file.txt
1 -----> /dev/tty0
2 -----> /dev/tty0Redirect stderr of all commands executed to a file in a terminal session:
This command is useful if you want to redirect the output of all commands executed in a shell session to a specific file!
$ exec 2>file
$ test_command1
$ test_command2
How to open a new file descriptor for reading and writing?
Depending on your numbers, you can create a custom file descriptor and point it to the file, which can be used to read and write the file. This use case describes the use of the built-in bash command — exec. This builtin, if specified before redirection, will make them effective until you explicitly change the redirection or exit the shell or scriptexec
Open a file for reading with a custom file descriptor:
# point a file descriptor to file
$ exec 7<file_name
# Reads a line from the file pointed by filedescriptor 7
$ read -u 7 line
# close fd after reading
$ exec 7>&-
Write to a file using a custom file descriptor:
This is similar to writing something to stdout, except you'll have an extra step pointing a file descriptor to the file to write to.
$ exec 3<file_write
$ echo "Test message" > &3
$ echo 3>&-To open a file for reading and writing, you can use the following exec command
#Use file descriptor for both reading and writing - Filename: Test.txt
$ exec 3 <>Test.txtExamples of more bash one-liners:
Discard command output:
Redirecting to the special file /dev/null will avoid printing the output of the command to stdout. /dev/null is one of many virtual files used in Linux to write output. Anything written to /dev/null is forgotten and never printed
$ test_command < /dev/nullEnter the command from the terminal:
Files can be created and appended to in a single command by taking input from the terminal. Below is an example. The syntax of <<EOF below is called a heredoc.
$ cat > 'temp.txt' <<EOF
Always in love with bash and
all the things we could do with it.
EOFPrint standard output to terminal and file:
Tee isn't technically part of bash, but is used quite often. It copies the input stream to standard output and zero or more files. Believe me, this is very handy, especially when debugging.
command -----> tee -----> stdout
|--------> fileWhere are Pipes used?
Pipe is a redirection function in Linux, which can send the output of one command/program/process to another command/program/process as input, so as to realize the cooperation between commands. The symbol of Pipe is "|", which can be used to connect multiple commands to form a command pipeline. For example:
ls | grep "txt" | wc -l
The function of this command pipeline is to first list all the files in the current directory, then filter out the files containing "txt", and finally count the number of these files. The output of each command becomes the input for the next command until the output of the last command is displayed on the terminal.
The most common use case for pipe is to send the output of one command as the input of another command.
Syntax:
command1 | command2
Examples:
$ ls . > file.txt
$ wc -l file.txt
The above two commands are equivalent to this one-liner
$ ls . | wc -lThis one-liner can be used to send a command's stdout and stderr to another command's stdin.
command1 2>&1 | command2Redirect a single line of text as input to another command
echo “Test text” | test_command"Test text" now enter test_command.
This article describes the redirection and pipe functions in Linux, and how to use them to achieve cooperation between commands and control of data flow. Through some simple examples, we show how to use the test command to test the validity of commands or expressions, and how to check the type and permissions of files. I also covered some commonly used redirection and pipe operators, along with their meaning and usage. Hope this article can help you better understand and use redirection and pipe functions in Linux, improve your command line skills and efficiency.
Thank you for reading this article, if you find this article helpful to you, please give me a like or leave a comment. If you have any questions or suggestions, you are also welcome to contact me. I will do my best to answer your questions and keep improving my blog content. Thank you again for your support and attention, and wish you a happy study!