What are the shell's control and redirection operators?
These are called shell operators and yes, there are more of them. I will give a brief overview of the most common among the two major classes, control operators and redirection operators, and how they work with respect to the bash shell.
A. Control operators
POSIX definition
In the shell command language, a token that performs a control function.
It is one of the following symbols:& && ( ) ; ;; <newline> | ||
And |&
in bash.
A !
is not a control operator but a Reserved Word. It becomes a logical NOT [negation operator] inside Arithmetic Expressions and inside test constructs (while still requiring an space delimiter).
A.1 List terminators
;
: Will run one command after another has finished, irrespective of the outcome of the first.command1 ; command2
First
command1
is run, in the foreground, and once it has finished,command2
will be run.A newline that isn't in a string literal or after certain keywords is not equivalent to the semicolon operator. A list of
;
delimited simple commands is still a list - as in the shell's parser must still continue to read in the simple commands that follow a;
delimited simple command before executing, whereas a newline can delimit an entire command list - or list of lists. The difference is subtle, but complicated: given the shell has no previous imperative for reading in data following a newline, the newline marks a point where the shell can begin to evaluate the simple commands it has already read in, whereas a;
semi-colon does not.&
: This will run a command in the background, allowing you to continue working in the same shell.command1 & command2
Here,
command1
is launched in the background andcommand2
starts running in the foreground immediately, without waiting forcommand1
to exit.A newline after
command1
is optional.
A.2 Logical operators
&&
: Used to build AND lists, it allows you to run one command only if another exited successfully.command1 && command2
Here,
command2
will run aftercommand1
has finished and only ifcommand1
was successful (if its exit code was 0). Both commands are run in the foreground.This command can also be written
if command1 then command2 else false fi
or simply
if command1; then command2; fi
if the return status is ignored.||
: Used to build OR lists, it allows you to run one command only if another exited unsuccessfully.command1 || command2
Here,
command2
will only run ifcommand1
failed (if it returned an exit status other than 0). Both commands are run in the foreground.This command can also be written
if command1 then true else command2 fi
or in a shorter way
if ! command1; then command2; fi
.Note that
&&
and||
are left-associative; see Precedence of the shell logical operators &&, || for more information.!
: This is a reserved word which acts as the “not” operator (but must have a delimiter), used to negate the return status of a command — return 0 if the command returns a nonzero status, return 1 if it returns the status 0. Also a logical NOT for thetest
utility.! command1 [ ! a = a ]
And a true NOT operator inside Arithmetic Expressions:
$ echo $((!0)) $((!23)) 1 0
A.3 Pipe operator
|
: The pipe operator, it passes the output of one command as input to another. A command built from the pipe operator is called a pipeline.command1 | command2
Any output printed by
command1
is passed as input tocommand2
.|&
: This is a shorthand for2>&1 |
in bash and zsh. It passes both standard output and standard error of one command as input to another.command1 |& command2
A.4 Other list punctuation
;;
is used solely to mark the end of a case statement. Ksh, bash and zsh also support ;&
to fall through to the next case and ;;&
(not in ATT ksh) to go on and test subsequent cases.
(
and )
are used to group commands and launch them in a subshell. {
and }
also group commands, but do not launch them in a subshell. See this answer for a discussion of the various types of parentheses, brackets and braces in shell syntax.
B. Redirection Operators
POSIX definition of Redirection Operator
In the shell command language, a token that performs a redirection function. It is one of the following symbols:
< > >| << >> <& >& <<- <>
These allow you to control the input and output of your commands. They can appear anywhere within a simple command or may follow a command. Redirections are processed in the order they appear, from left to right.
<
: Gives input to a command.command < file.txt
The above will execute
command
on the contents offile.txt
.<>
: same as above, but the file is open in read+write mode instead of read-only:command <> file.txt
If the file doesn't exist, it will be created.
That operator is rarely used because commands generally only read from their stdin, though it can come handy in a number of specific situations.
>
: Directs the output of a command into a file.command > out.txt
The above will save the output of
command
asout.txt
. If the file exists, its contents will be overwritten and if it does not exist it will be created.This operator is also often used to choose whether something should be printed to standard error or standard output:
command >out.txt 2>error.txt
In the example above,
>
will redirect standard output and2>
redirects standard error. Output can also be redirected using1>
but, since this is the default, the1
is usually omitted and it's written simply as>
.So, to run
command
onfile.txt
and save its output inout.txt
and any error messages inerror.txt
you would run:command < file.txt > out.txt 2> error.txt
>|
: Does the same as>
, but will overwrite the target, even if the shell has been configured to refuse overwriting (withset -C
orset -o noclobber
).command >| out.txt
If
out.txt
exists, the output ofcommand
will replace its content. If it does not exist it will be created.>>
: Does the same as>
, except that if the target file exists, the new data are appended.command >> out.txt
If
out.txt
exists, the output ofcommand
will be appended to it, after whatever is already in it. If it does not exist it will be created.>&
: (per POSIX spec) when surrounded by digits (1>&2
) or-
on the right side (1>&-
) either redirects only one file descriptor or closes it (>&-
).A
>&
followed by a file descriptor number is a portable way to redirect a file descriptor, and>&-
is a portable way to close a file descriptor.If the right side of this redirection is a file please read the next entry.
>&
,&>
,>>&
and&>>
: (read above also) Redirect both standard error and standard output, replacing or appending, respectively.command &> out.txt
Both standard error and standard output of
command
will be saved inout.txt
, overwriting its contents or creating it if it doesn't exist.command &>> out.txt
As above, except that if
out.txt
exists, the output and error ofcommand
will be appended to it.The
&>
variant originates inbash
, while the>&
variant comes from csh (decades earlier). They both conflict with other POSIX shell operators and should not be used in portablesh
scripts.<<
: A here document. It is often used to print multi-line strings.command << WORD Text WORD
Here,
command
will take everything until it finds the next occurrence ofWORD
,Text
in the example above, as input . WhileWORD
is oftenEoF
or variations thereof, it can be any alphanumeric (and not only) string you like. WhenWORD
is quoted, the text in the here document is treated literally and no expansions are performed (on variables for example). If it is unquoted, variables will be expanded. For more details, see the bash manual.If you want to pipe the output of
command << WORD ... WORD
directly into another command or commands, you have to put the pipe on the same line as<< WORD
, you can't put it after the terminating WORD or on the line following. For example:command << WORD | command2 | command3... Text WORD
<<<
: Here strings, similar to here documents, but intended for a single line. These exist only in the Unix port or rc (where it originated), zsh, some implementations of ksh, yash and bash.command <<< WORD
Whatever is given as
WORD
is expanded and its value is passed as input tocommand
. This is often used to pass the content of variables as input to a command. For example:$ foo="bar" $ sed 's/a/A/' <<< "$foo" bAr # as a short-cut for the standard: $ printf '%s\n' "$foo" | sed 's/a/A/' bAr # or sed 's/a/A/' << EOF $foo EOF
A few other operators (>&-
, x>&y
x<&y
) can be used to close or duplicate file descriptors. For details on them, please see the relevant section of your shell's manual (here for instance for bash).
That only covers the most common operators of Bourne-like shells. Some shells have a few additional redirection operators of their own.
Ksh, bash and zsh also have constructs <(…)
, >(…)
and =(…)
(that latter one in zsh
only). These are not redirections, but process substitution.
Warning regarding ‘>’
Unix beginners who have just learned about I/O redirection (<
and >
)
often try things like
command … input_file > the_same_file
or
command … < file > the_same_file
or, almost equivalently,
cat file | command … > the_same_file
(grep
, sed
, cut
, sort
, and spell
are examples of commands
that people are tempted to use in constructs like these.)
Users are surprised to discover that these scenarios result in the file becoming empty.
A nuance that doesn’t seem to be mentioned in the other answer can be found lurking in the first sentence of the Redirection section of bash(1):
Before a command is executed, its input and output may be redirected using a special notation interpreted by the shell.
The first five words should be bold, italic, underlined, enlarged, blinking, colored red, and marked with a icon, to emphasize the fact that the shell performs the requested redirection(s) before the command is executed. And remember also
Redirection of output causes the file … to be opened for writing …. If the file does not exist it is created; if it does exist it is truncated to zero size.
So, in this example:
sort roster > roster
the shell opens the
roster
file for writing, truncating it (i.e., discarding all its contents), before thesort
program starts running. Naturally, nothing can be done to recover the data.One might naïvely expect that
tr "[:upper:]" "[:lower:]" < poem > poem
might be better. Because the shell handles redirections from left to right, it opens
poem
for reading (fortr
’s standard input) before it opens it for writing (for standard output). But it doesn’t help. Even though this sequence of operations yields two file handles, they both point to the same file. When the shell opens the file for reading, the contents are still there, but they still get clobbered before the program is executed.
So, what to do about it?
Solutions include:
Check whether the program you’re running has its own, internal, capability to specify where the output goes. This is often indicated by a
-o
(or--output=
) token. In particular,sort -o roster roster
is roughly equivalent to
sort roster > roster
except, in the first case, the
sort
program opens the output file. And it’s smart enough not to open the output file until after it has read all of the input file(s).Similarly, at least some versions of
sed
have a-i
(edit in place) option that can be used to write the output back out to the input file (again, after all the input have been read). Editors likeed
/ex
,emacs
,pico
, andvi
/vim
allow the user to edit a text file and save the edited text in the original file. Note thated
(at least) can be used non-interactively.vi
has a related feature. If you type:%!command
Enter, it will write the contents of the edit buffer out tocommand
, read the output, and insert it into the buffer (replacing the original contents).
Simple but effective:
command … input_file > temp_file && mv temp_file input_file
This has the drawback that, if
input_file
is a link, it will (probably) be replaced by a separate file. Also, the new file will be owned by you, with default protections. In particular, this carries the risk that the file will be end up being world-readable, even if the originalinput_file
wasn’t.Variations:
command … input_file > temp_file && cp temp_file input_file && rm temp_file
which will still (potentially) leave thetemp_file
world-readable. Even better:cp input_file temp_file && command … temp_file > input_file && rm temp_file
These preserve the link status, owner, and mode (protection) of the file, potentially at the cost of twice as much I/O. (You may need to use an option like-a
or-p
oncp
to tell it to preserve attributes.)command … input_file > temp_file &&
cp --attributes-only --preserve=all input_file temp_file &&
mv temp_file input_file
(broken into separate lines only for readability) This preserves the mode of the file (and, if you’re root, the owner), but makes it owned by you (if you’re not root), and makes it a new, separate file.
This blog (“In-place” editing of files) suggests and explains
{ rm input_file && command … > input_file; } < input_file
This requires that the
command
be able to process standard input (but almost all filters can). The blog itself calls this a risky kludge and discourages its use. And this will also create a new, separate file (not linked to anything), owned by you and with default permissions.The moreutils package has a command called
sponge
:command … input_file | sponge the_same_file
See this answer for more information.
Here’s something that came as a complete surprise to me: syntaxerror says:
[Most of these solutions] will fail on a read-only file system, where “read-only” means that your
$HOME
will be writable, but/tmp
will be read-only (by default). For instance, if you have Ubuntu, and you’ve booted into the Recovery Console, this is commonly the case. Also, the here-document operator<<<
will not work there either, as it requires/tmp
to be read/write because it will write a temporary file into there as well.
(cf. this question includes anstrace
’d output)
The following may work in that case:
- For advanced users only:
If your command is guaranteed to produce the same amount of output data
as there is input (e.g.,
sort
, ortr
without the-d
or-s
option), you can trycommand … input_file | dd of=the_same_file conv=notrunc
See this answer and this answer for more information, including an explanation of the above, and alternatives that work if your command is guaranteed to produce the same amount of output data as there is input or less (e.g.,grep
, orcut
). These answers have the advantage that they do not require any free space (or they require very little). The answers above of the formcommand … input_file > temp_file && …
clearly require that there be enough free space for the system to be able to hold the entire input (old) file and output (new) file simultaneously; this is non-obviously true for most of the other solutions (e.g.,sed -i
andsponge
) as well. Exception:sort … | dd …
will probably require lots of free space, becausesort
needs to read all of its input before it can write any output, and it probably buffers most if not all of that data in a temporary file. - For advanced users only:
command … input_file 1<> the_same_file
may be equivalent to thedd
answer, above. Then<> file
syntax opens the named file on file descriptorn
for both input and output, without truncating it – sort of a combination ofn<
andn>
. Note: Some programs (e.g.,cat
andgrep
) may refuse to run in this scenario because they can detect that the input and the output are the same file. See this answer for a discussion of the above, and a script that makes this answer work if your command is guaranteed to produce the same amount of output data as there is input or less.
Warning: I haven’t tested Peter’s script, so I don’t vouch for it.
So, what was the question?
This has been a popular topic on U&L; it is addressed in the following questions:
- Is there a way to modify a file in-place?
- How can I make
iconv
replace the input file with the converted output? - Why does the command
shuf file > file
leave an empty file? - Can I read and write to the same file in Linux without overwriting it?
- Redirect to the same file as the source file processed by the command
- Why does this
sort
command give me an empty file? - Redirecting
tr
stdout to a file - grep: input file 'X' is also the output
- Do redirection operators open file descriptors in parallel?
- Redirection not overwriting file but just producing a blank one
… and that’s not counting Super User or Ask Ubuntu. I have incorporated a lot of the information from the answers to the above questions here in this answer, but not all. (I.e., for more information, read the above-listed questions and their answers.)
P.S. I have no affiliation with the blog that I cited, above.
More observations on ;
, &
, (
and )
Note that some of the commands in terdon’s answer may be null. For example, you can say
command1 ;
(with no
command2
). This is equivalent tocommand1
(i.e., it simply runs
command1
in the foreground and waits for it to complete. Comparably,command1 &
(with no
command2
) will launchcommand1
in the background and then issue another shell prompt immediately.By contrast,
command1 &&
,command1 ||
, andcommand1 |
don’t make any sense. If you type one of these, the shell will (probably) assume that the command is continued onto another line. It will display the secondary (continuation) shell prompt, which is normally set to>
, and keep on reading. In a shell script, it will just read the next line and append it to what it has already read. (Beware: this might not be what you want to happen.)Note: some versions of some shells may treat such incomplete commands as errors. In such cases (or, in fact, in any case where you have a long command), you can put a backslash (
\
) at the end of a line to tell the shell to continue reading the command on another line:command1 && \ command2
or
find starting-directory -mindepth 3 -maxdepth 5 -iname "*.some_extension" -type f \ -newer some_existing_file -user fred -readable -print
As terdon says,
(
and)
can be used to group commands. The statement that they are “not really relevant” to that discussion is debatable. Some of the commands in terdon’s answer may be command groups. For example,( command1 ; command2 ) && ( command3; command4 )
does this:
- Run
command1
and wait for it to finish. - Then, regardless of the result of running that first command,
run
command2
and wait for it to finish. Then, if
command2
succeeded,- Run
command3
and wait for it to finish. - Then, regardless of the result of running that command,
run
command4
and wait for it to finish.
If
command2
failed, stop processing the command line.- Run
- Run
Outside parentheses,
|
binds very tightly, socommand1 | command2 || command3
is equivalent to
( command1 | command2 ) || command3
and
&&
and||
bind tighter than;
, socommand1 && command2 ; command3
is equivalent to
( command1 && command2 ) ; command3
i.e.,
command3
will be executed regardless of the exit status ofcommand1
and/orcommand2
.