Default field separator for awk

Let's take a look at the GNU awk man page:

FS — The input field separator, a space by default. See Fields, above.

To the Fields section!

As each input record is read, gawk splits the record into fields, using the value of the FS variable as the field separator. If FS is a single character, fields are separated by that character. If FS is the null string, then each individual character becomes a separate field. Otherwise, FS is expected to be a full regular expression. In the special case that FS is a single space, fields are separated by runs of spaces and/or tabs and/or newlines.


Here's a pragmatic summary that applies to all major Awk implementations:

  • GNU Awk (gawk) - the default awk in some Linux distros
  • Mawk (mawk) - the default awk in some Linux distros (e.g., earlier versions of Ubuntu crysman reports that version 19.04 now comes with GNU Awk - see his comment below.)
  • BWK Awk - the default awk on BSD-like platforms, including macOS

On Linux, awk -W version will tell you which implementation the default awk is.
BWK Awk only understands awk --version (which GNU Awk understands in addition to awk -W version).

Recent versions of all these implementations follow the POSIX standard with respect to field separators[1] (but not record separators).

Glossary:

  • RS is the input-record separator, which describes how the input is broken into records:

    • The POSIX-mandated default value is a newline, also referred to as \n below; that is, input is broken into lines by default.
    • On awk's command line, RS can be specified as -v RS=<sep>.
    • POSIX restricts RS to a literal, single-character value, but GNU Awk and Mawk support multi-character values that may be extended regular expressions (BWK Awk does not support that).

  • FS is the input-field separator, which describes how each record is split into fields; it may be an extended regular expression.

    • On awk's command line, FS can be specified as -F <sep> (or -v FS=<sep>).
    • The POSIX-mandated default value is formally a space (0x20), but that space is not literally interpreted as the (only) separator, but has special meaning; see below.

By default:

  • any run of spaces and/or tabs and/or newlines is treated as a field separator
  • with leading and trailing runs ignored.

The POSIX spec. uses the abstraction <blank> for spaces and tabs, which is true for all locales, but could comprise additional characters in specific locales - I don't know if any such locales exist.

Note that with the default input-record separator (RS), \n, newlines typically do not enter the picture as field separators, because no record itself contains \n in that case.

Newlines as field separators do come into play, however:

  • When RS is set to a value that results in records themselves containing \n instances (such as when RS is set to the empty string; see below).
  • Generally, when the split() function is used to split a string into array elements without an explicit-field separator argument.
    • Even though the input records won't contain \n instances in case the default RS is in effect, the split() function when invoked without an explicit field-separator argument on a multi-line string from a different source (e.g., a variable passed via the -v option or as a pseudo-filename) always treats \n as a field separator.

Important NON-default considerations:

  • Assigning the empty string to RS has special meaning: it reads the input in paragraph mode, meaning that the input is broken into records by runs of non-empty lines, with leading and trailing runs of empty lines ignored.

  • When you assign anything other than a literal space to FS, the interpretation of FS changes fundamentally:

    • A single character or each character from a specified character set is recognized individually as a field separator - not runs of it, as with the default.
      • For instance, setting FS to [ ] - even though it effectively amounts to a single space - causes every individual space instance in each record to be treated as a field separator.
      • To recognize runs, the regex quantifier (duplication symbol) + must be used; e.g., [\t]+ would recognize runs of tabs as a single separator.
    • Leading and trailing separators are NOT ignored, and, instead, separate empty fields.
    • Setting FS to the empty string means that each character of a record is its own field.

  • As mandated by POSIX, if RS is set to the empty string (paragraph mode), newlines (\n) are also considered field separators, irrespective of the value of FS.

[1] Unfortunately, GNU Awk up to at least version 4.1.3 complies with an obsolete POSIX standard with respect to field separators when you use the option to enforce POSIX compliance, -P (--posix): with that option in effect and RS set to a non-empty value, newlines (\n instances) are NOT recognized as field separators. The GNU Awk manual spells out the obsolete behavior (but neglects to mention that it doesn't apply when RS is set to the empty string). The POSIX standard changed in 2008 (see comments) to also consider newlines field separators when FS has its default value - as GNU Awk has always done without -P (--posix).
Here are 2 commands that verify the behavior described above:

  • With -P in effect and RS set to the empty string, \n is still treated as a field separator:
    gawk -P -F' ' -v RS='' '{ printf "<%s>, <%s>\n", $1, $2 }' <<< $'a\nb'
  • With -P in effect and a non-empty RS, \n is NOT treated as a field separator - this is the obsolete behavior:
    gawk -P -F' ' -v RS='|' '{ printf "<%s>, <%s>\n", $1, $2 }' <<< $'a\nb'
    A fix is coming, according to the GNU Awk maintainers; expect it in version 4.2 (no time frame given).
    (Tip of the hat to @JohnKugelman and @EdMorton for their help.)

'[ ]+' works for me. Run awk -W version to get the awk version. Mine is GNU Awk 4.0.2.

# cat a.txt
tcp        0      0 10.192.25.199:65002     0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:26895         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:18422           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 10.192.25.199:8888      0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN
tcp        0      0 10.192.25.199:8093      0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8670            0.0.0.0:*               LISTEN

For example, I want to get the Listen port. So I need to use the awk default delimiter added with ':'

# cat a.txt  | awk -F '[ ]+|:' '{print $5}'
65002
26895
111
18422
22
8888
50010
50075
8093
8670

If you just want to test the default delimiter, you can run

# cat a.txt  | awk -F '[ ]+' '{print $4}'
10.192.25.199:65002
127.0.0.1:26895
0.0.0.0:111
0.0.0.0:18422
0.0.0.0:22
10.192.25.199:8888
0.0.0.0:50010
0.0.0.0:50075
10.192.25.199:8093
0.0.0.0:8670

The result is as expected.


The question the default delimiter is only space for awk? is ambiguous but I'll try to answer both of the questions you might be asking.

The default value of the FS variable (which holds the field separator that tells awk how to separate records into fields as it reads them) is a single space character.

The thing that awk uses to separate records into fields is a "field separator" which is a regular expression with some additional functionality that only applies when the field separator is a single blank character. That additional functionality is that:

  1. Leading and trailing white space is ignored during field splitting.
  2. Fields are separated at chains of contiguous space characters which includes blanks, tabs and newlines.
  3. If you want to use a literal blank character as a field separator you must specify it as [ ] instead of just a standalone literal blank char like you could in a regexp.

In addition to field separators being used to split records into fields as the input is read they are used in some other contexts, e.g. the 3rd arg for split(), so it's important for you to know which contexts require a string or a regexp or a fieldsep and the man page clearly specifies each.

Among other things, the above explains this:

$ echo ' a b c ' | awk '{printf "%d: <%s> <%s> <%s>\n", NF, $1, $2, $3}'
3: <a> <b> <c>
$ echo ' a b c ' | awk -F' ' '{printf "%d: <%s> <%s> <%s>\n", NF, $1, $2, $3}'
3: <a> <b> <c>
$ echo ' a b c ' | awk -F'[ ]' '{printf "%d: <%s> <%s> <%s>\n", NF, $1, $2, $3}'                              
5: <> <a> <b>

so if you don't understand why the first 2 produce the same output but the last is different, please ask.

Tags:

Linux

Unix

Awk

Posix

Separator

Related

How to find which points intersect with a polygon in geopandas? python pandas extract year from datetime: df['year'] = df['date'].year is not working Right way to compute cosine similarity between two arrays? How can I change the dots to another character on password field in iOS Swift Using multiple criteria to find a WebElement in Selenium android:theme="@android:style/Theme.Translucent.NoTitleBar" causing my activity to crash Atom text editor remove trailing whitespace on save python 2.7 equivalent of built-in method int.from_bytes how to add RequestMappingHandlerMapping and ResourceHandlers to a springMVC configuration class Swift turn a country code into a emoji flag via unicode How do I read a Parquet in R and convert it to an R DataFrame? Why is a static method considered a method?

Recent Posts