Iterate over lines instead of words in a for loop of shell script

Using for

for l in $() performs word splitting based on IFS:

$ for l in $(printf %b 'a b\nc'); do echo "$l"; done
a
b
c
$ IFS=$'\n'; for l in $(printf %b 'a b\nc'); do echo "$l"; done
a b
c

IFS doesn't have to be set back if it is not used later.

for l in $() also performs pathname expansion:

$ printf %b 'a\n*\n' > file.txt
$ IFS=$'\n'
$ for l in $(<file.txt); do echo "$l"; done
a
file.txt
$ set -f; for l in $(<file.txt); do echo "$l"; done; set +f
a
*

If IFS=$'\n', linefeeds are stripped and collapsed:

$ printf %b '\n\na\n\nb\n\n' > file.txt
$ IFS=$'\n'; for l in $(<file.txt); do echo "$l"; done
a
b

$(cat file.txt) (or $(<file.txt)) also reads the whole file to memory.

Using read

Without -r backslashes are used for line continuation and removed before other characters:

$ cat file.txt
\1\\2\
3
$ cat file.txt | while read l; do echo "$l"; done
1\23
$ cat file.txt | while read -r l; do echo "$l"; done
\1\\2\
3

Characters in IFS are stripped from the start and end of lines but not collapsed:

$ printf %b '1  2 \n\t3\n' | while read -r l; do echo "$l"; done
1  2
3
$ printf %b ' 1  2 \n\t3\n' | while IFS= read -r l; do echo "$l"; done
 1  2 
    3

If the last line doesn't end with a newline, read assigns l to it but exits before the body of the loop:

$ printf 'x\ny' | while read l; do echo $l; done
x
$ printf 'x\ny' | while read l || [[ $l ]]; do echo $l; done
x
y

If a while loop is in a pipeline, it is also in a subshell, so variables are not visible outside it:

$ x=0; seq 3 | while read l; do let x+=l; done; echo $x
0
$ x=0; while read l; do let x+=l; done < <(seq 3); echo $x
6
$ x=0; x=8 | x=9; echo $x
0

The for loop is not designed to loop over "lines". Instead it loops over "words".

Short terminology: "lines" are things separated by newlines. "words" are things separated by spaces (and newlines, among others). in bash lingo "words" are called "fields".

The idiomatic way to loop over lines is to use a while loop in combination with read.

ioscan -m dsf | while read -r line
do
  printf '%s\n' "$line"
done

Note that the while loop is in a subshell because of the pipe. This can cause some confusion with variable scope. In bash you can work around this by using process substitution.

while read -r line
do
  printf '%s\n' "$line"
done < <(ioscan -m dsf)

see also http://mywiki.wooledge.org/BashFAQ/024

The for loop splits the things to loop over using the characters in the $IFS variable as separators. IFS is short for Internal Field Separator. Usually $IFS contains a space, a tab, and a newline. That means the for loop will loop over the "words", not over the lines.

If you insist on using a for loop to loop over lines you have to change the value of $IFS to only newline. But if you do this you have to save the old value of $IFS and restore that after the loop, because many other things also depend on $IFS.

OLDIFS="$IFS"
IFS=$'\n' # bash specific
for line in $(ioscan -m dsf)
do
  printf '%s\n' "$line"
done
IFS="$OLDIFS"

in POSIX shells, that have no ANSI-C Quoting ($'\n'), you can do it like this:

IFS='
'

that is: put an actual new line between the quotes.

Alternatively you can use a subshell to contain the change to $IFS:

(
  # changes to variables in the subshell stay in the subshell
  IFS=$'\n'
  for line in $(ioscan -m dsf)
  do
    printf '%s\n' "$line"
  done
)
# $IFS is not changed outside of the subshell

But beware the command in the loop may itself depends on some sane setting for $IFS. Then you have to restore the $IFS before executing the command and set again before the next loop or some such. I do not recommend messing with $IFS. Too many commands depend on some sane values in $IFS and changing it is an endless nightmare of obscure bug hunting.

Iterate over lines instead of words in a for loop of shell script

Using for

Using read

Tags:

Linux

Shell

Related

Recent Posts