Output from ls has newlines but displays on a single line. Why?

When you pipe the output, ls acts differently.

This fact is hidden away in the info documentation:

If standard output is a terminal, the output is in columns (sorted vertically) and control characters are output as question marks; otherwise, the output is listed one per line and control characters are output as-is.

To prove it, try running

ls

and then

ls | less

This means that if you want the output to be guaranteed to be one file per line, regardless of whether it is being piped or redirected, you have to run

ls -1

(-1 is the number one)

Or, you can force ls | less to output in columns by running

ls -C

(-C is a capital C)

Your discovery highlights the primary reason why parsing the output of ls is always a bad idea. See Greg's wiki for a full explanation.

Think of your problem in reverse. You noticed that ls sometimes does and sometimes doesn't print newlines between it's output. For use in scripts or when forced by the -1 flag, it does. One newline at the end of each file. What there is no guarantee that each newline represents a new file name. In fact, if a filename contains a newline itself, the output of ls will be absolutly un-parsable. Consider these filenames:

file1
file2\nfile3
file4

When you ls -1 a directory with that in it, you would get something that looked like this:

file1
file2
file3
file4

Would you not naturally asume there were four files? So would any scripts that parse the output of ls. In reality there are three files, one of the with a tricky name, but you would not be able to figure that out from the output of ls.*

_{* Unless you were using the -l flag and noticed the output was borked, but your scripts would still choke.}