First Normal Form, why is it good and how does it reduce redundancy

The primary importance of first normal form is not that it eliminates redundancy, but rather, it's that it eliminates repeating groups.

Instead of having multiple columns of the same kind of data in a record, (0NF) you remove the repeated information into a separate relation and represent them as rows. This is what constitutes 1NF.

Tables that have columns like: phone_1, phone_2, phone_3 or that contain list-oriented data like: 212-555-1212, 212-555-1234, 416-967-1111 violate 1NF.

1NF is important because it is much more flexible than 0NF while being much easier to use when inserting, updating and reading data. This is because every type of data element (e.g. customer phone number) has exactly one column in which to find it and that column has only one piece of data for each record. This means that you can use simple SQL statements to read or write individual data elements without having to parse delimited strings or use constructions like: where phone_1=@Number or phone_2=@Number or phone_3=@Number and so forth.

Regarding 1NF vs 3NF, the normal forms are cumulative. A table in 3NF is also in 1NF, so it is just as true to say that "Every table in a relational database should be in 1NF" as it is to say that "Every table in a relational database should be in 3NF." I would say both of these are true, but I would add "unless you have a really good, well considered reason to denormalize".

"I guess another way to ask it is why is it important for values to be atomic?"

I'm currently working with applications that have comma-separated values in varchar/text fields, and it's a pain in the ass. Among other things, in the application you have to do all kids of silly string splitting and converting to get the data on the correct form, and on the DB side you can't use indexes on the field, as all searches have to be on the form WHERE mycolumn LIKE '%search%'

There are other reasons as well, but those two come to mind first.