What is the difference between PEM format to DSA/RSA/ECC? (Might I confuse PEM with these)?
First off, here's the full man page entry for ssh-keygen -o from my machine (ssh-keygen doesn't seem to have a version flag, but the man page is from February 17, 2016)
-o Causes ssh-keygen to save private keys using the new OpenSSH format rather than the more compatible PEM format. The new format has increased resistance to brute-force password cracking but is not supported by versions of OpenSSH prior to 6.5. Ed25519 keys always use the new private key format.
Seems pretty clear that this is just about the format of the file that's being produced. Also note that ssh-keygen will only store Ed25519 keys in the new format, regardless of what flags you pass in.
Since both of your questions today have had the same underlying question, let's deal with it.
... these "newer formats" DSA/RSA/ECC ...
Ok, so DSA, RSA, and ECC are not different formats, they are completely different algorithms and are completely unrelated to each other.
I wish I could find a better way of explaining this, but I'm not sure I can without getting too technical. Let's try this: it's a bit like saying that http and ftp are just different formats for transferring files, or .docx and .pptx are different formats of Office documents. Calling these "just format differences" is fundamentally wrong, the software is doing a very different thing in the two cases (albeit to produce the same end-result; transferring a file, or making a pretty document).
Now let's talk about formats.
So you want to save a private key in a file? PEM is a file format for storing general cryptgraphic information, but other file formats exist. PEM can be used for many things: private keys, or certificates, or the text of an email that you want to encrypt or sign. It's just a container "cryptographic stuff".
Analogy time: saving a Word document. You could save it in the old .doc format which is universally accepted by all versions of Office and also open source programs (PEM is also older and universally accepted), or you could use the newer .docx format (the -o OpenSSH format). Sometimes new features are not backwards compatible and can only be saved in the new format (like ed25519).
(Many thanks to @GordonDavisson for this analogy)
With the exception of backwards compatibility, which format you choose really has nothing to do with the contents of the file.
Examples:
PEM format with an RSA key.
Note that the message starts with -----BEGIN RSA PRIVATE KEY-----, this is standard industry-wide PEM format - any software that can read PEM will be able to read this:
$ ssh-keygen -a 100 -t rsa
$ cat .ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAwTLkTDZUisg0M/3BDBOjrmBvJXb8cdfveGG1KQhAhFDLMp5w
XKgGFMy339m3TGB1+ekKoXnX3dcRQtpQbuFeDB59cmKpDGZz7BzfetpjeIOGCzG+
Vt2BnDf2OlKb6EE7tC3PSmFgL0nQPSyz8x7rE7CMOQEnz1tKkK9Gpttcku8ifwmd
LAHlVaTV3BIiFJN23fwe7l0czFaV98iy8YRgk/Av3gRlsBABGT5qD8wPiI4Ygpxv
UuT2kq53lhBTWO9hIth5tUU0N7x7QN6/rRSi2uHcmJYXFkpsLie697G17TN5ln3E
X5frsrjVyu0JhjvC5mlRBudSJWcWm/KrcTQxdQIDAQABAoIBAFR6nWtZ4nPhATqu
veg6+jq4vkEim1ZodrUr/FxZ2GRDM+cJctaBPk+ACPMgL099ankB1v0u2x6M+WZD
MiKZ91bTSkVnMMZUUmIvaeU9c3tx/34LnVA8gX0+1zM/hh7zz1iFI3xBwh5LZ3wo
fPNVVLOCYn5WrAK2x48mpX02tG8m0OY5kdCdFW1f6UbCGk3K9ov82yQ+JbjWydOS
ZaS7bnJCRsjDhPm/Ooe2DZ8CKHRO3DOiZAAFYqsosxM2C12+alC+hdKHoJ0pWZWe
eckZiALTENi+PzgSq4ykdBaoreeI84lpIzYQO06rrjM2/fw1x/SQsDYhCsQnNRib
sipl2mECgYEA74rBQDIEWaIVVZ72foeZYOMmAUKCLQtNxVuGl/XoMZO3evN2vbOv
Nw/nn5lVt6PRykVUUl6yb5gHvRSGtGJSB1q6pBVN1EF2sfvUep6mmpEwbSbENiAX
IftE0ap7PKT1aLKbLllwdsJlLCRMJIk07AvsulbkhSC+UmJVkfjblgkCgYEAznkE
fzmWS358MLxb0JM+4v1DO/1ne8z29ddHNppxSNj2Xjf+MMvOY4KaRQa8dMwc3Eqa
1EvPqnK6ila4L08Ai4CyWoGkkkIQBO9jO3mLp24xk6TRo002DbfJRnu4qW4zeryp
gYeFfBzwhc/IQKjXhv8AAfEUFCSYiFaWBsENuw0CgYEAwmyDyCAQqdPFrzYT6cUT
t7EGYtVhpT/cgshz6Rk9uiekL9Y2VWjnWTC+liq1iRUdLSiydRzJhYwHE+/6GaUH
4VJB1PY5soLj3TiCUHg+z4vym1VwwmGvhPRV+jt+RU26ppz5GVic0LeduINJjgoT
e1d+cAwg9PELqQCJZa5wREkCgYBPEuPZAaoAsalIVOro32uHLS1xrSPTsvSlxFO+
orleB9Ga1eDguT0KuTrx0pmcNYucBmpzgbE/ev7b+khBvgTcaGZl6R6o8OoHqdKc
NXl5nucXv1iWLPzVlhxchQd8w/qtN9HHDKrflIm9BY2Qzdj1F3XeSIDDEhzkohyE
66yhhQKBgGh9knh8SxnVxaPk0Rk/bQope9AwIDuIpfsf4KC/EPSZmMNHP8xBh8al
eymUot7Pj6dck31V4C3q74NKobY3p6fZ5t7tP9K6Br+J/FQFhvAdFAwpTD2Bks5H
fhZO5cniPpydb0YvOnoaVnb0nzXVsf1jIgPKfsCsZxoyE0jLb9oV
-----END RSA PRIVATE KEY-----
OpenSSH format with an ed25519 key:
In addition to it being shorter (because ECC keys just are that much shorter) notice that the message starts with ----BEGIN OPENSSH PRIVATE KEY-----, this is in OpenSSH-specific format that other software may or may not be able to read:
$ ssh-keygen -o -a 100 -t ed25519
$ cat .ssh/id_ed25519.o
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACC7gIlwBMp+H6VVNZSHI0in2iCU/yi67WfeFPfuyAdkBAAAAJh7nk7je55O
4wAAAAtzc2gtZWQyNTUxOQAAACC7gIlwBMp+H6VVNZSHI0in2iCU/yi67WfeFPfuyAdkBA
AAAEBHl+qBAosBAUIGuvdDR8gJN/PEhempLe4NtyKiO7hCPLuAiXAEyn4fpVU1lIcjSKfa
IJT/KLrtZ94U9+7IB2QEAAAAEm1pa2VATWlrZS1zYW1zdW5nMQECAw==
-----END OPENSSH PRIVATE KEY-----
The following is from the OpenSSH 7.8 release notes, 2018-08-24. (http://www.openssh.com/txt/release-7.8)
This release includes a number of changes that may affect existing configurations:
ssh-keygen(1): write OpenSSH format private keys by default instead of using OpenSSL's PEM format. The OpenSSH format, supported in OpenSSH releases since 2014 and described in the PROTOCOL.key file in the source distribution, offers substantially better protection against offline password guessing and supports key comments in private keys. If necessary, it is possible to write old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments when generating or updating a key.