What is the difference between ISO and DMG images?

ISO files are files containing a disk with a ISO 9660 filesystem. This is a filesystem optimised for read only access on a slow disc. (Specifically it was designed for a CDROM).

ISO files are limited in quite a few ways. E.g:
You can not write to an ISO filesystem. (You can create a new FS though)
You could not create directories more than 8 levels deep. You could only use file names following the DOS 8.3 standard.

Note that not all ISO files follow this behaviour. Many programs allow you to write file names with a length of 32 characters, which will work in many cases, but not all. (e.g. a disk with such filenames will fail to work with DOS). There are also several extentions to ISO 9660, including the Rock ridge and the Joliet extensions.

---

DMG files are disk images. They are treated by OS/X as real disks. You can read and write to these.