Pros and cons for different configuration formats?

It's mostly personal preference, purpose, and available libraries. Personally I think xml is way too verbose for config files, but it is popular and has great libraries.

.cfg, .ini are legacy formats that work well and many languages have an included library that reads them. I've used it in Java, Python, C++ without issues. It doesn't really work as a data interchange format and if I am passing data I will probably use the same format for config and data interchange.

yaml, and json are between xml and cfg/ini. You can define many data structures in both, or it can be a simple key-value like with cfg. Both of these formats have great libraries in python and I'm assuming many other languages have libraries as well. I believe json is subset of yaml.

I've never used a python file as config, but it does seem to work well for django. It does allow you to have some code in the config which might be useful.

Last time I was choosing a format I chose yaml. It's simple but has some nice features, and the python library was easy to install and really good. Json was a close second and since the yaml library parsed json I chose yaml over it.

Note, this is pure opinion and speculation on my part but I suspect that the single biggest reason for the plethora of formats is likely due to the lack of a readily available, omnipresent configuration file parsing library. Lacking that, most programs have to write their own parsers so it would often come down to balance between how complex the configuration structure needs to be (hierarchical vs flat, purely data vs embedded logic like if statements, etc), how much effort the developers were willing to spend on writing a configuration file parser, and how much of a pain it should be for the end user. However, probably every reason you've listed and could think has likely been the motivation for a project or two in choosing their format.

For my own projects I tend to use .ini simply because there's an excellent parser already built in to Python and it's been "good enough" for most of my use cases. In the couple of cases it's been insufficient, I've used an XML based configuration file due, again, to the relative simplicity of implementation.