Configurable products and attribute set
After asking around this are the reasons I've got. Hope its satisfying for you, even though its probably what you expected.
- The adminhtml interface was made with the intention to make it hard for merchants to totally screw up.
For that reason many features that are provided by Magento as a Framework are not possible through the user interface.
The reason why only products in the same attribute set are selectable as associated simple products for configurables is because that's what was in the specs.
As you said, it kind of makes sense that way.
- Another reasons is thinking of the purpose of attribute sets. One of the reasons they exist is to reduce the number of attributes and options that have to be loaded and processed during a request. Applying this thinking to configurables makes sense, since it is a relatively resource expensive product type.
Because the backend was built in that way, the frontend logic for configurable never was expected to deal with simple products from different attribute sets.
Thats why the restrictions aren't implemented completely there.
Probably it would be easy to make configurables work with simples from different attribute sets. It just wasn't intended that way.
I didn't ask further and didn't look myself to see what code interaction exactly produced the strange result on the frontend. Probably its not important, since if I understood correctly you where asking for the reasoning behind the decision to exclude simple products from attribute sets different to the configurable, not the code explanation.
We use an extension called RapidFlow by Unirgy ( which I can't recommend highly enough ) for import and export. One of the features in the Pro version allows changing of attribute sets. Another is creation of products via CSV import. Occasionally we create new simple products for configurables, and sometimes by accident those simple products have different attribute sets than their parents.
Rapidflow will happily import these products, and change attribute sets. We tend not happy with the results. Configurable products that are configured by attributes outside of their attribute set fail to render in the product admin and must be repaired. If we don't change the attribute set of the parent, as you have noted, the children simply fail to be properly associated with their parent. They exist as Magento entities, can be edited, but will not appear as children on the front end product page, or the configurable parent's associated product list.
So, from a purely technical stand point, it is possible for simple products to be in a different attribute set than their parent. But as this behavior is not supported, even in EE, and so Occam's Razor says that when they were designing Magento the Varien devs didn't see the need.