How to define mutually exclusive attributes in XSD?

With XSD 1.0, you can use xs:keyelement.

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="tag">
    <xs:complexType>
        <xs:attribute name="name" type="xs:string" use="required"/>
        <xs:attribute name="abc"  type="xs:integer"/>      
        <xs:attribute name="def"  type="xs:integer"/>
     </xs:complexType>
    <xs:key name="attributeKey">
        <xs:selector xpath="."/>
        <xs:field xpath="@abc|@def"/>
    </xs:key>
</xs:element>   

Edit: If both attributes are present (even with different values), this creates two keys, so the XML validation will fail. On the other hand, the <xs: key> requires that a key is defined for the element, and therefore one of the two attributes must be present.

the following XML doc is not valid using the above XSD. (I'am using oXygen 17.0):

<tag xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="stack3.xsd" name="" abc="12" def="13"/>

Error:

cvc-identity-constraint.3: Field "./@abc|./@def" of identity constraint "attributeKey" matches more than one value within the scope of its selector; fields must match unique values

XSD 1.0

Can be done with clever trick using xs:key. See @Kachna's answer.

Note that some parsers may allow both attributes if they fail to fail for multiple selected values in xs:key. There is at least one known case of this happening in the past.

XSD 1.1

Can be done using xs:assert:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns:vc="http://www.w3.org/2007/XMLSchema-versioning"
           vc:minVersion="1.1">
  <xs:element name="tag">
    <xs:complexType>
      <xs:sequence/>
      <xs:attribute name="name" type="xs:string"/>
      <xs:attribute name="abc" use="optional" type="xs:integer"/>      
      <xs:attribute name="def" use="optional" type="xs:integer"/>
      <xs:assert test="(@abc and not(@def)) or (not(@abc) and @def)"/>      
    </xs:complexType>
  </xs:element>
</xs:schema>