How to solve "Both use the XML type name X, use XML attributes to specify a unique XML name and/or namespace for the type"?

You need to provide the Namespace by using the XmlElement attribute on the properties of your UISettings class:

public class UISettings
{
    public UISettings()
    {

        ItemTable = new ItemTable();
        EffectiveItemPermissionTable = new EffectiveItemPermissionTable();
    }
    [XmlElement(Namespace = "Item")]
    public ItemTable ItemTable { get; set; }
    [XmlElement(Namespace = "Permissions")]
    public EffectiveItemPermissionTable EffectiveItemPermissionTable { get; set; }
}

When applied here this will be your serialized output:

<?xml version="1.0" encoding="utf-16"?>
<UISettings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">  
   <ItemTable xmlns="Item">    
      <DisplayMode>Tiles</DisplayMode>  
   </ItemTable>  
   <EffectiveItemPermissionTable xmlns="Permissions">    
       <DisplayMode>FullPaths</DisplayMode>  
   </EffectiveItemPermissionTable>
</UISettings>

Alternatively, and maybe cleaner, you can provide the Namespace on the types:

[XmlType(Namespace="Item")]
public class ItemTable : Table<ItemTableNS.DisplayMode>
{ }

[XmlType(Namespace = "Permission")]
public class EffectiveItemPermissionTable : Table<EffectiveItemPermissionTableNS.DisplayMode>
{ }

This will serialize as:

<?xml version="1.0" encoding="utf-16"?>
<UISettings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <ItemTable>
    <DisplayMode xmlns="Item">Tiles</DisplayMode>
  </ItemTable>
  <EffectiveItemPermissionTable>
    <DisplayMode xmlns="Permission">FullPaths</DisplayMode>
  </EffectiveItemPermissionTable>
</UISettings>

I realize this answer probably comes way too late for the OP, but there is a way to do this without using namespaces, so I'm going to leave an answer here in case somebody comes along after me and needs the solution.

The problem is caused by the fact that the way the XmlSerializer names a type X<Y> is by giving it the name XOfY. Thus, when you have two types that both derive from Table<TDisplayMode>, you get that error, since they'll both be known internally as TableOfDisplayMode, despite actually using different enums.

This is because ItemTable and EffectiveItemPermissionTableare actually not inheriting from the same type! One inherits from Table<ItemTable.DisplayMode> and the other from Table<EffectiveItemPermissionTable.DisplayMode>. Not that this is limited to inheritance; you'd face the same problem if you tried to use them directly in the same XML object graph also.

Now, for the non-generic counterpart to this problem, you'd just smack an [XmlType] on the two types, and call it a day. But you can't do that here. While Table<ItemTable.DisplayMode> and Table<EffectiveItemPermissionTable.DisplayMode> are different types, they share the same class definition, so by trying to use [XmlType], you're giving them a different name, but still the same name.

So what can you do? XmlAttributeOverrides to the rescue! It lets you override the names the XmlSerializer gives to closed generic types, meaning that you can finally give a different name to Table<ItemTable.DisplayMode> and Table<EffectiveItemPermissionTable.DisplayMode>:

var xmlOverrides = new XmlAttributeOverrides();

var xmlAttribs = new XmlAttributes();   
xmlAttribs.XmlType = new XmlTypeAttribute("TableOfItemTableDisplayMode");
xmlOverrides.Add(typeof(Table<ItemTable.DisplayMode>), xmlAttribs);

xmlAttribs = new XmlAttributes();
xmlAttribs.XmlType = new XmlTypeAttribute("TableOfEffectiveItemPermissionTableDisplayMode");
xmlOverrides.Add(typeof(Table<EffectiveItemPermissionTable.DisplayMode>), xmlAttribs);

System.Xml.Serialization.XmlSerializer lSerializer =
    new System.Xml.Serialization.XmlSerializer(typeof(UISettings), xmlOverrides);

And voilà! Assuming now that you've also put [XmlType] with different names for your DisplayMode enums, so that their names don't conflict either, you've got yourself a working setup!