Merged ObservableCollection

You have to offset the index of the notification event.

Say you remove an item from the first collection at index 2. A collection changed event is fired with index 2.

If you remove an item from the second collection at index 2, the event is fired with the same index (2), but the item is actually enumerated after all of the items in the first collection.

Is there any reason you can't use CompositeCollection?

The reason the exception is being thrown is because you're not translating the indexes of the inner collections to the outer. You're just passing the exact same event args to the outer event (on MergedCollection), which is why WPF doesn't find the items where the index is telling it to find them.

You use a CompositeCollection like so:

<ListBox>
  <ListBox.Resources>
    <CollectionViewSource x:Key="DogCollection" Source="{Binding Dogs}"/>
    <CollectionViewSource x:Key="CatCollection" Source="{Binding Cats}"/>
  </ListBox.Resources>
  <ListBox.ItemsSource>
    <CompositeCollection>
      <CollectionContainer Collection="{Binding Source={StaticResource DogCollection}}"/>
      <CollectionContainer Collection="{Binding Source={StaticResource CatCollection}}"/>
    </CompositeCollection>
   </ListBox.ItemsSource>
   <!-- ... -->
</ListBox>

For details, see this answer.