Sitecore Lucene: content delivery server index not updating on publish

I ran into this same issue last night and have a more predictable resolution than creating a new IIS site:

The resolve was to set a distinct InstanceName in ScalabilitySettings.config for each CD server, instead of relying on the auto-generated name.

Setting this value immediately resolved the issue and restored the index update functionality upon Publish End Remote events.

Note: If you already have an InstanceName defined in your config, then you need to change it for this to work. I just increment the InstanceName with the date to force the change.

This is effectively fixing the same issue in the same way as the original poster did by changing to a new IIS site, as the OP's fix would have modified the auto-generated Instance Name based on the new IIS site name.

I believe the core problem with the OP (and also in my instance) is related to the EventQueue databases going out of sync with the CD instances and none of the servers being able to determine that an event has been generated / what content needs to update in the index. By changing the Instance Name (using either method) the servers appear to be new instances and start from scratch with their EventQueue tracking.

Every time I've seen issues like this in the past it's been related to major manipulations of Sitecore databases. Such as restorations, backup/restore to a new DB name, or rollbacks of databases due to deployment problems. I believe something in the above operations causes the EventQueues to get out of sync and the servers stop responding to the expected events.

I had this issue and it drove me nuts for a few months. I figured out that the answer lied in the rebuild strategy of the Lucene Index. The only way for Lucene to know to rebuild itself when the CM and CD are in separate instances of IIS, is for lucene to watch the EventQueue table and recognize that a change happened to an item that is either at the root, or child of the root that you specify in the crawler node. The strategy that you'll need to specify as the rebuild strategy to guarantee this behavior is below

<strategies hint="list:AddStrategy">
  <strategy ref="contentSearch/indexUpdateStrategies/remoteRebuild" />
</strategies>

If you use any other rebuild strategy with a remote instance of a content delivery server, the index will only be rebuilt in the CM instance's file system.