Ghost processes / batch processes (NA34 and NA39)
Not sure this is worthy of an answer, but too much to put as a comment. The only time I have seen anything remotely similar was with this and I was able to prove to support that a job had run and created duplicates.
The info I got back from Tier 3 at the time:
As we discussed, during the execution, an internal server error on the asynchronous process for the batch, resulted in cloned message in the asynchronous queue to not be removed resulting in duplicate messages getting worked. As a result of unlucky timing when an exception is thrown, this may result in cloned messages not being deleted and actually executed in parallel to actual messages
A Bug has been logged to fix this issue from Salesforce end.
It is scheduled in patch release next week.
'Last week' was just over 2 years ago. I can share my case # if you wish. The case status is currently: Closed - Bug Fix Submitted
Worth noting our customer stopped using our app ~6 months after but I never saw a re-occurrence in that time. It would also have been an EUn pod.
Good luck.
Seems I got some attention after submitting a case or this post or whatever, just got this notification: A bit late but at least I know the cause
On March 16, 2017, Salesforce identified a misconfiguration in our test environment that allowed Apex callouts and outbound messages to reach endpoints outside of the test environment. At the same time, these Apex callouts and outbound messages were being executed, as expected, in your production environment. The simultaneous execution of callouts and messages from both environments may have resulted in duplicate or unexpected results. The issue was resolved at approximately 5:30 p.m. PDT on March 16, 2017. However, the fix did not correct the duplicate or unexpected results from Apex callouts and outbound messages that occurred as early as February 28 through March 16, 2017.
We have identified that your org is associated with Apex callouts and/or outbound messages that executed in the testing environment between February 28 and March 16, 2017. However, we are unable to confirm if these jobs produced duplicate or unexpected results as they are customized to your implementation.
This just in from SF:
On March 16, 2017, Salesforce identified a misconfiguration in our test environment that allowed Apex callouts and outbound messages to reach endpoints outside of the test environment. At the same time, these Apex callouts and outbound messages were being executed, as expected, in your production environment. The simultaneous execution of callouts and messages from both environments may have resulted in duplicate or unexpected results. The issue was resolved at approximately 5:30 p.m. PDT on March 16, 2017. However, the fix did not correct the duplicate or unexpected results from Apex callouts and outbound messages that occurred as early as February 28 through March 16, 2017.
We have identified that your org is associated with Apex callouts and/or outbound messages that executed in the testing environment between February 28 and March 16, 2017. However, we are unable to confirm if these jobs produced duplicate or unexpected results as they are customized to your implementation.
Which sounds like it could have just the results that you were seeing!