How does SF calculate the CPU time?

Josh Kaplan the Apex Product Manager gave quite a detailed blog post on this new governor entitled Script Limits, Begone!. In it he states this...

What does this CPU timeout include? We are only counting things that require application server CPU use. For example, the time spent in the database retrieving records will not count, nor will time spent waiting for a callout to return. There are some things that use the app server CPU that we do not count, which are things beyond your control as a programmer. For example, you don’t control when your code needs compilation, so we don’t count that. We will be counting almost everything else that happens on the app server, including declarative actions. If DML in your code encounters a validation rule with a formula, we will count the time spent evaluating that formula.

The limit is non-deterministic which is why your seeing the variance. This could be considered a concern, given the limit it replaced, while troublesome to us, was at least predictable and scoped by namespace. However as you will see from Josh's excellent post (see section 'But I Have Fear And Doubt!') that Salesforce are all over this, monitoring orgs and reviewing any code both past and present that gets close I'm told. Josh has been present on SE a few times, let see if we can grab his attention to see if he would like to comment further beyond the blog statements.

There is also this statement in the documentation under Understanding Execution Governors and Limits. As well as the usual methods on the Limits class.

CPU time is calculated for all executions on the Salesforce application servers occurring in one Apex transaction—for the executing Apex code, and any processes that are called from this code, such as package code and workflows. CPU time is private for a transaction and is isolated from other transactions. Operations that don’t consume application server CPU time aren’t counted toward CPU time. For example, the portion of execution time spent in the database for DML, SOQL, and SOSL isn’t counted, nor is waiting time for Apex callouts.

I raised a case with SalesForce to ask them the same question. After a several weeks I finally got a reply to say:

We have consulted our backline team and they have confirmed that time will change as they are subjected to CPU availability.

Processors naturally take a variable amount of time to process code, so it's almost never going to be the exact amount of time with each execution. You will see the same behavior if you do the same thing on your local machine.

The important point here is to know that CPU time will vary when running the same code over and over again. Keep this in mind when writing code that comes close to the CPU limit. I've seen the CPU time vary by as much as 25% from one request to the next. SalesForce could not give me any information on how much of a variation that can be expected.

IMO, the dominant source of time variance is garbage collection that happens in the underlying platform, which is written in Java.

Please vote for the idea https://success.salesforce.com/ideaView?id=08730000000kzOTAAY to not account garbage collection time in a request's CPU time.