How to synchronize two microcontrollers to micro-second accuracy?

I don't mean to rain on your wireless parade. You've ran into a tough but unexpected requirement. Something like that warrants re-evaluation of the whole system design.

1st thing that comes to mind is to clock both units off one oscillator. You have Bluetooth communication, which hints that the range is on the order of 10m. You could connect your units with RG174 coax cable or an optical fiber, which would carry the clock.

2nd, there are precision oscillators. In order of increasing precision and cost.

• TCXO (temperature compensated crystal oscillator). 1 to 3 ppm drift, typically.
• OCXO (oven controlled crystal oscillator). Drift on the order of 0.02ppm. Some OCXO have drift down to 0.0001 ppm.
• Atomic clock (Rubidium standard, for example). I'm mentioning atomic clock mostly to give a frame of reference. More on that here.

3rd, precision oscillator trained with GPS. Every GPS satellite has several atomic clocks on board. Usually, there are plenty of GPS satellites in view. GPS is used for precision timing a lot (less known usage compared to sat nav). Most GPS receivers have a 1PPS output (one pulse per second), which provides timing accurate to 50ns.
To have a 0.5μs drift over 600s (10min), your clock (the 12MHz clock in your present design) should have drift less than 0.0008ppm. But if you can correct the timing error every so often from an low drift external source, the requirement for the drift in the clock can be more relaxed. If you can correct every second, then your clock could have a 0.5ppm drift.

GPS modules with 1pps outputs are readily available and inexpensive.

It isn't really necessary to discipline the CPU's oscillator to the GPS (e.g., with a PLL). As long as you can "timestamp" external events relative to the CPU clock, it's relatively straightforward to interpolate the time of your wave transmit and receive events between any two PPS events.

You can often use the combination of a hardware timer on the microcontroller, along with a software counter for its overflow events, to create a CPU cycle counter of arbitrary width. It can be tricky to deal correctly with rollover events, both of the hardware counter and the software counter, but in the end, you can have, say, a 32-bit counter that counts at the rate of the CPU clock (giving high resolution) and rolls over with a period longer than the intervals you're trying to measure (e.g., 429 seconds @ 10 MHz).

You can use this counter to timestamp different external events. If one of those events is 1-pps pulses from a GPS receiver, then the basic long-term accuracy of the CPU clock becomes a don't-care. The only thing that matters is its short-term stability. You can save GPS timestamps in a FIFO buffer, and compare the timestamps of other events to the values in that buffer. Since you know the GPS pulses are exactly one second apart, you can find the exact time of any other event by interpolating.

Suppose \$GPS_n\$ and \$GPS_{n+1}\$ are the CPU-clock timestamps for two successive GPS pulses. You also know the actual (atomic clock) times associated with each of those pulses (from the GPS messages), \$Time_n\$ and \$Time_{n+1}\$. If \$Ext\$ is the CPU-clock timestamp for some external event you want to measure that falls between \$GPS_n\$ and \$GPS_{n+1}\$, its exact time is:

$$Time_n + \frac{Ext - GPS_n}{GPS_{n+1} - GPS_n}$$

Finally, if you have this setup running on two separate systems, each with its own GPS receiver, you can compare the times calculated for various events on the two systems with high precision (typically on the order of ±100 ns), even if the CPU clocks of the two systems are not synchronized.

I have implemented a wireless clock synchronization for microcontrollers before, but only with millisecond accuracy, which was good enough for the application. From my reading, this paper explains microsecond synchronization quite well: http://www.math.u-szeged.hu/tagok/mmaroti/okt/2010t/ftsp.pdf

Essentially, if you have knowledge of the transmission event and the arrival event of a radio packet on the transmitter and receiver respectively, you have a common observable event (assuming you ignore the propagation time of the radio wave) between the 2 systems that can be used as a reference. The other neat feature mentioned in the paper is clock-skew estimation using linear regression.