How to detect canceled or refunded order for Google in-app billing?

Have you checked out the voided purchase list API? It returns a list of cancelled, refunded or charged-back purchases.

https://developers.google.com/android-publisher/api-ref/purchases/voidedpurchases/list