Is there an independent oversight body of Journal Impact Factors?
No.
Academic publishing is not regulated by any oversight body. There is not even a universal standard for what constitutes a publication.
There's multiple questions contained within your question that have different answers.
Is there independent oversight of Journal Impact Factors? ... Now I am wondering is there a oversight body that regulates the area?
This depends on what you mean by "independent". If you mean an official unbiased centralised not-for-profit professional organisation, then the best answer is probably no.
But ...
Specifically is there a body that sets the standards for impact factors and audits them so as that one impact factor can be compared with another with the knowledge that they are calculated in the same way?
The answer to this part is definitively yes. The notion of an impact factor (<- a recognised term of art) was invented by the founder of the Institute for Scientific Information (ISI), Eugene Garfield. The impact factor of journals has been computed by ISI since 1975 using a prescribed methodology.
In terms of comparability, the impact factor is specifically the average number of citations per publication in the journal, over a fixed time span (the two years previous, or five years for 5 year impact). Citations are collected from the Journal Citation Reports (JCR) index.
And so if you are a journal editor and you wish to have an impact factor, you need to apply to ISI (now owned by Thompson Reuters). They oversee the computation of the official Impact Factor metrics in common use today, as publicised on various journal websites.
In fact, Thompson Reuters (through ISI) own the concept of Impact Factor for journals and copyright said metrics.
So for sure there is a body "that sets the standards for impact factors and audits them so as that one impact factor can be compared with another with the knowledge that they are calculated in the same way". But I would hesitate to call Thompson ISI an independent body.
This answer is not intended to promote the idea of an Impact Factor, but merely to indicate its oversight, regulation and history as per the question.