How do we cross-verify if the device is doing exactly what it is supposed to do?

There is a saying at my company:

Quality Assurance is making sure that something does what it's supposed to. Security Assurance is making sure that it does only what it's supposed to.

So unfortunately, I think your questions will lead you to a full security audit or penetration test, which, as you point out, requires great knowledge and skills which most people don't have.

... how do we know any device is doing what it is supposed to do

We don't. But this lack of absolute certainty is not specific to IT security.

If you talk to your friends you don't know for sure if the they are telling the (full) truth. If you buy something you don't know for sure if it actually has all the qualities the vendor claims.

But it is not that you merely blindly hope that everything will be fine: you believe your friends because of the good experiences you had so far and because if somebody found out the lies they would risk to loose your friendship. You trust some vendors more than others for example because trusted friends recommend these or because some important brand would had too much too loose if they lied too much.

The same is true with devices you buy. Given that many lies about the quality eventually are detected (like built-in backdoors, selling your privacy, ...) major brands try their best to not lie to you since they have a lot to loose. Trust in the quality of their products is part of their business model. Contrary to this cheaper brands have not much too loose. Therefore it is more likely to find bugs, bad quality of the hard- and software and even backdoors in products from such cheap brands.

I can think of reverse engineering but that would require great knowledge and skills which most people dont have.

For the major brands it is usually enough of a risk that someone with enough skills will have some spare time (or even get paid for this) to dig deeper. And major brands have usually more customers and maybe even some customers who are willing to invest time or money in such analysis, like when using such devices in a business or government environment.

Cheaper brands on the other hand have both less exposure and have much less too loose, which means they also care much less. They might even use this limited exposure and risk as its own business value: since they have none or only few credibility in the first place they can easily partner with shady companies or organizations without loosing credibility. This results for example in cheap mobile phones coming pre-installed with adware or other PUP.

How do we cross-verify if the device is doing exactly what it is supposed to do?

As you already found out yourself: for the average person this is impossible. And even for experts it would be too costly and time-consuming to analyze every new device they use.

In the end it boils down to the trust you can have in the vendor. And as is with friends: such trust is hard (and maybe costly) to gain but much easier to loose. Thus major brands try to keep their hard-gained reputation up which makes it much less likely that you encounter serious issues on their devices - at least compared to the cheaper brands which have not much reputation anyway.