What can happen if someone acknowledges a grant on papers not related to that grant?

The big underlying question here is what it means for a paper to be "related to a grant". Different funding agencies have different rules on this, either a more "staff-oriented view" or a more "topic-oriented view".

• Many pure science funds, for instance the Swiss National Science Foundation, take a liberal stance on what kind of science can be enabled through each individual project. Essentially, while a project is given to support a specific idea, they are fully aware and quite happy that in addition to, and sometimes instead of, this project, other more or less related studies will also happen by the staff paid partly or in full through the project. At the end of the day, the SNF and other similar agencies care not so much about the specific piece of science that is described in the project, but that some valuable science is happening and that PhD students are graduating. Based on this view, researchers funded through SNF are expected to acknowledge them on every paper they do, no matter if it has to do with any of the science described in the project description. The collection of all papers acknowledging this project is then not understood as "all papers on topic X", but rather "all topics enabled by funding project Y, on topic X and any side projects that the funded students did in addition or instead".

• Other, especially more applied funds such as most of the European Commission's Horizon 2020 programme, are much more interested in specific topics than in funding just about any research. For them, acknowledgements have much more the meaning that you seem to imply, that is, that citing this project in the acknowledgements of the paper implies that this contributes to the mission of the project. How strongly this is enforced or checked depends on the project officer. Most people also seem to take a rather liberal approach in practice, but there are certainly project officers that have formally complained about projects using the allocated funds to sponsor other research.

Might he go into troubles if it is discovered by the funding agency that the papers are actually not related to the grant proposal?

So at the end of the day, it really depends on whether the agency actually considers this "topically unrelated but financially enabled by the grant" paper as "unrelated". I would say, in absence of any other information, you can safely assume that your co-author did nothing wrong, and that the agency may be fine with this acknowledgement.

Many grants require progress reports. The NIH system is pretty smart and automatically finds publications by the PI. The system lists if the publication acknowledges the grant. When reviewing officials check the progress report they look to make sure the publications that acknowledge the grant are related and that those that do not acknowledge the grant are not related. They obviously are not experts, but they often know enough to judge what is and is not related.

If they find that the grant was used to fund a lot of research that was very far removed from the aims, the PI might get a nasty email saying do not do it again. If the PI has a history of doing it, at some point the funder might cancel the contract. It might eventually go so far as to blacklist the PI. In the worse case, which I think is unlikely, the funder could probably sue to get the money back. In the US, it is the university who holds the grant, so the PI would have an additional level of bureaucratic protection.