How to say "I'm not interested" in a polite way?

Simply saying that you are more interested in the earlier research direction ('topic A') than the current one, should be fine. Thank him for getting back to you, but you don't need to say more.

No need, of course, to say you think that the new topic (B) is boring. But "Thanks" is all you need to express.

One thing that might happen is that you get a reply containing some lobbying for you to join. You should listen, but have no obligation to accept any offer, of course. The person might be a valuable contact in the future, even if you are in a different area.

Don't worry too much, can happen, has happened to me and I am sure to others as well. Just be polite and emphasize your own preferences.

You could write for instance that you really appreciate their offer as you admire their work, but that you plan to specialize on topic XY.