How can I set unlimited retention for an compacted topic in Kafka?

Choose only "compact" as the cleanup policy, and set an infinite retention.

log.cleanup.policy = [compact]
log.retention.bytes = -1
log.retention.ms = -1

I found a solution that I want to share. Unfortunately, Kafka documentation is not very clear on this, so perhaps this will help someone:

Do not set this :

log.cleanup.policy = [compact, delete]

This setting will mean that all topics, are both compacted and deleted. So your topic will get compacted as per compaction rules, but when segments (messages) get older than the set retention time (in my case it was 20 min), they get deleted as well.

Set default clean up policy to:

log.cleanup.policy = compact
or 
log.cleanup.policy = delete

(log.cleanup.policy = delete is the default config)

"log.cleanup.policy = compact" will mean that topics, by default, will be set to be compacted.

When you set this default policy, you do not need to make any changes. There is no need to set log.retention to -1 or any other value. Your topics will be compacted and old messages never deleted (as per compaction rules)

"log.cleanup.policy = delete" means that topics will by default get pruned past retention time.

If you choose this default policy, then you will need to override the cleanup.policy per topic; that is, set the cleanup.policy=compact explicitly on this topic. This will turn this specific topic to use compaction, rather than delete. You do not need to adjust log.retention.

PS, Intuitively, I would think that the default "log.cleanup.policy = [compact, delete]" gets overwritten when you specify "log.cleanup.policy = compact" on per topic basis, but this is not so. With "log.cleanup.policy = [compact, delete]" you are effectively overriding how compact topics work; you change compact to be compact+delete.

PS2, if you have trouble testing and getting your topic to compact, note that only the inactive file segment can be compacted; active segment will never be compacted. So for testing, set log.segment.bytes to something small, say 10000