Tombstone messages not removing record from KTable state store?
As documented in the JavaDocs of reduce()
Records with {@code null} key or value are ignored.
Because, the <key,null>
record is dropped and thus (genericRecord, v1) -> v1
is never executed, no tombstone is written to the store or changelog topic.
For the use case you have in mind, you need to use a surrogate value that indicates "delete", for example a boolean flag within your Avro record. Your reduce function needs to check for the flag and return null
if the flag is set; otherwise, it must process the record regularly.
Update:
Apache Kafka 2.6 adds the KStream#toTable()
operator (via KIP-523) that allows to transform a KStream
into a KTable
.