Why is Kafka consumer ignoring my "earliest" directive in the auto.offset.reset parameter and thus not reading my topic from the absolute first event?

It's because auto.offset.reset is only applied if there are no committed offsets for the group.

See the consumer configs documentation:

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server

If you want to restart from the beginning, you can either:

• use a new group name (for example append System.currentTimeMillis() to the group anme)

• explicitly move the position of the consumer to the start of the partition using seekToBeginning(): http://kafka.apache.org/11/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seekToBeginning-java.util.Collection-

The property auto.offset.reset is used only when there is no offset for the given consumer stored in Kafka. When you commit the record Kafka stores the offset of the record in a special topic and in the next run, your consumer will read the topic from the last committed offset. To read from the beginning you should call consumer.seekToBeginning or use unique group.id property.