How to remove a key from HashMap while iterating over it?

Use Iterator.remove().

Try:

Iterator<Map.Entry<String,String>> iter = testMap.entrySet().iterator();
while (iter.hasNext()) {
    Map.Entry<String,String> entry = iter.next();
    if("Sample".equalsIgnoreCase(entry.getValue())){
        iter.remove();
    }
}

With Java 1.8 and onwards you can do the above in just one line:

testMap.entrySet().removeIf(entry -> "Sample".equalsIgnoreCase(entry.getValue()));