Is Apex Sometimes Case Sensitive?

Yes. Collection membership is case sensitive.

This applies to Sets:

If the set contains String elements, the elements are case-sensitive. Two set elements that differ only by case are considered distinct.

and to Maps:

Map keys of type String are case-sensitive. Two keys that differ only by the case are considered unique and have corresponding distinct Map entries. Subsequently, the Map methods, including put, get, containsKey, and remove treat these keys as distinct.

Map keys and contains checks (and the same with Set or List) are always case sensitive, with the one exception of describes such as fields.getMap() or Schema.getGlobalDescribe(). However, as soon as you pull the keyset and put it somewhere else, you lose this magical property.

From SF docs on Mapss:

Map keys of type String are case-sensitive. Two keys that differ only by the case are considered unique and have corresponding distinct Map entries. Subsequently, the Map methods, including put, get, containsKey, and remove treat these keys as distinct.

From SF DOcs on Sets

If the set contains String elements, the elements are case-sensitive. Two set elements that differ only by case are considered distinct.

Maps and Set uses Hash Collision and hence they treat keys as the case sensitive.

The only exception is Set<Id> and Map<Id,Object>

Set<Id> setId = new Set<Id>{'0060D0000038F05'};

System.debug(setId.contains('0060D0000038F05QAE')); //Prints true