Dictionary in Swift with Mutable Array as value is performing very slow? How to optimize or construct properly?

Copy on write is a tricky thing, and you need to think carefully about how many things are sharing a structure that you're trying to modify. The culprit is here.

countToColorMap[colorCount]?.append(CountedColor(color: color as! UIColor, colorCount: colorCount))

This is generating a temporary value that is modified and put back into the dictionary. Since two "things" are looking at the same underlying data structure (the dictionary, and append), it forces a copy-on-write.

The secret to fixing this is to make sure that there's only one copy when you modify it. How? Take it out of the dictionary. Replace this:

if countToColorMap[colorCount] != nil {
    countToColorMap[colorCount]?.append(CountedColor(color: color as! UIColor, colorCount: colorCount))
} else {
    countToColorMap[colorCount] = [CountedColor(color: color as! UIColor, colorCount: colorCount)]
}

which has a runtime of:

Elapsed Time: 74.2517465990022
53217

with this:

var countForColor = countToColorMap.removeValue(forKey: colorCount) ?? []
countForColor.append(CountedColor(color: color as! UIColor, colorCount: colorCount))
countToColorMap[colorCount] = countForColor

which has a runtime of:

Elapsed Time: 0.370953808000195
53217

I had some work arounds until swift 4.2 came along

var countToColorMap = [Int: [CountedColor]]()

for (color, colorCount) in colorInfo {
    countToColorMap[colorCount, default: [CountedColor]()].append(CountedColor(color: color as! UIColor, colorCount: colorCount))
}

it is fast and readable