Why can't I divide integers in swift?

The OP seems to know how the code has to look like but he is explicitly asking why it is not working the other way.

So, "explicitly" is part of the answer he is looking for: Apple writes inside the "Language Guide" in chapter "The Basics" -> "Integer and Floating-Point Conversion":

Conversions between integer and floating-point numeric types must be made explicit

you just need to do this:

func averageOf(numbers: Int...) -> Double {
    var sum = 0, countOfNumbers = 0
    for number in numbers {
        sum += number
        countOfNumbers++
    }
    var result: Double = Double(sum) / Double(countOfNumbers)
    return result
}

You are assigning the output of / to a variable of type Double, so Swift thinks you want to call this function:

func /(lhs: Double, rhs: Double) -> Double

But the arguments you're passing it are not Doubles and Swift doesn't do implicit casting.