'self' used before all stored properties are initialized

In this case I would suggest you to make the property a variable but hiding it (make it seem like a constant) through a computed property:

class Country {
    let name: String

    private var _capitalCity: City!
    var capitalCity: City {
        return _capitalCity
    }

    init(name: String, capitalName: String) {
        self.name = name
        self._capitalCity = City(name: capitalName, country: self)
    }
}

Is there any way to resolve the syntax error while keeping capitalCity a constant?

Not the way you have things set up. The source of the problem is actually that in order to set capitalCity, you have to create a City whose country is self. That is the use of self to which the compiler is objecting:

self.capitalCity = City(name: capitalName, country: self)
                                                    ^^^^

Since you have configured City's country as a constant, you must supply this value when you initialize your City. Thus you have no way out; you must make capitalCity an Optional var so that it has some other initial value that is legal, namely nil. Your proposed solution actually works like this:

class Country
{
    let name: String
    var capitalCity: City! = nil // implicit or explicit

    init(name: String, capitalName: String)
    {
        self.name = name
        // end of initialization!
        // name is set (to name), and capitalCity is set (to nil)...
        // ... and so the compiler is satisfied;
        // now, we _change_ capitalCity from nil to an actual City,
        // and in doing that, we _are_ allowed to mention `self`
        self.capitalCity = City(name: capitalName, country: self)
    }
}

Just do:

private(set) var capitalCity: City!

which gives you the read-only public interface you want.

I understand that you find var capitalCity: City! contrived. I'd say that the selected answer is truly contrived; it adds lines of code that have no purpose other than resolving a language-related issue. Lines like that are meaningless and meaning is what matters most in code.