How to connect my Swift app to my Parse Server?

Parse Server now has some good documentation going and it basically recommends @fraxool's solution with a little neater syntax:

let configuration = ParseClientConfiguration {
    $0.applicationId = "YOUR_APP_ID"
    $0.clientKey = ""
    $0.server = "http://localhost:1337/parse"
}
Parse.initializeWithConfiguration(configuration)

Found the answer by myself, here is how to set a configuration (including the server URL) with Swift :

let parseConfiguration = ParseClientConfiguration(block: { (ParseMutableClientConfiguration) -> Void in
    ParseMutableClientConfiguration.applicationId = "APP_ID"
    ParseMutableClientConfiguration.clientKey = "CLIENT_KEY"
    ParseMutableClientConfiguration.server = "http://your_server.com:1337/parse"
})

Parse.initializeWithConfiguration(parseConfiguration)

Hope it will help someone else.