How can I obtain a CGColor from RGB values?

You have to give the values between 0 and 1.0. So divide the RGB values by 255.

Change Your code to

var red = UIColor(red: 100.0/255.0, green: 130.0/255.0, blue: 230.0/255.0, alpha: 1.0)
self.layer.borderColor = red.CGColor

You can get that down to one line in Swift 3:

avLayer.backgroundColor = UIColor.red.CGColor